Stanford turns raw screen recordings into task models that lift agent accuracy 30%

Four researchers at Stanford and Carnegie Mellon have a method for reading a raw screen recording — thousands of clicks, keystrokes and screenshots with no labels attached — and recovering an explicit model of the work it contains. Task Model Induction (TMI), by Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen and Diyi Yang, accepted to EMNLP 2026, reconstructs 74.9% of a session's execution steps against 30.3% for the strongest workflow-induction baseline, and the agent skills written from its output raise held-out task accuracy 30% over that baseline.

The premise is that most knowledge work is never documented and the expertise stays tacit. A passively recorded trace has all of it, but only as pixels and input events — and real sessions are multi-threaded, with users switching between unrelated goals and interleaving sub-goals inside a single task. Prior methods either assume the task is known in advance or flatten the whole recording into one continuous workflow.

Figure 1: screenshots and input events are grounded into semantic actions, whose interleaved spans are assigned to latent tasks. Credit: Jiang et al., arXiv:2608.20319.

#How it works

TMI runs in three stages. Event grounding shows a vision-language model the screenshot pair bracketing each event plus the raw operation, and asks what changed — turning click(1900.8, 29.5) into "bring the Codex app to the foreground" — then groups the results into semantic actions and activities. Latent task induction assigns each activity to the closest existing task or opens a new one, holding a task together across application and naming shifts with a set of referential identifiers (a project may appear as the repo study-frontend and the URL review-trial.web.app), then merges split tasks in a global consolidation pass.

Task model construction is where the paper's real argument sits. Each task gets two models induced independently: an objective model, a recursive decomposition of the goal into sub-goals; and a procedure model, a tree over the control-flow operators a trace can actually evidence — sequence, for-each and while. Selection is dropped, because a recording shows the strategy the user enacted, not the alternatives they rejected. The two are then reconciled: the procedure model corrects the boundaries of 64.5% of objective nodes, and the objective model corrects 21.9% of procedure-node boundaries. Ablations confirm neither half suffices — procedure-only scores 63.2% step accuracy, and inducing both in one joint pass produces half as many nodes.

Every pipeline stage runs on gpt-5.4 at temperature 1.0; gpt-5.5 and claude-sonnet-5 serve as independent judges, and gpt-5-mini does the downstream skill work.

Figure 3: one session that built a study frontend yields three latent tasks, none of them occupying a contiguous span. Credit: Jiang et al., arXiv:2608.20319.

#What the numbers cover

The intrinsic evaluation uses HumanWork, 38 recorded human sessions spanning 15 tasks across five professional domains — 42.8 hours and 48.7K raw events — plus 195 coding-agent runs from SkillsBench across Claude Code, Codex and Gemini CLI. Synthetic multi-task trajectories, built by cutting sessions into segments and shuffling them, sweep 2 to 15 concurrent tasks: TMI holds 0.974 Adjusted Rand Index against ground-truth groupings, with task-count error under 1 throughout. Induced task sets match the withheld ground-truth label in 94.74% of human sessions.

Real sessions are messy in ways a flat step list cannot express — error correction appears in 89% of them, exploratory search in 87%, redundant repetition in 84%. Step accuracy drops to 66.7% on the hardest of those spans, still more than double the baseline's rate over all nodes.

The extrinsic test feeds each representation into the same Codex skill creator and runs the resulting skill on held-out SkillLearnBench instances:

Note the expert-written skills: highest skill coverage at 93.59, lowest held-out accuracy of any generated source. The authors present that row as a quality reference point, not an upper bound.

#Why it matters

The interesting claim is not the benchmark delta but the artifact. A task model is symbolic and auditable — a tree where every node carries an objective, a control-flow operator and the specific actions grounding it — which makes it something an organization can inspect and reuse rather than a summary it has to trust. That is the shape a lot of "agents learn from watching you" pitches are missing.

The reference implementation is Apache-2.0, and it is a full stack: a signed macOS recorder, a Dockerized OCR + OmniParser + VLM grounding service, the seven-stage pipeline, and a local visualizer. The obvious caveat is the one the authors and the repo both flag: the recorder captures everything on screen, the paper's traces come from public research datasets rather than a real workplace, and privacy redaction before induction is left to future work.