NVIDIA finds skill doc-scans predict nothing about what a skill does at runtime

Twelve NVIDIA researchers have published ACES (Agentic Continuous Evaluation of Skills), a framework that grades an agent skill by running the same task twice — once with the skill loaded, once with it withheld — and reporting the difference as Skill Lift. Across 947 scored paired cases from 58 of 64 production skills and four agent harnesses, mean composite Skill Lift is 0.2134 (95% paired-case CI [0.1967, 0.2301]), with composite scores rising from 0.5326 at baseline to 0.7460 with the skill. The sharper finding is the one about the tooling everyone currently uses: on the 62 production skills that also had scan metadata, the static structural score correlates with live lift at Spearman ρ = -0.0181, and the LLM-judge rubric score at ρ = -0.0266. Both are indistinguishable from zero.

#The scans don't even agree with each other

Skills — a SKILL.md file of procedural know-how plus optional scripts, loaded on demand by Claude Code, Codex, Cursor and others — are reviewed today by four classes of tooling: structural checks, LLM-as-judge rubrics, script linters and security scanners. All four read the document; none runs it. The paper's analogy is compiling with -Wall -Werror: no warnings is not the same as correct behavior.

On a corpus of 145 real skills from internal NVIDIA repositories and public catalogs, 94.5% clear the default 70-point structural gate and 86.2% pass the LLM-judge rubric — but the two scores correlate at only Spearman ρ = 0.14. The frontmatter contract turns out to be aspirational rather than enforced: 99.3% of skills fail to declare tools and 97.9% omit a Limitations section.

Figure 1: 94.5% of skills clear the structural gate, but structural and LLM-judge scores disagree (Spearman ρ = 0.14). Credit: NVIDIA, Evaluating Skills, Not Just Agents.

#What the paired runs measure

Each paired trial holds the task, model, harness, workspace, sandbox, grading policy and any configured prerequisite, helper or decoy skills fixed; only the target skill's availability changes. Trajectories are normalized into a shared schema (the Agent Trajectory Interchange Format, ATIF) so the same six graders — security, skill execution, skill efficiency, accuracy, goal accuracy and free-form expected_behavior checks — can score any harness. The headline run recorded 2,022 trajectories, 11,642 tool calls and 9,091 behavior observations.

The decomposition matters more than the headline. The biggest gains are in skill execution (+0.3263), behavior checking (+0.2983) and skill efficiency (+0.2758) — discovery, routing, read-before-execute order, tool waste. Final-answer accuracy improves too, but by only +0.1431. Skill efficiency is also the least reliable: it has the third-largest mean lift but is positive in just 41.7% of paired cases.

Figure 8: Skill Lift by metric across 947 paired cases. Credit: NVIDIA, Evaluating Skills, Not Just Agents.

Composite lift was positive in 689 cases, zero in 171 and negative in 87 — which the authors treat as the point rather than an embarrassment. Traces separate "never discovered" from "discovered but misused": agents that found the skill and then produced a truncated answer, skipped verification, or burned tool calls without improving anything. Per-harness means differ widely (0.3611 for OpenCode, 0.2904 for Claude Code, 0.1264 for Codex, 0.0896 for Terminus-2), but coverage is uneven and the authors refuse to read them as a leaderboard.

#The results decay when the model improves

A same-harness Codex sweep shows lift shrinking as the baseline model gets stronger: absolute performance stays high, but the agent needs less help. That argues for re-running skill evaluations after every model update, not once at publication. A separate stress test varied how many skills were visible in the workspace — mean lift held at 0.133–0.149 from 1 to 20 skills, but at 50 the with-skill pass rate fell to 0.55 and mean wall time hit 1,290 seconds, up from 258.

NVIDIA's parallel engineering blog applies the same protocol to its own catalog of 300+ verified skills across 30+ products, scored 0–100: Correctness 46 → 87, Effectiveness 39 → 78, Efficiency 43 → 78. It adds that token savings are not automatic — jetson-optimize-memory cut tokens from 617,306 to 142,540, while cuopt-install pushed them from 25,227 to 55,582.

#Why it matters

Skill registries now host thousands of community-contributed files that every major coding agent will load into a context window, and the gate in front of them is a linter. This is the first production-scale measurement saying that gate carries no runtime signal — a skill can score 89/100 and still route the agent wrong. The caveats are the authors' own: the corpus skews toward system-access, deployment and platform skills; harness coverage is uneven; the 947 cases are clustered by skill and harness and are not independent evaluations; and Skill Lift is a marginal contribution under a declared workspace, not an intrinsic property of a skill. The implementation is open source as NVIDIA SkillEvaluator.