Adversarial Review: three agents hit 75.2% on SWE-bench Verified by forcing disagreement

A three-agent code-review protocol outscored a five-agent one on every benchmark its authors tested — and the more useful finding is the failure mode they hit on the way there. Adversarial Review (AR), from Eric S. Qiu (Cornell University) and Joyce Gill (Stanford University), reaches 75.2% pass@1 on SWE-bench Verified against 71.6% for a zero-shot Claude Code baseline, and does it with a main agent, one reviewer and one critic. The paper was accepted to the ICML 2026 Workshop on DL4C.

The shape is deliberately small. A main agent M produces an artifact (code or a plan). The artifact is then frozen while a reviewer subagent R writes a review and a fresh critic subagent C audits that review, pushing back until the two converge or a cap of five inner rounds is reached. Only then does M edit, producing the next version. The inner loop exchanges review text; edits happen only in the outer loop. All agent and subagent calls in the paper use Claude Sonnet 4.5 at medium reasoning.

Figure 1: the AR workflow — the inner loop exchanges review text only, artifact edits happen only in the outer loop. Credit: Qiu and Gill, arXiv:2608.18167.

#More reviewers did nothing; a critic did

The authors build AR one step at a time on LiveCodeBench (105 stdin-style tasks, 57 tagged hard), each step motivated by a measured failure of the last. Zero-shot, Self-Refine, one reviewer and two independent reviewers all land in the same cluster — self-critique fails because "the critic and the generator are the same model making the same mistakes." MARS, a five-agent system, is the first to break out. AR beats it with three.

#The false-consensus failure

The result does not transfer cleanly. On SWE-PRBench — 100 real GitHub PR diffs, scored by a GPT-5.2 judge (Cohen's kappa 0.75 against human annotators) on F1 over matched comments — naive AR is the worst of the four review-capable methods at F1 = 0.457, below single-reviewer's 0.495.

Two case studies explain why. In one, the reviewer hedges, the critic confirms the hedges and adds a speculative bug of its own; the judge marks 3 of 5 comments fabricated. In the other, the critic raises a real bug, the reviewer rebuts with a file-level argument citing no code, and the critic gives in — the bug drops out of the final review. The authors call this false consensus: agreement that "can look like independent validation" while reflecting only conversational pressure to converge.

The fix is prompt text, not more agents. The critic must pick one of three verdicts instead of two — AGREE, DISAGREE_EVIDENCE: <code citation>, or DISAGREE_CONCERN: <epistemic objection> — and on a concern-only objection the reviewer must cite code in the diff that confirms the flag or drop it. That single iteration takes AR to F1 = 0.533, the highest in the subset.

#The protocol survives as a SKILL.md

The SWE-bench Verified run is the part practitioners will care about: instead of a Python orchestrator, AR is written as a plain-text SKILL.md and handed to Claude Code with full tool access, which decides for itself when to spawn R and C as Task subagents. It still works — 75.2% against 72.6% for MARS and 71.6% for zero-shot on all 500 tasks, at roughly 4.5x the tokens of zero-shot. The authors report AR on the cost-quality Pareto frontier of all three benchmarks: nothing they tested is both cheaper and better.

Two traces show both edges. On matplotlib-20826, zero-shot and MARS patch the caller and fail hidden tests; AR's critic catches the reviewer's first-round mistake and the agent patches the callee instead — a ~20-line root-cause fix that passes. On astropy-14182, the reviewer asks "what if the user calls read() separately?", the critic fails to reject it as out of scope, and AR ships a 32-line patch that fails where zero-shot's 24-line one passed.

#Why it matters

The default response to weak agentic review is more agents, and this is a careful argument that the default is wrong: two interacting reviewing agents beat four non-interacting ones, and what does the work is forcing disagreement to be explicit and grounded in cited code. Elvis (@omarsar0), who surfaced the paper, frames it the same way — scaling agent count gives diminishing returns on repository-level tasks.

The caveats are the authors' own. This is a preprint on a single model family, SWE-PRBench leans on an LLM judge that may penalize valid comments differing from the human reviewer's, and SWE-bench Verified measures hidden tests rather than maintainability. AR also costs 4.5x the tokens and can over-edit; the paper's own suggestion is an adaptive variant that invokes the loop only when the main agent is uncertain or the patch touches risky code.