A one-person music benchmark that has been quietly tracking how badly language models write counterpoint just recorded its first clean pass. GPT-6 Astra produced a four-part chorale in G minor with no voice-leading errors, according to Auggie, the classical musician behind the Augmented Fifth newsletter and the Bach Benchmark. He says the harmony reaches as far as a Neapolitan sixth, and that it is "the first model to ever write passing tones on this benchmark, a significant leap in musical understanding." The post has drawn more than 8,000 likes.
GPT-6 Astra's chorale in G minor, as engraved from the model's LilyPond output. Credit: Auggie.
The benchmark is a single fixed prompt, run one-shot: "In LilyPond (version 2.24), write a 4-part chorale in the style of Bach, 3/4 time, G minor. Use two staves — soprano and alto on the top staff, tenor and bass on the bottom staff. Respond with only the code block." Astra was run at Extra High reasoning effort. LilyPond is a text markup language that compiles to engraved notation, so the model has to emit code that both compiles and, when it does, survives the marking a first-year harmony student gets: no parallel fifths or octaves, voices inside their ranges, chords that make sense.
When Auggie introduced the benchmark, the best result he had ever seen was Claude Fable 5 — the run this new post quote-tweets. It was still, in his words, imperfect: "there are still elementary mistakes, such as the parallel octaves in measures 10 and 13."
The previous best on the benchmark: Claude Fable 5's chorale, with parallel octaves in bars 10 and 13. Credit: Augmented Fifth.
The two other models he ran on the same prompt failed in more basic ways. GPT-5.5 wrote long sequences of parallel intervals and sent voices well outside their range — the bass in the second system drops so far below the staff it needs a ladder of ledger lines. Gemini 3.5 Flash avoided both traps but broke the harmony instead, putting a B natural and a B flat in the same chord; it also failed to one-shot the prompt, needing a second turn to fix LilyPond errors.
GPT-5.5 on the same prompt: parallel motion marked in grey, and a bass that leaves its range entirely. Credit: Augmented Fifth.
Avoiding parallel fifths is rule-following; you can check it with a script. Passing tones are the opposite kind of skill — non-chord tones that fill the gap between two harmony notes on a weak beat, which is what makes a chorale sound written rather than merely legal. A model that produces them is imitating idiom, not just satisfying constraints. That is the claim Auggie is making, and it is a qualitative one.
It also fits the pattern in his other tests, which have been falling fast. His undergraduate music-theory exam — twelve deliberately nasty chord-spelling questions, including a "7#5 chord built on the supertonic of A-sharp minor" that requires a triple sharp — was passed by every current model he tried, with GPT-5.6 Sol scoring 100%. He retired it as a benchmark on the spot. Applied retroactively, Claude Sonnet 4 scored 0% against Sonnet 5's 91%, and GPT-4.1 scored 16% against GPT-5.5's 83%.
This is one evaluator, one prompt, one run, and a subjective rubric with no numeric score. The chorale was published as a video of the engraved score with audio; the LilyPond source has not been posted, so no one else has recompiled it or marked it independently. Luis Garicano, replying to the post, was generous but hedged: "This seems right and beautiful to me (albeit too short to really know)." Others rejected the frame entirely — one composer replied that if a model "cannot think or feel, then its compositions are meaningless."
Formal work on the same substrate exists — LilyBench, presented at Ital-IA 2026, pairs a 200-prompt LilyPond generation suite with ten understanding tasks and finds open-weight models can emit compilable LilyPond zero-shot while structural reasoning stays hard. The Bach Benchmark is not that. It is a musician with a fixed prompt and an ear — which is also why the result travelled.
Auggie's post on XThe Bach Benchmark, Augmented FifthTesting LLMs on Undergraduate Music TheoryClaude Fable 5's choraleLuis Garicano's replyLilyBench (arXiv 2606.08722)
