Kjartan Olafsson

Microsoft's Thinkingbox grades agents on the database: 66.5% once, 47.5% every time

Microsoft released Thinkingbox and a 507-task Thinkingbox-bench that grades agents by checking backend database state, finding Claude Opus 5 tops single-attempt success at 66.50% but passes all 20 tries on only 47.53% of tasks.