Measuring What Frontier Models Know (IKP)

• Bojie Li introduces Incompressible Knowledge Probes (IKP), 1,400 obscure factual questions across 7 tiers of difficulty, to measure factual recall in 188 models from 27 vendors including closed APIs.
• Factual accuracy scales log-linearly with log(model parameters) on open-weight models (R²=0.917), allowing black-box size estimates: GPT-5.5 ~9T, Claude Opus 4.6 ~5T, with wide uncertainty ranges noted in follow-up.
• Over three years, factual capacity shows no compression at fixed parameter counts, rejecting the Densing Law prediction of knowledge densification, while reasoning benchmarks saturate.

#Estimated size per model:

• GPT-5.5 ~9T
• Claude Opus 4.7 ~4T
• GPT-5.4 ~2.2T
• Claude Sonnet 4.6 ~1.7T
• Gemini 2.5 Pro ~1.2T

Sources: tweet, paper, ikp