MLST: AI is learning at the wrong level of abstraction

Matthieu Wyart, a statistical physicist, joined Tim Scarfe on Machine Learning Street Talk to answer a question that sounds simple and is not: why can deep networks discover abstractions that shallow models miss?

#The argument

Wyart's answer is that the structure is in the data, not the architecture. Language and images are built from parts within parts — a hidden hierarchy — and depth is what lets a network recover those coarse-grained variables. That recovery is his account of how deep nets escape the curse of dimensionality, and it draws directly on the Random Hierarchy Model work.

Two consequences follow that are worth arguing with:

• Next-token prediction can still recover compositional structure — the objective is not the limitation people often take it for.
• But predicting latents rather than tokens could be far more sample-efficient. The episode cites JEPA and a sample-complexity theory for learning from your own latents.

#The wider tour

The conversation runs from jamming transitions and rough loss surfaces — Wyart's home turf — through Chomsky and context-free grammars, to where current systems fall short of genuine scientific invention rather than recombination. It closes on and , including a and work , and on the limits of physics-inspired theory generally.