Is Pruning LLMs a Lie?

With enough training tokens, a smaller dense LLM trained from scratch matches or beats structured pruning. Coarse depth/width pruning isn’t transferring knowledge — it’s closer to an expensive neural architecture search.

A new paper (“Small LLMs: Pruning vs. Training from Scratch”) ran strict token-matched evaluations across six pruning methods and five granularities on Llama-3.1-8B. In the equal-total-tokens regime, a 4.5B dense model trained from scratch outperformed both depth-pruned and width-pruned variants. Unstructured sparse methods still held an edge, but they offer no real speedup on standard GPUs.

Practical takeaway: When data is scarce, prune a large foundation model for a head start. When data is abundant and you need a dense model, train the target size from scratch — pretraining a huge parent just to prune it is often wasteful.

Source: @che_shr_cat