LLM as a Judge — ranking full slates by preference
November 6, 2025Posted by Federico Ulfo
Instead of simulating clicks and scrolls, researchers let LLMs reason which playlist, feed, or product lineup you’d actually prefer.
And it worked. Across Amazon, Spotify, MovieLens, and MIND datasets, they found:
- LLMs can rank full slates (not just single items) with strong coherence
- Logical consistency directly predicts preference accuracy
- Pretrained models generalize no fine-tuning required