ML Street Talk — Transformers Need Glasses!
March 16, 2025Posted by Federico Ulfo
ML Street Talk, is one of my new favorite AI podcast, incredible topic quality and guests.
Federico Barbero discusses why transformers struggle with tasks like counting and copying long text due to architectural bottlenecks and limitations in maintaining information fidelity. He draws comparisons to over-squashing in graph neural networks and highlights the role of the softmax function in these challenges, while also proposing practical modifications to improve transformer performance.