Phillip Isola

MIT Paper Proposes Transformer-Guided, Time-Parallel Training for Nonlinear RNNs

MIT researchers propose Supervised Memory Training, which uses predictive-state labels from a Transformer encoder to train nonlinear RNN updates in parallel without backpropagation through time.