Permutation Invariance and Equivariance in Transformers
Master the math of permutation invariance in self-attention. Prove equivariance, analyze CBoW limitations, and learn why Transformers need positional signals.
Content adapted from Attention Is All You Need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin.Original Source