Scaled Dot-Product Attention: Math, Variance, and Stability
Master the mechanics of attention. Derive dot-product variance, prove how 1/√dk scaling prevents softmax saturation, and analyze the Jacobian to ensure stable gradient flow in Transformers.
Content adapted from Attention Is All You Need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin.Original Source