MACHINE LEARNING / NOTES
Understanding attention
How a model finds what matters.
A weighted view of the input
A query is compared with a set of keys. The resulting scores determine how much each value contributes.
Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V
Divide by the square root of the key dimension, then turn the scores into weights.