4.1 Self-attention geometry and scaled dot-product in 3D
Self-attention lets every token ask a question of every other token, and a dot product in a learned space decides who answers.
01Queries, keys and values: three views of every token
A transformer layer receives a sequence of token embeddings stacked as the rows of . Self-attention rebuilds each row by pulling in information from the other rows. The whole mechanism comes down to one question: for token , which other tokens matter, and by how much?
Attention answers it by projecting every token three times with learned matrices. The query encodes what token is looking for, the key encodes what token can be found by, and the value is what token hands over once it is selected.
the robot picked up the red block: 2 positional features plus noun, modifier and verb flags
a 3-dimensional head, so every query and key in the scene is a real arrow
The score of key for query is their dot product, (lesson 0.2, Vectors, builds this identity). In the scene the amber arrow is the query of the selected token (picked) and the thin cyan arrows are the keys of all seven tokens. A key that points the same way as the query, and is long, scores high.
A worked example small enough to do by hand. Take three tokens, the, cat and sat, in a 2-dimensional head. The query of sat is and the keys are , , . The dot products are , and . Dividing by gives scores 0.71, 2.83 and 2.12. Exponentiate: 2.03, 16.92 and 8.34, which add up to 27.29. Divide each by that total and the weights are 0.07, 0.62 and 0.31: they are positive and sum to 1, and cat, the key that points most nearly along the query, gets the most.
Attention Is All You Need, Vaswani et al., 2017, Sec. 3.2
Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 3.3–3.4
02From dot products to attention beams
One matrix product computes every score at once: is an table whose row says how well each key matches query . A softmax across that row turns scores into weights, and the weights mix the values.
- 1
One score per (query, key) pair. The division keeps scores on a unit scale; step 3 explains why.
Reading the scene. While the animation plays, the scene walks through one row: each key lights up in turn with its score at its tip, then the softmax turns the scores into beams, then value pulses flow in and the emerald grows (the telemetry line step says which stage is showing; the cockpit's *Animate the computation* switches to the finished row only). Each beam runs from the query to key ; its cross-section is proportional to and the heaviest beam is emerald. A ring around the query shows how much it attends to itself. Pulses travel from the keys into the query because that is the direction values flow. The emerald arrow is , a weighted average of the value vectors, so it always lies inside their convex hull. Hover to see this: the scene redraws every value arrow from the query, and the tip of lands among their tips, pulled toward the heaviest one.
The bar strip shows one row. Stack all seven rows and you have the attention matrix , the picture most papers draw. The matrix below is computed from the scene's head, temperature, scaling and mask, so it changes with the cockpit; click a row label to make that token the query in the scene.
Attention Is All You Need, Vaswani et al., 2017, Sec. 3.2.1, Eq. 1
The rest of this lesson is in Pro
- 03Why divide by √dₖ: a variance argument
- 04Causal masking: no peeking at the future
- 05Multi-head attention: several geometries at once
- Python challenge: Scaled dot-product attention from scratch
- Knowledge check, 3 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is self-attention in a transformer?
- Self-attention computes, for every token, a weighted average of value vectors from all tokens in the sequence. The weights come from a softmax over dot products between that token's query vector and every token's key vector, where queries, keys and values are learned linear projections of the embeddings.
- Why is scaled dot-product attention divided by the square root of d_k?
- If query and key components have unit variance, their dot product has variance d_k. Dividing by √d_k restores unit variance, which keeps the softmax away from saturation, where its gradients vanish and training stalls.
- What does a causal mask do in attention?
- A causal mask adds −∞ to the scores of all future positions (key index greater than query index) before the softmax. Those weights become exactly zero and the rest renormalise, so a decoder can be trained on every position in parallel without seeing the tokens it must predict.
Next in Transformers and LLMs
4.2Tokenisation and embedding spacesByte-pair encoding from scratch, and how embedding tables turn token ids into directions with meaning.Sources
- Attention Is All You Need, Vaswani et al., 2017
- Build a Large Language Model (From Scratch), Sebastian Raschka