Backpropagation recall
Supplementary questions and answers · AI-assisted. Try an answer before opening the explanation. Each question returns to the original notebook context.
Why must gradients accumulate?¶
A value influences the loss through two different paths. Why does assigning its gradient twice give the wrong result?
Reveal the explanation
The total derivative adds the contributions of all paths. Assignment can overwrite an earlier contribution. Accumulating with += retains both. This also explains why gradients must be cleared before a new independent backward pass.
Why is the weight matrix transposed?¶
For logits = h @ W2 + b2, why is dh = dlogits @ W2.T?
Reveal the explanation
Each entry of h affects all output channels in its row. Its gradient sums the output gradients multiplied by the connecting weights. W2.T arranges those weights for that sum. The shapes (32, 27) @ (27, 64) → (32, 64) provide a useful check.
Why does broadcasting turn into a sum?¶
A bias of shape (27,) is added to a (32, 27) array. Which dimension must its gradient sum over, and why?
Reveal the explanation
The same bias entry is reused for every example. All 32 uses contribute to its derivative, so sum over the batch dimension, dim=0. Broadcasting does not create 32 independently learned biases.
What happens when an embedding index repeats?¶
Two positions look up the same row of an embedding table. What happens to that row’s gradient?
Reveal the explanation
Both positions depend on the same parameters. Their gradient contributions add into that row. A backward implementation that overwrites the row loses a contribution.
Why do the logit gradients sum to zero?¶
For softmax cross-entropy with one correct class, why does the sum across the logit gradients vanish?
Reveal the explanation
For one example, the gradient is p - y, where predicted probabilities sum to one and the one-hot target also sums to one. Their difference sums to zero. Averaging over examples scales the result without changing that property. It is also consistent with softmax being unchanged when the same constant is added to every logit.
Why are examples coupled by BatchNorm?¶
During training, why can changing one example affect another example’s normalized activation?
Reveal the explanation
Their normalized values share the batch mean and variance. Changing one value changes these statistics, which can change every normalized value in that channel. Those shared dependencies also matter during backpropagation.