Reading guide — matrix gradients

Reading guide: matrix gradients¶

Supplementary explanation · AI-assisted. Read alongside the original matrix multiplication notes and the two handwritten derivation pages that follow them. The original comments are unchanged.

What each dimension represents¶

Your output layer computes

Here, $H$ has shape $32\times64$, $W$ has shape $64\times27$, and $Z$ has shape $32\times27$. Each row is one example; each output column corresponds to a vocabulary entry. The bias vector is shared across all 32 rows.

Let $G = \partial L / \partial Z$. The backward pass starts with $G$, which tells us how the scalar loss changes with each logit.

Follow one entry before using matrices¶

For one output entry,

An entry $H_{ik}$ contributes to every output column in row $i$. Those contributions must be added:

Writing those sums as a matrix product gives

A weight $W_{kj}$ is reused across the batch, so its gradient adds contributions over examples:

The same bias $b_j$ contributes to every example’s $j$th logit:

This explains the sum over the batch dimension in your bias-gradient code.

Shapes check the result; dependencies explain it¶

Your shape reasoning is a useful way to catch errors. A matching shape alone does not prove a derivative is correct: different expressions can share a shape. The index-wise chain rule above establishes the expression; dimensions provide a second check.

You do not need another division by batch size here if the incoming $G$ already contains the factor from a mean loss. The normalization travels backward with the gradient.

Check your understanding¶

Before looking back at the formulas, explain why the weight gradient sums across examples while the hidden-state gradient sums across output channels. Then compare your explanation with the arrows and sums in the original handwritten pages.

Try the recall questions.