About these notes
This collection documents Satish Bajagain’s study of neural networks through Andrej Karpathy’s Zero to Hero lectures, additional reading, independent derivations and experiments.
The original notebooks mix lecture explanations, personal understanding and explanations developed with outside resources, including AI assistance. They are study records; individual passages are not all independently attributed yet. Linked sources and the original context remain available.
Reading the original material¶
The original notebook text is preserved. Comment-only cells are displayed as readable notes with their # markers retained. Cells that mix comments and code keep both together. Raw cells and explanations stored inside HTML comments are made visible. Section titles are added for navigation.
Saved outputs record earlier notebook sessions. They are not claims that every cell was rerun in sequence for this site. Interrupted runs and intermediate mistakes remain part of the record.
Images retain their original resolution. Use the “Open original image” links to inspect handwritten steps. The download bundle includes the original files and their relative paths.
Added explanations¶
Reading guides and recall answers are supplementary material prepared with AI assistance and checked against the relevant source cells and mathematics. They do not replace or silently correct the original notes. Each guide identifies its source context.
Scope of this collection¶
All seven current notebooks are available in the reading edition. The first supplementary guide and recall set focus on backpropagation. The BatchNorm slider needs Jupyter because its interactive state was not saved with the notebook. There is no online execution or account-based progress tracking.
Some diagrams were collected from outside resources, and their individual source attributions are still being completed. They are retained in their original study context and are not all claimed as original artwork.
Attention scaling¶
Editorial correction · 10 October 2026 · AI-assisted review. The preserved v2.py script uses C**-0.5 in Head.forward, where C is the input embedding width. Standard scaled dot-product attention divides by the square root of the per-head key dimension, so that expression should use k.shape[-1]**-0.5:
scores = (q @ k.transpose(-2, -1)) * (k.shape[-1] ** -0.5)Apply the causal mask before the softmax, as in the original implementation. The original script’s embedding width and head width differ, so these scales are not interchangeable. Using the larger width reduces score magnitudes and changes the attention distribution. This observation does not establish how retraining would change the model’s validation loss.
See Vaswani et al., Attention Is All You Need, §3.2.1 for the definition. Original notebook cells and download files remain unchanged as historical study material; the correction above is explicitly separate. The collection is a completed learning project, not a claim that all intermediate code is free of mistakes.