neural-from-scratch — case study

Why bother

PyTorch is a beautiful piece of software, and that beauty is mostly built on top of one trick: a computation graph that knows how to take its own derivative. For a long time I knew this in the way you know the capitals of countries you've never been to. I could write a forward pass; I could call loss.backward(); I had no mental picture of what was happening in between.

So I followed Karpathy's micrograd lecture and then went further than I needed to. The result is nfs — neural-from-scratch — a library small enough to fit in a single afternoon's reading, but actually capable of training a real model.

What it does

Tensor type with autograd: every operation records its parents and a backward function.
Layers: Linear, ReLU, Sigmoid, Tanh, Softmax, Dropout, BatchNorm1D.
Losses: MSE and cross-entropy with stable log-softmax.
Optimizers: SGD, SGD+momentum, Adam.
Training loop: mini-batching, shuffling, a simple LR scheduler.
~600 lines total, no dependencies beyond NumPy and matplotlib for plots.

The autograd, briefly

Every tensor remembers its parents and how to send a gradient back to them:

class Tensor:
    def __init__(self, data, _parents=(), _op=""):
        self.data = np.asarray(data, dtype=np.float32)
        self.grad = np.zeros_like(self.data)
        self._parents = _parents
        self._backward = lambda: None

    def __add__(self, other):
        out = Tensor(self.data + other.data, (self, other), "+")
        def _back():
            self.grad  += out.grad
            other.grad += out.grad
        out._backward = _back
        return out

    def backward(self):
        # topo sort, then call _backward in reverse
        ...
        self.grad = np.ones_like(self.data)
        for node in reversed(topo):
            node._backward()

That's the whole trick. Every other op is the same pattern: do the forward computation, register a closure that knows how to push the gradient backward, return a new tensor that points at its parents. Backward is a topological sort plus a loop. I had to write it three times before I really believed it was that small.

What it taught me

Backprop isn't deep, it's just bookkeeping. The hard part is being careful with shapes and broadcasting; the calculus is high-school stuff.
Numerical stability is the actual hidden subject. My first softmax+cross-entropy implementation produced NaNs whenever a logit went above 20. Fixing it taught me more about floating point than four years of CS courses had.
Optimizers are smaller than they look. Adam is 12 lines. Knowing what β₁ and β₂ actually do changed how I read papers.

aha moment

The point where my BatchNorm implementation started training stably is the point I genuinely understood what BatchNorm does. Not from the paper, not from a blog post — from getting it wrong in a way that produced wildly diverging gradients and then fixing it.

Result

MNIST in three Linear+ReLU layers, trained with Adam for 5 epochs, batch size 128, lr 1e-3: 97.1% test accuracy. Same model in PyTorch, same hyperparameters: 97.3%. I'll take it.

More importantly, I now read PyTorch source code without flinching. When something doesn't behave the way I expect, I have a real mental model to check against. That was the whole point.