Trust, but Verify: Checking Backprop to 1e-16

For KTH's Deep Learning in Data Science course I built CIFAR-10 classifiers from scratch in NumPy, first a linear softmax model and then a two-layer network with ReLU. Doing it from scratch meant I derived and wrote every gradient in the backward pass myself.

The problem with writing backprop by hand is that a slightly wrong gradient will still train. The loss goes down and the accuracy looks believable. You could run a whole hyperparameter study on a broken backward pass and never find out, because wrong gradients don't crash anything. They just limit how good the model can get.

So before training anything, I made checking the gradients the first milestone.

Step 1: use autograd as the reference

I had PyTorch compute the same loss with the same weights, let autograd work out the gradients, and compared them element by element with mine:

def relative_error(g_mine, g_torch, eps=1e-12):
    num = np.abs(g_mine - g_torch)
    den = np.maximum(eps, np.abs(g_mine) + np.abs(g_torch))
    return (num / den).max()

I wasn't aiming for "close enough". I wanted the difference to be down at machine precision, where the only error left comes from floating-point rounding. The max error ended up around 1e-16. That's a useful number because it basically gives you a yes or no answer. At 1e-16 the math is the same. At 1e-4 something is wrong, and blaming numerical issues is just an excuse.

One thing that caught me out was dtype. NumPy uses float64 by default and PyTorch uses float32. If you compare the two directly you'll see errors around 1e-7 and waste time looking for a bug that doesn't exist. Cast everything to float64 first.

Step 2: overfit a tiny subset

A gradient check shows that the backward pass matches the forward pass. It doesn't show that the training loop is correct. The learning rate update, batch shuffling and regularization could all still be wrong.

The cheapest way to test that is to take about 100 images and train until the model memorizes them. A correct implementation should get close to 100% training accuracy and near-zero loss. If it can't overfit 100 images, it won't learn anything from 50,000. This catches different bugs from the gradient check, so the two together cover most of what can go wrong.

Why it was worth it

Because I trusted the gradients, I could trust the results that came after:

• The cyclical learning rate schedule (Smith 2015) behaved the way the paper describes, and you could see it clearly in the loss curves.
• The coarse-to-fine λ search over about 45k images gave smooth validation results that I could reproduce, and I picked λ = 1.07e-3 for the final 3-cycle run.
• In the bonus experiments, adding data augmentation kept shifting the best λ lower. That suggests augmentation and L2 regularization can partly replace each other, and I was comfortable saying that because I knew the gradients were exact.

Final test accuracy was 39.30% for the linear model and 49.77% for the two-layer network with all four bonus improvements. That's decent for raw pixels, but the number I'm happiest with is still the 1e-16.

Takeaway

It's the same idea as working with hardware, where you calibrate a sensor against a reference before you trust its readings. I think about backprop the same way now: check your tools before you run the experiment.