CIFAR-10 Two-Layer Classifier

Overview

For the second assignment in KTH's Deep Learning in Data Science course I built a two-layer fully connected network in NumPy, with no model.fit() and no autograd. That meant writing the forward pass, softmax cross-entropy loss, ReLU and all of backpropagation myself, and then training it on CIFAR-10.

The baseline got 46.01% test accuracy and my final configuration got 49.77%. That's close to the best you can expect from a two-layer fully connected net on raw CIFAR-10 pixels.

The pipeline

• Per-dimension normalization, mini-batch gradient descent with L2 regularization, and cyclical (triangular) learning rates from Smith (2015) with η between 1e-5 and 1e-1.
• Every analytic gradient gets compared against PyTorch autograd, and the max error is around 1e-16. Before any real training run I also made sure the network could overfit a tiny subset.
• λ was picked with a log-scale search over about 45k training images. The coarse pass covered 1e-5 to 1e-1 and the fine pass covered 5e-4 to 7e-3. I ended up using λ = 1.07e-3 for the final 3-cycle run.

Bonus part

For the bonus I stacked four improvements on top of each other: a wider hidden layer (m = 256), horizontal flips, random translations and dropout. One result I liked was that every time I added augmentation, the best λ moved lower. That fits with the idea that augmentation and L2 regularization partly do the same job.

The LaTeX report covers the gradient checks, how the cyclical learning rate behaves over 1 and 3 cycles, and the best configurations from both search stages.

Source

Code and report: github.com/rippyboii/cifar10-twolayer-classifer