CIFAR-10 Softmax Classifier

Overview

This was the first assignment in KTH's Deep Learning in Data Science course. I wrote a single-layer softmax classifier in NumPy and trained it on CIFAR-10 with mini-batch gradient descent and L2 regularization. The best settings (λ = 0.1, η = 0.001) gave 39.30% test accuracy, which is reasonable for a linear model working on raw pixels.

The pipeline

• Data normalization, the forward pass, cross-entropy loss and backpropagation, all derived and written by hand.
• I checked my gradients against PyTorch autograd. The max error was around 1e-16, so I knew the math was right before running any experiments.
• I plotted each class's weight matrix as an image. You can make out a rough template for each class, like a blurry car, or sky above a green field for deer.

Hyperparameter study

I swept over learning rates and regularization strengths and wrote up the results in a LaTeX report. Two things stood out:

1. With a high learning rate the loss keeps bouncing around the minimum and never settles.
2. Stronger regularization makes the weight templates smoother and easier to recognize, but test accuracy drops as λ goes up.

Source

Code and report: github.com/rippyboii/cifar10-softmax-classifier