Neural networks, and why depth turned out to matter so much.
14 topics · 23 curated works
No prior grounding assumed.
Neural Networks and Deep Learning
Michael Nielsen · 2015
Builds a working network from nothing and introduces each component only once the previous version has visibly failed, so the architecture arrives as…
+3 more at this level
Assumes you know the vocabulary.
Feature Visualization
Olah, Mordvintsev & Schubert · 2017
Argues the clearest way to see what a neuron in a trained network has learned is to synthesise, by gradient ascent, the input image that most excites…
Primary sources and full treatments.
Attention Is All You Need
Vaswani et al. · 2017
Replaces recurrence with self-attention, producing the Transformer architecture that underpins essentially all current language models.
+17 more at this level
12 of 23 works
Michael Nielsen
Builds a working network from nothing and introduces each component only once the previous version has visibly failed, so the architecture arrives as a series of fixes.
link checked 17 Sept 2026Grant Sanderson (3Blue1Brown)
Walks through what the layers of a network actually compute, using digit recognition as the worked example.
19 minuteslink checked 17 Sept 2026Grant Sanderson (3Blue1Brown)
Develops backpropagation visually from gradient descent on a cost surface, showing what the algorithm computes rather than restating the chain rule.
link checked 17 Sept 2026Grant Sanderson (3Blue1Brown)
Builds up the convolution operation visually from probability and moving averages through to image kernels, making the operation underneath every convolutional layer concrete before any network is discussed.
23 minuteslink checked 17 Sept 2026Olah, Mordvintsev & Schubert
Argues the clearest way to see what a neuron in a trained network has learned is to synthesise, by gradient ascent, the input image that most excites it — and works through the practical tricks needed to make that optimisation produce interpretable images rather than noise.
≈6,000 wordslink checked 17 Sept 2026Vaswani et al.
Replaces recurrence with self-attention, producing the Transformer architecture that underpins essentially all current language models.
15 pageslink checked 17 Sept 2026Rumelhart, Hinton & Williams
Assigning error to hidden units by the chain rule lets a multi-layer network discover its own internal representations instead of having them designed by hand.
link checked 17 Sept 2026Sepp Hochreiter & Jürgen Schmidhuber
A gated memory cell carries gradient across long intervals unchanged, which is what made recurrent networks trainable on sequences longer than a few dozen steps.
link checked 17 Sept 2026LeCun, Bottou, Bengio & Haffner
Introduces the convolutional neural network (LeNet-5) and argues that learning the feature extractor end to end from pixels, with weight sharing and pooling built into the architecture, beats hand-designed feature pipelines on document recognition.
46 pageslink checked 17 Sept 2026Glorot & Bengio
Traces the difficulty of training deep networks to how weight initialisation interacts with activation functions and layer-to-layer signal variance, and derives an initialisation scheme (Xavier initialisation) that keeps that variance stable across depth.
8 pageslink checked 17 Sept 2026Bengio, Courville & Vincent
Surveys the principles behind learning good data representations automatically rather than by hand-engineering features, and argues the quality of a representation can be judged by how well it disentangles the underlying explanatory factors of the data.
30 pageslink checked 17 Sept 2026Diederik P. Kingma & Jimmy Ba
Combines momentum with per-parameter adaptive learning rates, becoming the default optimiser for deep networks.
link checked 17 Sept 2026