Essay2019
The Bitter Lesson
Richard Sutton
General methods that leverage computation consistently beat hand-engineered human knowledge, and researchers keep relearning this.
≈1,100 wordslink checked 17 Sept 2026Machine Learning Foundations
The statistical learning theory underneath everything else here.
15 topics · 21 curated works
Topics
- 01Foundations & Overviews4
- 02Supervised Learning1
- 03Unsupervised Learning1
- 04Bias-Variance Tradeoff1
- 05Overfitting & Regularisation1
- 06Cross-Validation1
- 07Feature Engineering1
- 08Linear & Logistic Regression1
- 09Decision Trees & Random Forests1
- 10Gradient Boosting2
- 11Support Vector Machines1
- 12Clustering Methods2
- 13Dimensionality Reduction2
- 14Statistical Learning Theory1
- 15Evaluation Metrics1
Reading in Machine Learning Foundations
21A way in
Start here
No prior grounding assumed.
The Bitter Lesson
Richard Sutton · 2019
General methods that leverage computation consistently beat hand-engineered human knowledge, and researchers keep relearning this.
+3 more at this level
Then
Assumes you know the vocabulary.
A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise
Ester, Kriegel, Sander & Xu · 1996
Defines clusters by local point density rather than distance to a centroid, letting DBSCAN find arbitrarily shaped clusters and label sparse regions…
+5 more at this level
Go deeper
Primary sources and full treatments.
Some Methods for Classification and Analysis of Multivariate Observations
James MacQueen · 1967
Introduces the k-means algorithm as a way of partitioning observations to minimise within-cluster variance, the procedure nearly every later…
+10 more at this level
12 of 21 works
Paper2003
A Tutorial on Principal Component Analysis
Jonathon Shlens
Derives PCA from first principles as a change of basis that maximises variance and decorrelates the data, building the linear-algebra intuition that most treatments assume the reader already has.
link checked 17 Sept 2026Essay2009
The Unreasonable Effectiveness of Data
Halevy, Norvig & Pereira
For messy human problems such as language, a simple model over far more data beats an elegant model over less, so effort is better spent on the corpus.
link checked 17 Sept 2026Paper2012
A Few Useful Things to Know about Machine Learning
Pedro Domingos
Twelve lessons practitioners otherwise learn the hard way: generalisation is the only goal, more data beats a cleverer algorithm, and intuition fails in high dimensions.
link checked 17 Sept 2026Paper1996
A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise
Ester, Kriegel, Sander & Xu
Defines clusters by local point density rather than distance to a centroid, letting DBSCAN find arbitrarily shaped clusters and label sparse regions as noise instead of forcing every point into a group.
link checked 17 Sept 2026FreeIntermediate
Paper1998
A Tutorial on Support Vector Machines for Pattern Recognition
Christopher J. C. Burges
Walks from linear maximum-margin classifiers through the kernel trick to soft-margin SVMs, building the geometric intuition that the original optimisation papers leave implicit.
link checked 17 Sept 2026FreeIntermediate
Paper2006
The Relationship Between Precision-Recall and ROC Curves
Davis & Goadrich
Proves that a curve dominates in ROC space if and only if it dominates in precision-recall space, and argues precision-recall is the more informative view whenever the classes are heavily imbalanced.
link checked 17 Sept 2026FreeIntermediate
Paper2010
A Survey of Cross-Validation Procedures for Model Selection
Arlot & Celisse
Surveys what cross-validation actually estimates and when k-fold, leave-one-out and bootstrap variants disagree, rather than treating cross-validation as one interchangeable procedure.
link checked 17 Sept 2026FreeIntermediate
Paper2015
Deep Feature Synthesis: Towards Automating Data Science Endeavors
Kanter & Veeramachaneni
Proposes automatically generating candidate features from relational data by stacking mathematical operations across table relationships, competing directly with hand-engineered features in real data science competitions.
link checked 17 Sept 2026FreeIntermediate
Paper2016
XGBoost: A Scalable Tree Boosting System
Chen & Guestrin
Describes the sparsity-aware split finding, weighted quantile sketch and system-level engineering that made gradient boosting fast enough to dominate structured-data competitions.
link checked 17 Sept 2026FreeIntermediate
Paper1967
Some Methods for Classification and Analysis of Multivariate Observations
James MacQueen
Introduces the k-means algorithm as a way of partitioning observations to minimise within-cluster variance, the procedure nearly every later clustering method is compared against.
link checked 17 Sept 2026Paper1984
A Theory of the Learnable
Leslie G. Valiant
Formalises what it means for an algorithm to learn a concept from examples in polynomial time, founding the PAC-learning framework that later supervised-learning theory is built on.
link checked 17 Sept 2026In order written
1967 – 2019- 1984A Theory of the LearnableLeslie G. Valiant
- 1996A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with NoiseEster, Kriegel, Sander & Xu
- 1998A Tutorial on Support Vector Machines for Pattern RecognitionChristopher J. C. Burges
- 2000A Unified Bias-Variance Decomposition for Zero-One and Squared LossPedro Domingos
- 2000Nonlinear Dimensionality Reduction by Locally Linear EmbeddingRoweis & Saul
- 2001Random ForestsLeo Breiman
- 2001Greedy Function Approximation: A Gradient Boosting MachineJerome H. Friedman
- 2001The Elements of Statistical LearningHastie, Tibshirani & Friedman
- 2003A Tutorial on Principal Component AnalysisJonathon Shlens
- 2006The Relationship Between Precision-Recall and ROC CurvesDavis & Goadrich
- 2009The Unreasonable Effectiveness of DataHalevy, Norvig & Pereira
- 2010A Survey of Cross-Validation Procedures for Model SelectionArlot & Celisse
- 2011An Analysis of Single-Layer Networks in Unsupervised Feature LearningCoates, Lee & Ng
- 2012A Few Useful Things to Know about Machine LearningPedro Domingos
- 2015Deep Feature Synthesis: Towards Automating Data Science EndeavorsKanter & Veeramachaneni
- 2016XGBoost: A Scalable Tree Boosting SystemChen & Guestrin
- 2017Understanding Deep Learning Requires Rethinking GeneralizationZhang et al.
- 2019The Bitter LessonRichard Sutton
- 2019Stanford CS229: Machine LearningAnand Avati (Stanford Online)
Also covered elsewhere
This subject genuinely sits in more than one domain. These fields approach the same ground with different methods.