Linear Algebra in Numpy

“Genius is one percent inspiration and ninety-nine percent perspiration.” – Thomas Edison

Week 2 taught us how to store and move numbers: shapes, axes, broadcasting, universal functions. This week we start computing with them.

Almost every model in the second half of this course is a linear-algebra statement wearing a statistical costume. Fitting a regression is projecting a vector onto a subspace. Regularizing that regression is shrinking along directions the data barely explored. Reducing dimension before clustering is throwing away the smallest singular values. NumPy’s linalg module is where all of these statements are actually evaluated, and the three chapters below are the vocabulary.

TipChapter Objectives

In this chapter, we will learn the following:

  1. Solving linear systems the right way: solve instead of inv, and why the distinction is not pedantry

  2. The QR and eigenvalue decompositions, and the condition number as a measure of how much a matrix amplifies error

  3. Orthogonality and projection: the normal equations, the projection matrix, and least squares as a projection onto a column space

  4. The SVD, optimal low-rank approximation, and PCA as the SVD of a centred data matrix

ImportantThis week is load-bearing

Nothing here is a detour. Three later lectures are promissory notes written against this week:

When What comes back
Nov 3—regression and ridge Least squares is projection onto the column space (3.2). The SVD explains exactly what ridge shrinks and why (3.3). Multicollinearity is a large condition number (3.1).
Nov 17—k-nearest neighbours The curse of dimensionality is distance concentration, which is a statement about norms and inner products in \mathbb{R}^d.
Dec 8—clustering PCA runs before k-means so that Euclidean distance means something once the noise directions are gone.

If a definition here feels abstract, the honest answer is: it will not be in five weeks. Learn it now while it is cheap.