1.4 Matrix Operations


: 40 minutes

Just like vectors, matrices have their own arithmetic. Addition and scalar multiplication behave exactly as you would guess. Multiplication does not, and the reason it does not is the most important idea of the week: a matrix is a function.

Transpose

Let \bold{A}\in\mathbb{R}^{m\times n} be a matrix. The transpose, denoted \bold{A}^T, is the n\times m matrix having the rows of \bold{A} as its columns and the columns of \bold{A} as its rows. Equivalently, it is the reflection of \bold{A} across its main diagonal.

In other words, if \bold{B}=\bold{A}^T, then b_{ij}=a_{ji} for all i and j.

\bold{A}=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix} \quad\Longrightarrow\quad \bold{A}^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix}.

A square matrix \bold{A} is called symmetric if \bold{A}^T=\bold{A} and skew-symmetric if \bold{A}^T=-\bold{A}. Transposing twice returns you to where you started: (\bold{A}^T)^T=\bold{A}.

In a data matrix, transposing swaps the two readings of the same table: records become features and features become records. This is why \bold{A}^T\bold{A} (an n\times n matrix of feature-against-feature dot products) is the object that appears whenever we ask a question about the columns of a dataset—correlations, covariances, and the normal equations of regression.

Note Skew-symmetry

Can the last diagonal element of a skew-symmetric matrix be -5?

For a skew-symmetric matrix, all the diagonal elements must be zero.

Addition (and Subtraction)

Two matrices can be added or subtracted only if their sizes match exactly—same number of rows and same number of columns. The addition is element-wise, and the result has that same size. \bold{U}+\bold{V} =\begin{bmatrix}u_{11}&u_{12}&\ldots&u_{1n}\\ u_{21}&u_{22}&\ldots&u_{2n}\\ \vdots&\vdots&\ddots&\vdots \\ u_{m1}&u_{m2}&\ldots&u_{mn}\end{bmatrix} + \begin{bmatrix}v_{11}&v_{12}&\ldots&v_{1n}\\ v_{21}&v_{22}&\ldots&v_{2n}\\ \vdots&\vdots&\ddots&\vdots \\ v_{m1}&v_{m2}&\ldots&v_{mn}\end{bmatrix} =\begin{bmatrix}u_{11}+v_{11}&u_{12}+v_{12}&\ldots&u_{1n}+v_{1n}\\ u_{21}+v_{21}&u_{22}+v_{22}&\ldots&u_{2n}+v_{2n}\\ \vdots&\vdots&\ddots&\vdots \\ u_{m1}+v_{m1}&u_{m2}+v_{m2}&\ldots&u_{mn}+v_{mn}\end{bmatrix} .

Multiplication by a Scalar

For a scalar k\in\mathbb{R}, multiplying a matrix \bold{U}\in\mathbb{R}^{m\times n} by k scales every entry by k. The outcome is denoted k\bold{U} and has the same size as \bold{U}. k\bold{U} =k\begin{bmatrix}u_{11}&u_{12}&\ldots&u_{1n}\\ u_{21}&u_{22}&\ldots&u_{2n}\\ \vdots&\vdots&\ddots&\vdots \\ u_{m1}&u_{m2}&\ldots&u_{mn}\end{bmatrix} =\begin{bmatrix}ku_{11}&ku_{12}&\ldots&ku_{1n}\\ ku_{21}&ku_{22}&\ldots&ku_{2n}\\ \vdots&\vdots&\ddots&\vdots \\ ku_{m1}&ku_{m2}&\ldots&ku_{mn}\end{bmatrix} .

Matrix-Matrix Multiplication

Matrix multiplication is defined only when the number of columns in the first matrix equals the number of rows in the second. For \bold{A}_{m \times n} and \bold{B}_{n \times p}, the product \bold{C} = \bold{AB} is an m \times p matrix whose entries are c_{ij} = \sum_{k=1}^{n} a_{ik} \cdot b_{kj}.

Read that formula as a dot product: c_{ij} is row i of \bold{A} dotted with column j of \bold{B}. The shared dimension n is exactly the length those two vectors must share, which is why the inner dimensions have to agree.

The shape rule. Write the sizes side by side, (m \times n)\,(n \times p). The two inner numbers must match, and they cancel; the two outer numbers survive as the shape of the answer: (m \times n)\,(n \times p) \;\longrightarrow\; m\times p. Doing this check before computing anything catches most mistakes for free.

Example

Let us now multiply the following two matrices.

\bold{A}= \begin{bmatrix} 2 & 3 & 1 \\ 4 & 0 & 5 \\ 1 & 2 & 3 \end{bmatrix}\text{ and } \bold{B}= \begin{bmatrix} 1 & 2 \\ 3 & 1 \\ 2 & 4 \end{bmatrix}.

The product \bold{C} = \bold{AB} is calculated as follows. For a 3 \times 3 matrix multiplied by a 3 \times 2 matrix, the result is a 3 \times 2 matrix.

Each element c_{ij} of the result matrix \bold{C} is computed as: c_{ij} = \sum_{k=1}^{3} a_{ik} \cdot b_{kj}

First row of \bold{C}: \begin{align} c_{11} &= (2)(1) + (3)(3) + (1)(2) = 2 + 9 + 2 = 13 \\ c_{12} &= (2)(2) + (3)(1) + (1)(4) = 4 + 3 + 4 = 11 \end{align}

Second row of \bold{C}: \begin{align} c_{21} &= (4)(1) + (0)(3) + (5)(2) = 4 + 0 + 10 = 14 \\ c_{22} &= (4)(2) + (0)(1) + (5)(4) = 8 + 0 + 20 = 28 \end{align}

Third row of \bold{C}: \begin{align} c_{31} &= (1)(1) + (2)(3) + (3)(2) = 1 + 6 + 6 = 13 \\ c_{32} &= (1)(2) + (2)(1) + (3)(4) = 2 + 2 + 12 = 16 \end{align}

Therefore, the product matrix is: \begin{equation} \bold{C} = \bold{AB} = \begin{bmatrix} 13 & 11 \\ 14 & 28 \\ 13 & 16 \end{bmatrix} \end{equation}

Properties (and One Non-Property)

For matrices of compatible sizes:

  • Associative: (\bold{AB})\bold{C}=\bold{A}(\bold{BC}).
  • Distributive: \bold{A}(\bold{B}+\bold{C})=\bold{AB}+\bold{AC}.
  • Identity: \bold{I}_m\bold{A}=\bold{A}\bold{I}_n=\bold{A} for \bold{A}\in\mathbb{R}^{m\times n}.
  • Transpose reverses: (\bold{AB})^T=\bold{B}^T\bold{A}^T.
  • Not commutative: in general \bold{AB}\neq\bold{BA}.

That last point deserves emphasis. Often \bold{BA} is not even defined: with \bold{A} of size 3\times2 and \bold{B} of size 2\times4, the product \bold{AB} is 3\times 4 while \bold{BA} pairs an inner 4 against an inner 3 and is illegal. Even when both products exist and have the same shape, they usually differ: \begin{bmatrix}0&1\\0&0\end{bmatrix}\begin{bmatrix}0&0\\1&0\end{bmatrix}=\begin{bmatrix}1&0\\0&0\end{bmatrix}, \qquad \begin{bmatrix}0&0\\1&0\end{bmatrix}\begin{bmatrix}0&1\\0&0\end{bmatrix}=\begin{bmatrix}0&0\\0&1\end{bmatrix}.

This is not a defect of the definition. Matrices are functions, and \bold{AB} means “do \bold{B}, then do \bold{A}”. Putting on socks and then shoes is not the same as putting on shoes and then socks.

Note Transpose of a Product

Let \bold{A} be 3\times 2 and \bold{B} be 2\times 5. Which expression is both defined and equal to (\bold{A}\bold{B})^T?

Shapes settle this before any arithmetic. The product \bold{A}\bold{B} is 3\times 5, so (\bold{A}\bold{B})^T is 5\times 3.

  • \bold{A}^T is 2\times3 and \bold{B}^T is 5\times2, so \bold{A}^T\bold{B}^T pairs an inner 3 with an inner 5—undefined.
  • \bold{B}^T\bold{A}^T is (5\times2)(2\times3)=5\times3, which matches, and indeed (\bold{A}\bold{B})^T=\bold{B}^T\bold{A}^T always.
  • \bold{A}\bold{B} is 3\times5, the wrong shape, and \bold{B}\bold{A} is undefined (5 against 3).

The reversal is not an accident of notation. Transposing swaps the roles of rows and columns, and the entry c_{ij} of \bold{A}\bold{B} pairs row i of \bold{A} with column j of \bold{B}; after transposing, that same number must pair row j of \bold{B}^T with column i of \bold{A}^T, which is exactly the (j,i) entry of \bold{B}^T\bold{A}^T.

Linear Transformation

A transformation (function or mapping) T : \mathbb{R}^n \rightarrow \mathbb{R}^m denoted T from \mathbb{R}^n to \mathbb{R}^m is a rule that assigns to each vector \bold{x} \in \mathbb{R}^n a vector T(\bold{x}) \in \mathbb{R}^m.

Every matrix \bold{A}\in\mathbb{R}^{m\times n} is such a transformation, acting by multiplication:

T(\bold{x}) = A\bold{x}

The set \mathbb{R}^n is called the domain of T, and the set \mathbb{R}^m is called the codomain of T.

It is linear, meaning it respects the two vector operations of 1.2: \bold{A}(\bold{x}+\bold{y})=\bold{A}\bold{x}+\bold{A}\bold{y}, \qquad \bold{A}(c\bold{x})=c\,\bold{A}\bold{x}. Consequently \bold{A} sends linear combinations to linear combinations, straight lines to straight lines, and the origin to the origin. This is the single most useful sentence in the module: from week 1 through the regression session on Nov 3, “fitting a model” will mean choosing a linear map, and “the model cannot represent this pattern” will mean the pattern is not in the reach of any linear map.

For a matrix A with dimensions m\times n, the matrix-vector product is computed by

A\bold{x} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} = \begin{bmatrix} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n \\ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n \\ \vdots \\ a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n \end{bmatrix} = \bold{b}

A\bold{x} = \bold{b}

To perform the matrix multiplication A\bold{x}, the number of columns in A must match the number of entries in the vector \bold{x}. That is, if A is an m \times n matrix and \bold{x} is an n \times 1 column vector, the multiplication is valid and the result will be an m \times 1 column vector. If the dimensions do not align, the product is undefined.

A\bold{x} Is a Combination of the Columns

The formula above computes \bold{b}=A\bold{x} one output entry at a time, each entry a dot product of a row of A with \bold{x}. There is a second way to read exactly the same arithmetic, and it is the one worth carrying forward. Write \bold{a}_1,\ldots,\bold{a}_n for the columns of A. Then

A\bold{x} =\begin{bmatrix}\bold{a}_1 & \bold{a}_2 & \cdots & \bold{a}_n\end{bmatrix} \begin{bmatrix}x_1\\x_2\\\vdots\\x_n\end{bmatrix} = x_1\bold{a}_1 + x_2\bold{a}_2+\cdots+x_n\bold{a}_n.

In words: \bold{x} is a recipe, and A\bold{x} is a linear combination of the columns of A with \bold{x} supplying the coefficients. Two consequences follow immediately.

  • The set of all vectors \bold{b} you can possibly reach, as \bold{x} ranges over \mathbb{R}^n, is the span of the columns of A. It is called the column space of A.
  • Therefore A\bold{x}=\bold{b} has a solution if and only if \bold{b} lies in the column space.

That “if and only if” is where the course goes next. Real data almost never puts \bold{b} exactly in the column space, so A\bold{x}=\bold{b} has no solution; 3.2 Projection and Orthogonality then asks for the point of the column space closest to \bold{b}, and the Nov 3 regression session identifies that point as the fitted values of a linear regression. Everything rests on the picture in this subsection.

A Matrix Is Determined by Where It Sends the Basis

Take \bold{x}=\bold{e}_1=[1,0,\ldots,0]^T in the column formula. Every coefficient vanishes except the first, so A\bold{e}_1=\bold{a}_1. In general A\bold{e}_j is simply the j^{\text{th}} column of A.

So a matrix carries no information beyond the images of the standard basis vectors: decide where \bold{e}_1 and \bold{e}_2 should land, write those two images as columns, and you have built the matrix. The interactive figure below makes this concrete—watch A\bold{e}_1 and A\bold{e}_2 in the codomain as you drag the entries of A.

Determinant of a 2×2 Matrix

For a square matrix \bold{A} = \begin{bmatrix} a & b \\ c & d \end{bmatrix}, the determinant is the single number \det(\bold{A}) = ad - bc.

Its meaning is geometric. The map \bold{x}\mapsto\bold{A}\bold{x} takes the unit square—spanned by \bold{e}_1 and \bold{e}_2—to the parallelogram spanned by the columns \bold{a}_1 and \bold{a}_2, and |\det(\bold{A})| is the area of that parallelogram. So \det(\bold{A}) is the factor by which \bold{A} scales area, and its sign records whether \bold{A} flips orientation.

The extreme case carries all the weight:

\det(\bold{A})=0 \iff \text{the two columns are parallel} \iff \text{$\bold{A}$ flattens the plane onto a line (or a point).}

Once the plane has been flattened, distinct inputs share an output and no rule can undo the map. A matrix with \det(\bold{A})=0 is called singular.

Determinants exist for every n\times n matrix, and |\det| is then a volume scale factor. We only need the 2\times2 formula here; the 3\times3 case appears in the practice problems.

Inverse of a 2×2 Matrix

The inverse of a matrix is the “undo” button for its linear transformation. \bold{A}^{-1} is the matrix satisfying \bold{A}^{-1}\bold{A}=\bold{A}\bold{A}^{-1}=\bold{I}, so that applying \bold{A} and then \bold{A}^{-1} returns every vector to where it started. For a 2\times2 matrix,

\bold{A}^{-1} = \frac{1}{\det(\bold{A})} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}.

The formula divides by \det(\bold{A}), so it works precisely when \det(\bold{A}) \neq 0—the algebra and the geometry of the previous section agree, as they must. A matrix is invertible exactly when it is not singular.

An inverse also solves systems: if \bold{A} is invertible, then \bold{A}\bold{x}=\bold{b} has the unique solution \bold{x}=\bold{A}^{-1}\bold{b}, for every \bold{b}. (In practice we solve the system directly rather than forming \bold{A}^{-1}, which is slower and numerically worse—but the statement is what matters here.)

Note Zero Determinant

A 2\times2 matrix \bold{A} has \det(\bold{A})=0. Which statement must be true?

Read \det(\bold{A}) geometrically: it is the (signed) factor by which \bold{A} scales area. A determinant of 0 means the unit square is flattened to zero area—the image of the whole plane is a line through the origin, or just the origin itself.

Once two different inputs land on the same output, no rule can send that output back to both of them, so no matrix \bold{A}^{-1} can undo \bold{A}. This is the same statement as the algebraic one: the formula \bold{A}^{-1}=\frac{1}{\det(\bold{A})}\begin{bmatrix} d & -b \\ -c & a\end{bmatrix} divides by \det(\bold{A}).

The other options fail. \bold{A}=\begin{bmatrix}1&1\\1&1\end{bmatrix} has determinant 0 without being the zero matrix. Transposing never changes the determinant, so \det(\bold{A}^T)=0 too. And since the image is only a line, most vectors \bold{b} are unreachable, so \bold{A}\bold{x}=\bold{b} has no solution for those \bold{b}.

Exercises

Note Shapes Before Arithmetic

A is 3\times2 and B is 2\times4. Answer both questions without computing any product—write the sizes side by side and apply the shape rule.

AB_shape   = (3, 4)
BA_defined = False
Note A\bold{x} the Other Way Round

matvec below computes A\bold{x} the usual way: one output entry per row of A. Recompute the same vector using the column view of the section above—as x_1\bold{a}_1+x_2\bold{a}_2+x_3\bold{a}_3, a linear combination of the columns. The two answers must agree.

b = [x[0] * c0 + x[1] * c1 + x[2] * c2
     for c0, c1, c2 in zip(cols[0], cols[1], cols[2])]
Note Where Do the Basis Vectors Go?

A 2\times2 matrix is completely determined by the images of \bold{e}_1=[1,0] and \bold{e}_2=[0,1], which are exactly its two columns.

Use that to build the matrix \bold{R} that rotates every vector 90^\circ counterclockwise. Do not look up a rotation formula: draw the two basis vectors, rotate each a quarter turn, and read off where they land.

R = [[0, -1],
     [1,  0]]

The columns are the images \bold{R}\bold{e}_1=[0,1] (east turns to north) and \bold{R}\bold{e}_2=[-1,0] (north turns to west).

Note Invert It, or Refuse

Write inv2(A) for a 2\times2 matrix: return the inverse as a list of lists, or return None when the inverse does not exist. Decide which case you are in before dividing by anything.

def inv2(A):
    (a, b), (c, d) = A
    det = a * d - b * c
    if det == 0:
        return None
    return [[d / det, -b / det], [-c / det, a / det]]