1.5 Practice Problems

Work these with pen and paper first. Where a problem asks you to explain or justify, a sentence in words is the answer we are after—the arithmetic is the easy part.

Vectors, Norms, and the Dot Product

Exercise 5.1 Let \bold{u}=[1,-2,2] and \bold{v}=[3,0,4]. Compute \langle\bold{u},\bold{v}\rangle, \|\bold{u}\|, \|\bold{v}\|, and \cos\theta for the angle \theta between them.

Solution. \langle\bold{u},\bold{v}\rangle=3+0+8=11; \|\bold{u}\|=\sqrt{1+4+4}=3; \|\bold{v}\|=\sqrt{9+0+16}=5; so \cos\theta=11/15\approx0.733, an acute angle of about 42.8^\circ.

Exercise 5.2 Find all scalars t for which [1,2] and [t,-3] are orthogonal. Then find all t for which they are parallel.

Solution. Orthogonal means t-6=0, so t=6 and nothing else. Parallel means [t,-3]=c[1,2] for some scalar c; the second coordinate forces c=-3/2, so t=-3/2. Notice that the two conditions each pin down a single t, and the two values differ—orthogonality and parallelism are the two extremes of the same one-parameter family.

Exercise 5.3 Show that \|\bold{u}\|=0 if and only if \bold{u}=\bold{0}. Where in the argument do you use the fact that the entries are real numbers?

Solution. \|\bold{u}\|^2=\sum_i u_i^2 is a sum of squares of real numbers, hence a sum of non-negative terms. A sum of non-negative terms is zero only when every term is zero, so u_i^2=0 and therefore u_i=0 for every i. The realness is exactly what guarantees u_i^2\ge 0; over the complex numbers u_i^2 can be negative, which is why the complex inner product uses |u_i|^2 instead.

Exercise 5.4 A student’s marks on four components are \bold{s}=[90, 80, 70, 60].

  1. Compute the weighted sum \langle\bold{s},\bold{w}\rangle for \bold{w}=[0.4,0.3,0.2,0.1].
  2. Now use \bold{w}'=2\bold{w}. What happens to the weighted sum? Is either result a weighted average?

Solution.

  1. 36+24+14+6=80. The weights sum to 1, so this is a genuine weighted average.
  2. Doubling the weights doubles the dot product to 160. Since \sum_i w'_i=2\neq1, that number is a weighted sum but not an average. A dot product is always a weighted sum; it is an average only under the extra condition \sum_i w_i=1.

Exercise 5.5 Let \bold{u} be a non-zero vector. Show that \hat{\bold{u}}=\bold{u}/\|\bold{u}\| has norm 1, and that \langle\hat{\bold{u}},\hat{\bold{v}}\rangle equals \cos\theta for the angle \theta between \bold{u} and \bold{v}.

Exercise 5.6 Two vectors \bold{u},\bold{v} satisfy \|\bold{u}\|=\|\bold{v}\|. Show that \bold{u}+\bold{v} and \bold{u}-\bold{v} are orthogonal, and say what this means about the diagonals of a rhombus.

Solution. Expand the dot product using bilinearity: \langle\bold{u}+\bold{v},\,\bold{u}-\bold{v}\rangle=\langle\bold{u},\bold{u}\rangle-\langle\bold{u},\bold{v}\rangle+\langle\bold{v},\bold{u}\rangle-\langle\bold{v},\bold{v}\rangle=\|\bold{u}\|^2-\|\bold{v}\|^2=0. Geometrically \bold{u}+\bold{v} and \bold{u}-\bold{v} are the two diagonals of the parallelogram built on \bold{u} and \bold{v}. Equal side lengths make it a rhombus, and we have just shown its diagonals meet at right angles.

Exercise 5.7 Prove the Pythagorean identity: if \bold{u}\perp\bold{v}, then \|\bold{u}+\bold{v}\|^2=\|\bold{u}\|^2+\|\bold{v}\|^2. Give a 2-dimensional example showing the identity fails when \bold{u} and \bold{v} are not orthogonal.

Solution. \|\bold{u}+\bold{v}\|^2=\langle\bold{u}+\bold{v},\bold{u}+\bold{v}\rangle=\|\bold{u}\|^2+2\langle\bold{u},\bold{v}\rangle+\|\bold{v}\|^2, and the cross term vanishes exactly when \bold{u}\perp\bold{v}. For a counterexample take \bold{u}=\bold{v}=[1,0]: the left side is 4, the right side is 2. This identity is why the “sum of squared errors” of a regression splits cleanly into explained and unexplained parts—a fact we cash in during 3.2.

Exercise 5.8 Let \bold{a}=[1,1,1] and \bold{b}=[2,6,7]. Find the scalar \hat{x} making \|\bold{b}-\hat{x}\bold{a}\| as small as possible, using only the requirement that the error \bold{b}-\hat{x}\bold{a} be orthogonal to \bold{a}. What familiar statistic is \hat{x}?

Solution. Orthogonality gives \langle\bold{a},\bold{b}-\hat{x}\bold{a}\rangle=0, hence \hat{x}=\langle\bold{a},\bold{b}\rangle/\langle\bold{a},\bold{a}\rangle=15/3=5: the mean of the entries of \bold{b}. Averaging is a projection onto the all-ones direction. This problem is 3.2 in miniature.

Matrices

Exercise 5.9 Can two matrices be always added?

Solution. No, the matrices must have the same size, i.e., the same number of rows and columns.

Exercise 5.10 If addition is possible, add the following matrices: \bold{A}=\begin{bmatrix} -1 & 2 & 3\\ 0 & 1 & 5 \end{bmatrix} \text{ and } \bold{B}=\begin{bmatrix} 1/2 & 1 & 3\\ 2 & -2 & 0 \end{bmatrix}.

Solution. The matrix addition is possible since both the matrices have 2 rows and 3 columns.

\bold{A}+\bold{B}=\begin{bmatrix} -1 + 1/2 & 2 + 1 & 3 + 3\\ 0 + 2 & 1 + (-2) & 5 + 0 \end{bmatrix} = \begin{bmatrix} -1/2 & 3 & 6\\ 2 & -1 & 5 \end{bmatrix}.

Exercise 5.11 For matrices \bold{A},\bold{B}, is it always true that \bold{A}+\bold{B}=\bold{B}+\bold{A}? Explain your answer.

Solution. Whenever the sum is defined, yes. Entry by entry the claim reads a_{ij}+b_{ij}=b_{ij}+a_{ij}, which is just commutativity of addition of real numbers. Contrast this with multiplication, where no such entry-by-entry shortcut exists.

Exercise 5.12 If multiplication is possible, multiply the following matrices: \bold{A}=\begin{bmatrix} 1 & 2 & 3\\ 0 & -1 & 5 \end{bmatrix} \text{ and } \bold{B}=\begin{bmatrix} 1 & 2 & 1\\ 1 & 0 & 2 \end{bmatrix}.

Solution. It is not possible. Lining up the sizes gives (2\times3)(2\times3): the inner dimensions 3 and 2 disagree. Always run this check before computing anything. (The product \bold{A}\bold{B}^T is defined, and is 2\times2.)

Exercise 5.13 If multiplication is possible, multiply the following matrices: \bold{A}=\begin{bmatrix} 1 & 2\\ 0 & -1\\ -1 & 3 \end{bmatrix} \text{ and } \bold{B}=\begin{bmatrix} 1 & 5 & 1\\ 1 & 12 & 2 \end{bmatrix}.

Solution. Sizes: (3\times2)(2\times3), so the product is 3\times3. \bold{AB}=\begin{bmatrix} 3 & 29 & 5\\ -1 & -12 & -2\\ 2 & 31 & 5 \end{bmatrix}. Compute \bold{BA} as well: it is 2\times2, so the two products are not even the same shape.

Exercise 5.14 For matrices \bold{A} and \bold{B}, is it always true that \bold{AB}=\bold{BA}? Explain your answer.

Solution. No, and it fails in three escalating ways. Often \bold{BA} is undefined; even when it is defined it may have a different shape (previous problem); and even for square matrices of the same size the two products usually differ, as with \begin{bmatrix}0&1\\0&0\end{bmatrix}\begin{bmatrix}0&0\\1&0\end{bmatrix}=\begin{bmatrix}1&0\\0&0\end{bmatrix} versus \begin{bmatrix}0&0\\1&0\end{bmatrix}\begin{bmatrix}0&1\\0&0\end{bmatrix}=\begin{bmatrix}0&0\\0&1\end{bmatrix}. The reason is that \bold{AB} means “apply \bold{B}, then apply \bold{A}”, and the order in which you do two things generally matters.

Exercise 5.15 Give an example of a 4\times 2 diagonal matrix.

Solution. “Diagonal” only requires a_{ij}=0 for i\neq j; it does not require squareness. One example is \begin{bmatrix}3&0\\0&-7\\0&0\\0&0\end{bmatrix}. As a map \mathbb{R}^2\to\mathbb{R}^4 it stretches the two input directions and pads the answer with zeros.

Exercise 5.16 Intuitively justify the following fact: (\bold{AB})^T=\bold{B}^T\bold{A}^T.

Solution. Start with shapes: if \bold{A} is m\times n and \bold{B} is n\times p, then (\bold{AB})^T is p\times m, and \bold{B}^T\bold{A}^T is (p\times n)(n\times m)=p\times m, while \bold{A}^T\bold{B}^T is not even defined in general. So the reversed order is the only candidate.

For the content: the (i,j) entry of (\bold{AB})^T is the (j,i) entry of \bold{AB}, namely row j of \bold{A} dotted with column i of \bold{B}. Transposing turns that row of \bold{A} into column j of \bold{A}^T and that column of \bold{B} into row i of \bold{B}^T—which is precisely the (i,j) entry of \bold{B}^T\bold{A}^T.

Exercise 5.17 Is the sum of two symmetric matrices always symmetric?

Solution. Yes. (\bold{A}+\bold{B})^T=\bold{A}^T+\bold{B}^T=\bold{A}+\bold{B}. Compare with the next problem, where the analogous claim for products fails—transposition distributes over sums but reverses products.

Exercise 5.18 Is the product of two symmetric matrices always symmetric?

Solution. The answer is no. For symmetric matrices A and B, the product is symmetric iff \bold{AB}=\bold{BA}. (Hint: (\bold{AB})^T=\bold{B}^T\bold{A}^T)

Exercise 5.19 Give an example of a skew-symmetric matrix of order 4\times 4 with at least 4 non-zero elements.

Solution. (Hint: What does the diagonal of a skew symmetric matrix look like?)

Exercise 5.20 For any matrix \bold{A}, explain why

  1. \bold{A}+\bold{A}^T is symmetric.

  2. \bold{A}-\bold{A}^T is skew-symmetric.

Solution.

  1. Putting \bold{B}=\bold{A}+\bold{A}^T, we have \bold{B}^T=(\bold{A}+\bold{A}^T)^T=\bold{A}^T+(\bold{A}^T)^T = \bold{A}^T+\bold{A}=\bold{B}
  2. Similar

Exercise 5.21 If \bold{A} has an inverse, then show that the inverse is unique.

Exercise 5.22 If \bold{A} is the inverse of \bold{B}, then show that \bold{B} is the inverse of \bold{A}.

Exercise 5.23  

  1. Is \text{det} (\bold{A}+\bold{B})=\text{det} (\bold{A})+ \text{det} (\bold{B})?
  2. Is \text{det} (\bold{AB})=\text{det} (\bold{A}) \text{det} (\bold{B})?

Solution.

  1. the 2 \times 2 case can be checked directly. How do you prove the n \times n case?

Exercise 5.24 Suppose \bold{A} is a 2 \times 2 matrix and \bold{B} is the matrix we get by interchanging the two rows of \bold{A}. Show that \text{det} \bold{B} = - \text{det} \bold{A}.

Exercise 5.25 Find the determinant of \begin{bmatrix}-1 & 2 & 3\\ 0 & 1 & 5\\ 4 & 1 & -1\end{bmatrix}.

Solution. Expanding along the first row, (-1)\bigl(1\cdot(-1)-5\cdot1\bigr)-2\bigl(0\cdot(-1)-5\cdot4\bigr)+3\bigl(0\cdot1-1\cdot4\bigr)=6+40-12=34. Since 34\neq0, this matrix is invertible and scales volume in \mathbb{R}^3 by a factor of 34.

Exercise 5.26 Find a 3\times 3 matrix whose determinant is 6 and the absolute value of each entry is no bigger than 3.

Solution. (Hint: what is the determinant of a (upper) triangular matrix?)

Exercise 5.27 \begin{bmatrix} 2 & 1 & 3 \end{bmatrix}\begin{bmatrix} -1 & 0 & -1\\ -1 & 1 & 0\\ 0 & 1 & 1 \end{bmatrix}\begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}=?

Before computing, predict the size of the answer from the shapes alone. Then compute it twice, grouping first as (\bold{u}^T\bold{M})\bold{v} and then as \bold{u}^T(\bold{M}\bold{v}).

Solution. The shapes are (1\times3)(3\times3)(3\times1), so the result is 1\times1: a single number.

Grouping to the left, \bold{u}^T\bold{M}=[-3,\;4,\;1], and dotting with [1,0,-1]^T gives -3+0-1=-4.

Grouping to the right, \bold{M}\bold{v}=[0,-1,-1]^T, and [2,1,3] dotted with that is 0-1-3=-4. The two agree because matrix multiplication is associative—and the second grouping is far less work, which is exactly why the order of operations matters for efficiency even though it never changes the answer.

Exercise 5.28 Find the inverse of the following matrix: \bold{A}=\begin{bmatrix} 1 & 2 & 3\\ 0 & -1 & 2\\ 1 & -2 & 0 \end{bmatrix}.

Solution. \det(\bold{A})=4+4+3=11\neq0, so the inverse exists, and \bold{A}^{-1}=\frac{1}{11}\begin{bmatrix} 4 & -6 & 7\\ 2 & -3 & -2\\ 1 & 4 & -1 \end{bmatrix}. Always verify by multiplying out: \bold{A}\bold{A}^{-1} should come back as \bold{I}_3.

Exercise 5.29 Find the inverse of the following matrix, or explain why it has none: \begin{bmatrix} -1 & 0 & -1\\ -1 & 1 & 0\\ 0 & 1 & 1 \end{bmatrix}.

Solution. The determinant is 0, so there is no inverse. The reason is visible in the columns: the third column is the sum of the first two. Every product \bold{M}\bold{x} is a combination of the columns, so the third column adds nothing new and the whole image lies in the plane spanned by the first two—the map flattens \mathbb{R}^3 onto a plane and cannot be undone.

Exercise 5.30 If \bold{A}=\begin{bmatrix} 5 & 3\\ -1 & -2\\ \end{bmatrix}, then show that A satisfies the equation A^2-3A-7I=0.

Solution. (Hint: try to factor the LHS)

Exercise 5.31 Consider a n \times n matrix \bold{A}. Then:

  1. If \bold{A} is invertible and \bold{AB}=0 for some n \times n matrix \bold{B}, then \bold{B}=0.
  2. If \bold{A} is not invertible, then we can find a n \times n matrix \bold{B} with \bold{AB}=0 but \bold{B} \ne 0.

Solution.

  1. Multiply \bold{AB}=\bold{0} on the left by \bold{A}^{-1}: \bold{A}^{-1}\bold{A}\bold{B}=\bold{A}^{-1}\bold{0}, i.e. \bold{B}=\bold{0}.
  2. For any non-invertible 2 \times 2 matrix \bold{A}= \begin{bmatrix} a & b\\ c & d \end{bmatrix}, where a,b,c,d are not all 0, you can choose \bold{B}= \begin{bmatrix} -b & -d\\ a & c \end{bmatrix}. How do you do the general case?

Matrices as Linear Maps

These problems rehearse the idea the module is built on. Sketching the picture is usually faster than computing.

Exercise 5.32 Write down the 2\times2 matrix of each transformation of the plane, by asking only where \bold{e}_1=[1,0] and \bold{e}_2=[0,1] land.

  1. Reflection across the horizontal axis.
  2. Scaling by 3 in the horizontal direction and leaving the vertical direction alone.
  3. Rotation by 180^\circ.
  4. Projection onto the horizontal axis (send every point straight down to that axis).

Solution. The images of \bold{e}_1 and \bold{e}_2 are literally the columns: \begin{bmatrix}1&0\\0&-1\end{bmatrix},\quad \begin{bmatrix}3&0\\0&1\end{bmatrix},\quad \begin{bmatrix}-1&0\\0&-1\end{bmatrix},\quad \begin{bmatrix}1&0\\0&0\end{bmatrix}. Their determinants are -1, 3, 1, and 0. Read them: the reflection preserves area but flips orientation; the scaling triples area; the rotation changes nothing; the projection destroys area entirely, which is precisely why it cannot be undone.

Exercise 5.33 Let \bold{A}=\begin{bmatrix}1&2\\2&4\end{bmatrix}.

  1. Compute \det(\bold{A}).
  2. Describe the set of all vectors \bold{b} for which \bold{A}\bold{x}=\bold{b} has a solution.
  3. Find a non-zero \bold{x} with \bold{A}\bold{x}=\bold{0}.

Solution.

  1. \det(\bold{A})=1\cdot4-2\cdot2=0.
  2. The columns [1,2] and [2,4] are parallel, so every \bold{A}\bold{x} is a multiple of [1,2]. Solutions exist exactly for \bold{b} on that line and for no other \bold{b}.
  3. \bold{x}=[2,-1] works, as does any multiple of it. A whole line of inputs is crushed to the origin, which is another way of saying the map is not reversible.

Exercise 5.34 A transformation T:\mathbb{R}^2\to\mathbb{R}^2 is defined by T(\bold{x})=\bold{A}\bold{x}+\bold{c} for a fixed non-zero vector \bold{c}. Show that T is not linear. Which of the two defining properties fails, and what is the simplest input that exposes the failure?

Solution. Both properties fail, and the quickest witness is \bold{x}=\bold{0}: a linear map must satisfy T(\bold{0})=\bold{0}, but here T(\bold{0})=\bold{c}\neq\bold{0}. Concretely, T(\bold{x}+\bold{y})=\bold{A}\bold{x}+\bold{A}\bold{y}+\bold{c} while T(\bold{x})+T(\bold{y})=\bold{A}\bold{x}+\bold{A}\bold{y}+2\bold{c}.

Such a map is called affine. Linear regression with an intercept is affine in this sense, which is why we handle the intercept by appending a column of ones to the data matrix—restoring linearity by enlarging the matrix.

Exercise 5.35 Let \bold{X}\in\mathbb{R}^{100\times4} be a data matrix (100 records, 4 features) and let \bold{w}\in\mathbb{R}^4 be a weight vector.

  1. What is the shape of \bold{X}\bold{w}, and what does each of its entries mean?
  2. What is the shape of \bold{X}^T\bold{X}, and what does its (i,j) entry mean?
  3. Is \bold{X}^T\bold{X} symmetric? Justify it in one line.

Solution.

  1. (100\times4)(4\times1)=100\times1: one number per record, namely that record’s features dotted with \bold{w}. This is the vector of predictions of a linear model.
  2. (4\times100)(100\times4)=4\times4. Its (i,j) entry is the dot product of feature column i with feature column j—a feature-against-feature summary, with no trace of individual records left.
  3. Yes: (\bold{X}^T\bold{X})^T=\bold{X}^T(\bold{X}^T)^T=\bold{X}^T\bold{X}. This matrix is the one that must be inverted in the normal equations of regression.

Exercise 5.36 In plain Python, M = [[0] * 3] * 3 looks like a 3\times3 zero matrix. Setting M[0][0] = 1 then changes three entries instead of one. Explain what happened, and write a correct expression for a 3\times3 zero matrix.

Solution. [0] * 3 builds one row; the outer * 3 then stores the same row object three times, so the three rows are aliases of a single list. Mutating one mutates all.

A correct construction builds a fresh row each time: M = [[0] * 3 for _ in range(3)]. This class of bug disappears with NumPy, where a shape is a real property of a real array rather than an accident of nesting.

The math tree