1.2 Vector Operations


: 20 minutes

Vector operations are the building blocks of every data transformation we will write this semester. Each one has a formula you can compute by hand and a picture that tells you what it does—and the picture is the part worth keeping.

Vector Addition

Two vectors can be added only when they are of the same length. The addition is element-wise, and the result has that same length. \bold{u}+\bold{v} =\begin{bmatrix}u_1\\u_2\\\vdots \\ u_n\end{bmatrix}+\begin{bmatrix}v_1\\v_2\\\vdots \\v_n\end{bmatrix}=\begin{bmatrix}u_1+v_1\\u_2+v_2\\\vdots \\u_n+v_n\end{bmatrix}.

Adding vectors places them head-to-tail, and the result is the diagonal of the parallelogram they form.

\bold{u} + \bold{v} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} + \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} u_1 + v_1 \\ u_2 + v_2 \end{bmatrix}

Vector Subtraction

Two vectors can be subtracted only when they are of the same length. The subtraction is element-wise, and the result has that same length. \bold{u}-\bold{v} =\begin{bmatrix}u_1\\u_2\\\vdots \\ u_n\end{bmatrix}-\begin{bmatrix}v_1\\v_2\\\vdots \\v_n\end{bmatrix}=\begin{bmatrix}u_1-v_1\\u_2-v_2\\\vdots \\u_n-v_n\end{bmatrix}.

The difference \bold{u}-\bold{v} is the arrow that carries you from the tip of \bold{v} to the tip of \bold{u}. Read that way, subtraction is the natural way to talk about the gap between two records, and its length is the distance between them.

Subtracting vectors gives the vector that points from the tip of (\bold{v}) to the tip of (\bold{u}).

\bold{u} - \bold{v} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} - \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} u_1 - v_1 \\ u_2 - v_2 \end{bmatrix}

Scalar Multiplication

For a scalar k\in\mathbb{R}, multiplying a vector \bold{u}\in\mathbb{R}^n by k scales every element by k. The outcome is denoted by k\bold{u}. k\bold{u} =k\begin{bmatrix}u_1\\u_2\\\vdots \\ u_n\end{bmatrix}=\begin{bmatrix}ku_1\\ku_2\\\vdots \\ku_n\end{bmatrix}.

Multiplying a vector by a scalar stretches or shrinks its length, and reverses its direction if the scalar is negative.

k\bold{u} = k \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} = \begin{bmatrix} k u_1 \\ k u_2 \end{bmatrix}

Linear Combinations

Addition and scalar multiplication are the only two operations a vector space really has, and everything else is built from them. A linear combination of \bold{u}_1,\ldots,\bold{u}_k\in\mathbb{R}^n is any vector of the form c_1\bold{u}_1+c_2\bold{u}_2+\cdots+c_k\bold{u}_k,\qquad c_1,\ldots,c_k\in\mathbb{R}.

The set of all such combinations is called the span of \bold{u}_1,\ldots,\bold{u}_k. Two independent vectors in \mathbb{R}^3, for instance, span a plane through the origin. Keep this picture: in 3.2 Projection and Orthogonality we will ask which point of a span sits closest to a given vector, and the answer is what a regression fit turns out to be.

Note Linear Combination

If c\bold{x}+d\bold{y}=\bold{z}, can you express \bold{y} in terms of c,\bold{x},\bold{z}?

Possible only when d\neq0. In that case, we can write \bold{y}=\frac{1}{d}(\bold{z}-c\bold{x}).

Dot Product

Given two vectors \bold{u},\bold{v}\in\mathbb{R}^n of the same length, their dot product \langle\bold{u},\bold{v}\rangle (also written \bold{u}^T\bold{v} or \bold{u}\cdot\bold{v}, and also called the inner product) multiplies the entries in matching positions and adds up the results: \langle\bold{u},\bold{v}\rangle=\sum_{i=1}^n u_iv_i.

Note what kind of object comes out: two vectors go in, and a single scalar comes out. That is why the dot product is the natural way to turn a whole row of data into one number—a score, a prediction, a similarity.

For example, let \bold{u} hold a student’s marks on four components of a course and let \bold{w} hold the corresponding weights. Then \langle\bold{u},\bold{w}\rangle is the weighted sum of the marks; it is a weighted average exactly when the weights sum to 1.

Weighted sum and weighted average are not the same thing. Doubling every weight doubles \langle\bold{u},\bold{w}\rangle but must not change an average. Whenever you want an average, check that \sum_i w_i=1.

Every linear model you will meet this semester—linear regression, logistic regression, the support vector machine—predicts by computing \langle\bold{x},\bold{w}\rangle for a feature vector \bold{x} and a learned weight vector \bold{w}. Training such a model is the search for a good \bold{w}.

Length of a Vector

The dot product of a vector with itself gives the squared length, or norm, of the vector: \|\bold{u}\|=\sqrt{\langle\bold{u},\bold{u}\rangle}=\sqrt{u_1^2+\cdots+u_n^2}.

In \mathbb{R}^2 this is exactly the Pythagorean theorem, and in higher dimensions we take it as the definition. A vector with \|\bold{u}\|=1 is called a unit vector; dividing any non-zero \bold{u} by its norm produces the unit vector \bold{u}/\|\bold{u}\| pointing the same way, an operation called normalizing.

The distance between two vectors is the norm of their difference, \|\bold{u}-\bold{v}\|. That single formula is the whole basis of k-nearest neighbours and of k-means clustering later in the course.

Geometry of the Dot Product

The reason the dot product matters is that this purely arithmetic sum secretly measures an angle: \langle\bold{u},\bold{v}\rangle=\|\bold{u}\|\,\|\bold{v}\|\cos\theta, where \theta is the angle between the two vectors. Rearranging, \cos\theta=\frac{\langle\bold{u},\bold{v}\rangle}{\|\bold{u}\|\,\|\bold{v}\|}, a quantity data scientists call the cosine similarity. Because it divides out both lengths, it compares directions only: a document and the same document repeated twice have cosine similarity 1.

Three cases are worth memorising, and all three follow from the sign of \cos\theta:

Sign of the dot product and the angle
\langle\bold{u},\bold{v}\rangle angle \theta meaning
>0 acute the vectors point broadly the same way
=0 90^\circ the vectors are orthogonal
<0 obtuse the vectors point broadly opposite ways

The middle row is the one the rest of the course is built on. Orthogonality is defined by \langle\bold{u},\bold{v}\rangle=0 in every dimension, and 3.2 Projection and Orthogonality uses it to find the closest point in a subspace: the error of the best approximation is exactly the part that is orthogonal to everything you were allowed to use.

Note Sign of the Dot Product

Let \bold{u},\bold{v}\in\mathbb{R}^n be non-zero vectors with \langle\bold{u},\bold{v}\rangle<0. Which statement must be true?

Write the dot product in its geometric form, \langle\bold{u},\bold{v}\rangle=\|\bold{u}\|\,\|\bold{v}\|\cos\theta. Both norms are strictly positive because the vectors are non-zero, so the sign of the dot product is exactly the sign of \cos\theta. A negative dot product therefore forces \cos\theta<0, i.e. \theta\in(90^\circ,180^\circ]; and \theta=180^\circ is allowed only when the vectors are anti-parallel, which is still not orthogonal.

Orthogonality would mean \langle\bold{u},\bold{v}\rangle=0, so the second option is wrong. The third fails on \bold{u}=[1,1], \bold{v}=[-1,-1]: every entry of \bold{u} is positive, yet the dot product is -2. The fourth fails on the same example, where the two norms are equal. The dot product sees only the angle, never the individual signs or lengths.

Exercises

Note Find the Bug

The function below is supposed to return \langle\bold{u},\bold{v}\rangle. It raises no error, returns a number, and is wrong for almost every input. Exactly one line is at fault—find it and fix it.

def dot(u, v):
    total = 0
    for i in range(len(u)):
        total += u[i] * v[i]      # accumulate, do not overwrite
    return total
Note Weighted Sum or Weighted Average?

A course is graded on homework, quizzes, a project, and participation with the weights below. Compute the student’s course grade as a dot product, using only sum, arithmetic, and a comprehension—no loops needed.

grade = sum(s * w for s, w in zip(scores, weights))

Note Angles Without a Protractor

Three vectors are given: \bold{u}=[3,4], \bold{v}=[-4,3], and \bold{w}=[6,8]. Write cos_sim(a, b), the cosine of the angle between two vectors, using only zip, sum, and **.

Before you run it, predict the three numbers. What is the relationship between \bold{u} and \bold{w}? Between \bold{v} and \bold{w}?

def cos_sim(a, b):
    dot  = sum(ai * bi for ai, bi in zip(a, b))
    norm_a = sum(ai * ai for ai in a) ** 0.5
    norm_b = sum(bi * bi for bi in b) ** 0.5
    return dot / (norm_a * norm_b)