1.2 Vector Operations

: 20 minutes

Linear algebra operations are the building blocks of data transformations. Whether you’re scaling a vector or multiplying matrices, these operations make abstract ideas concrete and power—everything from graphics to machine learning!

Vector Addition

Two vectors can be added only when the are of the same length. The addition is element-wise addition, and the resulting vector is also of the same length. \bold{u}+\bold{v} =\begin{bmatrix}u_1\\u_2\\\vdots \\ u_n\end{bmatrix}+\begin{bmatrix}v_1\\v_2\\\vdots \\v_n\end{bmatrix}=\begin{bmatrix}u_1+u_2\\u_2+v_2\\\vdots \\u_n+v_n\end{bmatrix}.

TipGeometry of Vector Addition

Adding vectors places them head-to-tail, and the result is the diagonal of the parallelogram they form.

\bold{u} + \bold{v} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} + \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} u_1 + v_1 \\ u_2 + v_2 \end{bmatrix}

viewof u1_add = Inputs.range([-8, 8], {
  step: 0.5, 
  value: 3, 
  label: tex`u_1`, 
  width: 180
})

viewof u2_add = Inputs.range([-8, 8], {
  step: 0.5, 
  value: 2, 
  label: tex`u_2`, 
  width: 180
})

// Vector v components for addition
viewof v1_add = Inputs.range([-8, 8], {
  step: 0.5, 
  value: -2, 
  label: tex`v_1`, 
  width: 180
})

viewof v2_add = Inputs.range([-8, 8], {
  step: 0.5, 
  value: 4, 
  label: tex`v_2`, 
  width: 180
})

// Display vectors and sum
html`<div style="text-align: center; font-size: 1.2em; margin: 1.5em 0;">
  <p>
    ${tex`\bold{u} = \begin{bmatrix} ${u1_add} \\ ${u2_add} \end{bmatrix}, \quad \bold{v} = \begin{bmatrix} ${v1_add} \\ ${v2_add} \end{bmatrix}, \quad \bold{u} + \bold{v} = \begin{bmatrix} ${u1_add + v1_add} \\ ${u2_add + v2_add} \end{bmatrix}`}
  </p>
</div>`

Plot.plot({
  style: "overflow: visible; display: block; margin: 0 auto;",
  width: 600,
  height: 500,
  grid: true,
  x: {label: "x_1", domain: [-12, 12]},
  y: {label: "x_2", domain: [-12, 12]},
  marks: [
    Plot.ruleY([0], {stroke: "#ddd"}),
    Plot.ruleX([0], {stroke: "#ddd"}),
    
    // Vector u (blue)
    Plot.arrow([{x1: 0, y1: 0, x2: u1_add, y2: u2_add}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "steelblue", strokeWidth: 3, fill: "steelblue"
    }),
    
    // Vector v (red)
    Plot.arrow([{x1: 0, y1: 0, x2: v1_add, y2: v2_add}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "crimson", strokeWidth: 3, fill: "crimson"
    }),
    
    // Vector sum u + v (green)
    Plot.arrow([{x1: 0, y1: 0, x2: u1_add + v1_add, y2: u2_add + v2_add}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "green", strokeWidth: 3, fill: "green"
    }),
    
    // Parallelogram lines
    Plot.line([{x: u1_add, y: u2_add}, {x: u1_add + v1_add, y: u2_add + v2_add}], {
      x: "x", y: "y", stroke: "#999", strokeWidth: 1, strokeDasharray: "5,5"
    }),
    Plot.line([{x: v1_add, y: v2_add}, {x: u1_add + v1_add, y: u2_add + v2_add}], {
      x: "x", y: "y", stroke: "#999", strokeWidth: 1, strokeDasharray: "5,5"
    }),
    
    // Labels
    Plot.text([
      {x: u1_add/2, y: u2_add/2 + 0.5, text: "u"},
      {x: v1_add/2, y: v2_add/2 - 0.5, text: "v"},
      {x: (u1_add + v1_add)/2, y: (u2_add + v2_add)/2 + 0.8, text: "u+v"}
    ], {
      x: "x", y: "y", text: "text", fontSize: 14, fontWeight: "bold"
    })
  ]
})

Vector Subtraction

Two vectors can be subtracted only when the are of the same length. The subtraction is element-wise subtraction, and the resulting vector is also of the same length. \bold{u}-\bold{v} =\begin{bmatrix}u_1\\u_2\\\vdots \\ u_n\end{bmatrix}-\begin{bmatrix}v_1\\v_2\\\vdots \\v_n\end{bmatrix}=\begin{bmatrix}u_1-u_2\\u_2-v_2\\\vdots \\u_n-v_n\end{bmatrix}.

TipGeometry of Vector Subtraction

Subtracting vectors gives the vector that points from the tip of (\bold{v}) to the tip of (\bold{u}).

\bold{u} - \bold{v} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} - \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} u_1 - v_1 \\ u_2 - v_2 \end{bmatrix}

viewof u1_sub = Inputs.range([-8, 8], {
  step: 0.5, 
  value: 4, 
  label: tex`u_1`, 
  width: 180
})

viewof u2_sub = Inputs.range([-8, 8], {
  step: 0.5, 
  value: 3, 
  label: tex`u_2`, 
  width: 180
})

// Vector v components for subtraction
viewof v1_sub = Inputs.range([-8, 8], {
  step: 0.5, 
  value: 1, 
  label: tex`v_1`, 
  width: 180
})

viewof v2_sub = Inputs.range([-8, 8], {
  step: 0.5, 
  value: 2, 
  label: tex`v_2`, 
  width: 180
})

// Display vectors and difference
html`<div style="text-align: center; font-size: 1.2em; margin: 1.5em 0;">
  <p>
    ${tex`\bold{u} = \begin{bmatrix} ${u1_sub} \\ ${u2_sub} \end{bmatrix}, \quad \bold{v} = \begin{bmatrix} ${v1_sub} \\ ${v2_sub} \end{bmatrix}, \quad \bold{u} - \bold{v} = \begin{bmatrix} ${u1_sub - v1_sub} \\ ${u2_sub - v2_sub} \end{bmatrix}`}
  </p>
</div>`

Plot.plot({
  style: "overflow: visible; display: block; margin: 0 auto;",
  width: 600,
  height: 500,
  grid: true,
  x: {label: "x_1", domain: [-12, 12]},
  y: {label: "x_2", domain: [-12, 12]},
  marks: [
    Plot.ruleY([0], {stroke: "#ddd"}),
    Plot.ruleX([0], {stroke: "#ddd"}),
    
    // Vector u (blue)
    Plot.arrow([{x1: 0, y1: 0, x2: u1_sub, y2: u2_sub}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "steelblue", strokeWidth: 3, fill: "steelblue"
    }),
    
    // Vector v (red)
    Plot.arrow([{x1: 0, y1: 0, x2: v1_sub, y2: v2_sub}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "crimson", strokeWidth: 3, fill: "crimson"
    }),
    
    // Vector -v (red dashed)
    Plot.arrow([{x1: 0, y1: 0, x2: -v1_sub, y2: -v2_sub}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "crimson", strokeWidth: 2, fill: "crimson", 
      strokeDasharray: "5,5", fillOpacity: 0.6
    }),
    
    // Vector difference u - v (purple)
    Plot.arrow([{x1: 0, y1: 0, x2: u1_sub - v1_sub, y2: u2_sub - v2_sub}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "purple", strokeWidth: 3, fill: "purple"
    }),
    
    // Vector from v to u (alternative representation)
    Plot.arrow([{x1: v1_sub, y1: v2_sub, x2: u1_sub, y2: u2_sub}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "#ff6600", strokeWidth: 2, fill: "#ff6600", 
      strokeDasharray: "3,3"
    }),
    
    // Labels
    Plot.text([
      {x: u1_sub/2, y: u2_sub/2 + 0.5, text: "u"},
      {x: v1_sub/2, y: v2_sub/2 - 0.5, text: "v"},
      {x: -v1_sub/2, y: -v2_sub/2 + 0.5, text: "-v"},
      {x: (u1_sub - v1_sub)/2, y: (u2_sub - v2_sub)/2 + 0.8, text: "u-v"}
    ], {
      x: "x", y: "y", text: "text", fontSize: 14, fontWeight: "bold"
    })
  ]
})

Scalar Multiplication

For a scalar k\in\R, the scalar multiplication of a vector \bold{u}\in\R^n by k is element-wise scalar multiplication by k, and the outcome is denoted by k\bold{u}. k\bold{u} =k\begin{bmatrix}u_1\\u_2\\\vdots \\ u_n\end{bmatrix}=\begin{bmatrix}ku_1\\ku_2\\\vdots \\ku_n\end{bmatrix}.

TipGeometry of Scalar Multiplication

Multiplying a vector by a scalar stretches or shrinks its length, and reverses its direction if the scalar is negative.

k\bold{u} = k \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} = \begin{bmatrix} k u_1 \\ k u_2 \end{bmatrix}

viewof scalar = Inputs.range([-3, 3], {
  step: 0.1, 
  value: 1.5, 
  label: tex`k`, 
  width: 200
})

// Vector u components for scalar multiplication
viewof u1_scalar = Inputs.range([-6, 6], {
  step: 0.5, 
  value: 3, 
  label: tex`u_1`, 
  width: 180
})

viewof u2_scalar = Inputs.range([-6, 6], {
  step: 0.5, 
  value: 2, 
  label: tex`u_2`, 
  width: 180
})

// Display vectors and scalar product
html`<div style="text-align: center; font-size: 1.2em; margin: 1.5em 0;">
  <p>
    ${tex`k = ${scalar.toFixed(1)}, \quad \bold{u} = \begin{bmatrix} ${u1_scalar} \\ ${u2_scalar} \end{bmatrix}, \quad k\bold{u} = \begin{bmatrix} ${(scalar * u1_scalar).toFixed(1)} \\ ${(scalar * u2_scalar).toFixed(1)} \end{bmatrix}`}
  </p>
</div>`

Plot.plot({
  style: "overflow: visible; display: block; margin: 0 auto;",
  width: 600,
  height: 500,
  grid: true,
  x: {label: "x_1", domain: [-12, 12]},
  y: {label: "x_2", domain: [-12, 12]},
  marks: [
    Plot.ruleY([0], {stroke: "#ddd"}),
    Plot.ruleX([0], {stroke: "#ddd"}),
    
    // Original vector u (blue)
    Plot.arrow([{x1: 0, y1: 0, x2: u1_scalar, y2: u2_scalar}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "steelblue", strokeWidth: 3, fill: "steelblue"
    }),
    
    // Scaled vector ku (orange)
    Plot.arrow([{x1: 0, y1: 0, x2: scalar * u1_scalar, y2: scalar * u2_scalar}], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "orange", strokeWidth: 3, fill: "orange"
    }),
    
    // Unit vector in same direction (if scalar != 0)
    scalar !== 0 ? Plot.arrow([{
      x1: 0, y1: 0, 
      x2: u1_scalar / Math.sqrt(u1_scalar*u1_scalar + u2_scalar*u2_scalar), 
      y2: u2_scalar / Math.sqrt(u1_scalar*u1_scalar + u2_scalar*u2_scalar)
    }], {
      x1: "x1", y1: "y1", x2: "x2", y2: "y2",
      stroke: "#666", strokeWidth: 1, fill: "#666", 
      strokeDasharray: "2,2"
    }) : null,
    
    // Labels
    Plot.text([
      {x: u1_scalar/2, y: u2_scalar/2 + 0.5, text: "u"},
      {x: (scalar * u1_scalar)/2, y: (scalar * u2_scalar)/2 - 0.8, text: `${scalar.toFixed(1)}u`}
    ], {
      x: "x", y: "y", text: "text", fontSize: 14, fontWeight: "bold"
    })
  ]
})

Note Linear Combination

If c\bold{x}+d\bold{y}=\bold{z}, can you express \bold{y} in terms of c,\bold{x},\bold{z}?

\bold{y}=\frac{1}{d}(\bold{z}-c\bold{x})\bold{y}=\frac{1}{d}(\bold{z}+c\bold{x})\bold{y}=\frac{\bold{z}-c\bold{x}}{d}Not always possible

Possible only when d\neq0. In that case, we can write \bold{y}=\frac{1}{d}(\bold{z}-c\bold{x}).

Dot Product

Given two vectors \bold{u},\bold{v}\in\mathbb{R}^n of length the same length, their dot product \langle\bold{u},\bold{v}\rangle (also known as inner product) is the sum over the products of the elements at the same position: \langle\bold{u},\bold{v}\rangle=\sum_{i=1}^n u_iv_i.

Dot products are a useful tool in data science with a variety of practical applications. For example, if \bold{u} presents a set of features and \bold{w} a set of corresponding weights, then their inner product \langle \bold{u}, \bold{v}\rangle gives us the weighted average.

TipGeometry of Dot Product

After normalizing two vectors to have unit length, the dot products express the cosine of the angle between them.