Matrix-Vector Multiplication

In Lesson 1, we stacked the weights of three studio neurons into the horizontal rows of a single weight matrix $W$.

When an incoming script's feature vector $x$ arrives at a neural network layer, the model must evaluate all of those rows against that single input simultaneously. That operation is matrix-vector multiplication.

Despite the formal name, matrix-vector multiplication is not a new branch of arithmetic. It is simply multiple dot products executed in parallel.

---

The Row-by-Column Dot Product Rule

To calculate the $i$-th entry of the output vector $z$, we compute the dot product between the $i$-th row of the weight matrix $W$ and the input vector $x$:

$$z_i = \sum_{j=1}^n W_{ij} x_j = W_{i1} x_1 + W_{i2} x_2 + \dots + W_{in} x_n = w_i \cdot x$$

Writing the full operation in expanded algebraic form:

$$\begin{bmatrix} W_{11} & W_{12} & \dots & W_{1n} \\ W_{21} & W_{22} & \dots & W_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ W_{m1} & W_{m2} & \dots & W_{mn} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} = \begin{bmatrix} W_{11}x_1 + W_{12}x_2 + \dots + W_{1n}x_n \\ W_{21}x_1 + W_{22}x_2 + \dots + W_{2n}x_n \\ \vdots \\ W_{m1}x_1 + W_{m2}x_2 + \dots + W_{mn}x_n \end{bmatrix} = \begin{bmatrix} w_1 \cdot x \\ w_2 \cdot x \\ \vdots \\ w_m \cdot x \end{bmatrix} = \begin{bmatrix} z_1 \\ z_2 \\ \vdots \\ z_m \end{bmatrix}$$

Every row of the matrix sweeps horizontally across its columns, multiplying each weight $W_{ij}$ by the corresponding feature value $x_j$, and sums the individual products into a single scalar value.

---

Walk-Through: Calculating Matrix Scores for 'Die Hard in Space'

Now let's execute this exact operation with concrete numbers using our running studio predictor from Lesson 1.

Recall our $3 \times 4$ Weight Matrix ($W$), where each row provides the weights for one neuron across the locked genre order [Action, Romance, Comedy, Sci-Fi]:

$$W = \begin{bmatrix} 5.0 & -2.0 & 1.0 & 4.0 \\ -5.0 & 5.0 & 5.0 & 0.0 \\ 5.0 & 0.0 & 0.0 & 4.0 \end{bmatrix} \begin{matrix} \text{(Row 1: Hit, } w_1\text{)} \\ \text{(Row 2: Oscar, } w_2\text{)} \\ \text{(Row 3: Franchise, } w_3\text{)} \end{matrix}$$

Our incoming script feature vector for 'Die Hard in Space' is:

$$x = \begin{bmatrix} 1.0 \\ 0.0 \\ 0.5 \\ 1.0 \end{bmatrix} \begin{matrix} \text{(Action)} \\ \text{(Romance)} \\ \text{(Comedy)} \\ \text{(Sci-Fi)} \end{matrix}$$

Multiplying the weight matrix $W$ by the input vector $x$:

$$W x = \begin{bmatrix} 5.0 & -2.0 & 1.0 & 4.0 \\ -5.0 & 5.0 & 5.0 & 0.0 \\ 5.0 & 0.0 & 0.0 & 4.0 \end{bmatrix} \begin{bmatrix} 1.0 \\ 0.0 \\ 0.5 \\ 1.0 \end{bmatrix} = \begin{bmatrix} (5.0)(1.0) + (-2.0)(0.0) + (1.0)(0.5) + (4.0)(1.0) \\ (-5.0)(1.0) + (5.0)(0.0) + (5.0)(0.5) + (0.0)(1.0) \\ (5.0)(1.0) + (0.0)(0.0) + (0.0)(0.5) + (4.0)(1.0) \end{bmatrix}$$

We evaluate each row dot product step by step:

• Row 1 (Hit): $w_1 \cdot x = (5.0 \times 1.0) + (-2.0 \times 0.0) + (1.0 \times 0.5) + (4.0 \times 1.0) = 5.0 + 0.0 + 0.5 + 4.0 = \mathbf{+9.5}$
• Row 2 (Oscar): $w_2 \cdot x = (-5.0 \times 1.0) + (5.0 \times 0.0) + (5.0 \times 0.5) + (0.0 \times 1.0) = -5.0 + 0.0 + 2.5 + 0.0 = \mathbf{-2.5}$
• Row 3 (Franchise): $w_3 \cdot x = (5.0 \times 1.0) + (0.0 \times 0.0) + (0.0 \times 0.5) + (4.0 \times 1.0) = 5.0 + 0.0 + 0.0 + 4.0 = \mathbf{+9.0}$

The resulting output is a 3-dimensional column vector holding the raw decision scores:

$$W x = \begin{bmatrix} +9.5 \\ -2.5 \\ +9.0 \end{bmatrix} \begin{matrix} \text{(Hit Raw Score)} \\ \text{(Oscar Raw Score)} \\ \text{(Franchise Raw Score)} \end{matrix}$$

We can also inspect the arithmetic in a structured feature ledger, keeping incoming features stacked along the vertical axis:

Feature Dimension	Feature Value ($x$)	Row 1: Hit ($w_1$)	Row 2: Oscar ($w_2$)	Row 3: Franchise ($w_3$)
Action ($j=1$)	$1.0$	$1.0 \times 5.0 = +5.0$	$1.0 \times -5.0 = -5.0$	$1.0 \times 5.0 = +5.0$
Romance ($j=2$)	$0.0$	$0.0 \times -2.0 = 0.0$	$0.0 \times 5.0 = 0.0$	$0.0 \times 0.0 = 0.0$
Comedy ($j=3$)	$0.5$	$0.5 \times 1.0 = +0.5$	$0.5 \times 5.0 = +2.5$	$0.5 \times 0.0 = 0.0$
Sci-Fi ($j=4$)	$1.0$	$1.0 \times 4.0 = +4.0$	$1.0 \times 0.0 = 0.0$	$1.0 \times 4.0 = +4.0$
Raw Dot Product ($w_i \cdot x$)		$+9.5$	$-2.5$	$+9.0$

Notice that Row 1 reproduces the exact raw score of $+9.5$ calculated for 'Die Hard in Space' by our single artificial neuron in Topics 1 and 2. Stacking vectors into a matrix did not change the arithmetic—it packaged three independent evaluation channels into a single mathematical operation.

---

The Dimension Compatibility Rule

Not every matrix can multiply every vector. The physical setup dictates strict dimensional alignment.

In our studio predictor, 'Die Hard in Space' supplies 4 genre features ($n = 4$). For each row in the matrix to compute a dot product with that script, it must provide exactly 4 weights—one for each genre. If a matrix had only 3 columns, there would be no weight to evaluate Sci-Fi. If it had 5 columns, the 5th weight would have no incoming feature to pair with.

Therefore, the number of columns in the weight matrix must equal the number of rows in the input vector.

Likewise, because each row in $W$ represents one neuron, stacking $m$ rows produces exactly $m$ individual dot products—a column vector of length $m$.

In formal linear algebra notation, the Dimension Compatibility Rule is written:

$$(m \times n) \times (n \times 1) \longrightarrow (m \times 1)$$

This rule establishes two structural constraints:

1. The Inner Dimensions Must Match ($n = n$): The number of columns in $W$ ($n$) must equal the number of rows in $x$ ($n$). Every incoming feature coordinate must have a corresponding weight parameter in each row.
2. The Outer Dimensions Determine the Output Shape ($m \times 1$): The number of rows in $W$ ($m$) determines the number of entries in the resulting output vector. Stacking $m$ neurons produces an $m$-dimensional column vector.

A matrix $W \in \mathbb{R}^{m \times n}$ acts as a dimensional transformation machine. It accepts an $n$-dimensional input vector and transforms it into an $m$-dimensional output vector:

$$\text{Input } (x \in \mathbb{R}^n) \quad \xrightarrow{\quad \text{Weight Matrix } (W \in \mathbb{R}^{m \times n}) \quad} \quad \text{Output } (z \in \mathbb{R}^m)$$

Depending on the chosen dimensions, a matrix transformation alters coordinate representations in one of three ways:

• Dimensional Expansion ($m > n$): Expands the representation into a higher-dimensional space with more output neurons than incoming features.
• Dimensional Compression ($m < n$): Compresses the representation into a lower-dimensional space, summarizing inputs into fewer coordinates.
• Dimensional Preservation ($m = n$): Maps features into an output space of the exact same dimensionality.

---

Hardware Parallelism: Why AI Runs on GPUs

Understanding matrix-vector multiplication as parallel row dot products reveals why modern deep learning relies on Graphics Processing Units (GPUs) rather than Central Processing Units (CPUs).

In the output vector:

$$z = W x = \begin{bmatrix} w_1 \cdot x \\ w_2 \cdot x \\ \vdots \\ w_m \cdot x \end{bmatrix}$$

The calculation of Row 1 ($w_1 \cdot x$) does not rely on the result of Row 2 ($w_2 \cdot x$) or Row 3 ($w_3 \cdot x$). All $m$ row dot products are completely independent.

A standard CPU contains a small number of powerful, general-purpose cores engineered to execute instructions sequentially: compute Row 1, then Row 2, then Row 3. A GPU contains thousands of smaller, specialized arithmetic cores engineered to execute multiplications and additions across all rows simultaneously in the exact same physical clock cycle.

To visualize how hardware architectures execute matrix operations differently, compare sequential CPU dot products with simultaneous GPU core execution below:

Because every row dot product in $W x$ operates on the exact same input vector $x$ without waiting on neighboring rows, a GPU evaluates the entire matrix transformation in the time it takes to compute a single dot product. This mathematical independence is the foundational reason neural networks train and scale on GPU hardware.

---