Weight Matrix and Matrix-Vector Multiplication

In Topics 1 and 2, we built an artificial neuron to answer a single question: "Will this movie be a hit?"

It took a script's feature vector $x$, computed its linear score, and passed that score through an activation function to yield a single scalar output. In neural networks, this output is denoted by $a$ (short for activation):

$$z = w \cdot x + b \qquad \longrightarrow \qquad a = \sigma(z)$$

Real-world problems are rarely that simple. In practice, systems must answer multiple questions from the exact same input data.

What if the movie studio wants to evaluate that same script from multiple angles? It might ask three distinct questions:

1. Will this movie be a hit?
2. Does this script have Oscar potential?
3. Does this script have international appeal?

But what makes a commercial blockbuster isn't what wins Best Picture or what sells overseas. Each question evaluates the script's features differently.

To answer all three questions, we need three separate neurons evaluating the same script—each neuron equipped with its own set of weights, bias, and activation function.

Across this topic, we will see how each stage of the pipeline is computed when running multiple neurons in parallel—starting with how multiple sets of weights evaluate the same script:

---

One Script, Three Weight Vectors

In Topics 1 and 2, we represented our script, 'Die Hard in Space', as a 4-dimensional feature vector across [Action, Romance, Comedy, Sci-Fi]:

$$x = \begin{bmatrix} 1.0 \\ 0.0 \\ 0.5 \\ 1.0 \end{bmatrix} \begin{matrix} \text{(Action)} \\ \text{(Romance)} \\ \text{(Comedy)} \\ \text{(Sci-Fi)} \end{matrix}$$

To evaluate all three questions, the studio defines three separate weight vectors—each reflecting what it believes different stakeholders value in a script:

$$\begin{array}{cccc} & \textbf{Hit} & \textbf{Oscar} & \textbf{International} \\ & \text{\small(Audience Preferences)} & \text{\small(Academy Preferences)} & \text{\small(Foreign Market Preferences)} \\[6pt] \begin{matrix} \text{(Action)} \\ \text{(Romance)} \\ \text{(Comedy)} \\ \text{(Sci-Fi)} \end{matrix} & w_{\text{Hit}} = \begin{bmatrix} 5.0 \\ -2.0 \\ 1.0 \\ 4.0 \end{bmatrix} & w_{\text{Oscar}} = \begin{bmatrix} -3.0 \\ 4.0 \\ 1.0 \\ -2.0 \end{bmatrix} & w_{\text{Intl}} = \begin{bmatrix} 4.0 \\ -1.0 \\ 0.0 \\ 5.0 \end{bmatrix} \end{array}$$

To evaluate the script against each criterion, we calculate three independent dot products (with weights shown in bold):

$$\begin{aligned} w_{\text{Hit}} \cdot x &= (\mathbf{5.0} \times 1.0) + (\mathbf{-2.0} \times 0.0) + (\mathbf{1.0} \times 0.5) + (\mathbf{4.0} \times 1.0) &&= \mathbf{+9.5} \\[4pt] w_{\text{Oscar}} \cdot x &= (\mathbf{-3.0} \times 1.0) + (\mathbf{4.0} \times 0.0) + (\mathbf{1.0} \times 0.5) + (\mathbf{-2.0} \times 1.0) &&= \mathbf{-4.5} \\[4pt] w_{\text{Intl}} \cdot x &= (\mathbf{4.0} \times 1.0) + (\mathbf{-1.0} \times 0.0) + (\mathbf{0.0} \times 0.5) + (\mathbf{5.0} \times 1.0) &&= \mathbf{+9.0} \end{aligned}$$

Each dot product produces a single raw score based on the respective preferences. Hit: +9.5, Oscar: -4.5, International: +9.0.

From Weight Vectors to the Weight Matrix ($W$)

Look at the bold numbers in the three dot product calculations above. Each line is just one of our original weight vectors laid flat on its side.

In linear algebra, flipping the axes is called the transpose, denoted by a superscript $T$. Transposing our vertical weight vectors turns them into horizontal row vectors ($w^T$).

When we stack these rows into a single grid, we get the Weight Matrix ($W$):

A matrix is a grid of numbers organized into rows and columns. When it comes to AI, it is simply multiple vectors bundled together: if a vector is an ordered list of numbers, a matrix is a stack of vectors that share the exact same dimension.

Notice the structure of our Weight Matrix:

• Each row is the weight vector of one neuron (one evaluation question).
• Each column corresponds to one script feature ([Action, Romance, Comedy, Sci-Fi]).

Because Position is Identity, the columns of the weight matrix strictly match the rows of the script vector: column 1: Action, column 2: Romance, column 3: Comedy, and column 4: Sci-Fi.

Because we have 3 neurons evaluating 4 features, the Weight Matrix has 3 rows and 4 columns (shape $3 \times 4$).

Matrix-Vector Multiplication ($Wx$)

To evaluate our script across all three neurons at once, we multiply the Weight Matrix $W$ by the input vector $x$.

Matrix-vector multiplication follows one universal mechanical rule: Row $\times$ Column.

Each row in the matrix performs a dot product against the column vector: multiplying matching positions and adding them up:

$$Wx = \begin{bmatrix} 5.0 & -2.0 & 1.0 & 4.0 \\ -3.0 & 4.0 & 1.0 & -2.0 \\ 4.0 & -1.0 & 0.0 & 5.0 \end{bmatrix} \begin{bmatrix} 1.0 \\ 0.0 \\ 0.5 \\ 1.0 \end{bmatrix} = \begin{bmatrix} (\mathbf{5.0} \times 1.0) + (\mathbf{-2.0} \times 0.0) + (\mathbf{1.0} \times 0.5) + (\mathbf{4.0} \times 1.0) \\ (\mathbf{-3.0} \times 1.0) + (\mathbf{4.0} \times 0.0) + (\mathbf{1.0} \times 0.5) + (\mathbf{-2.0} \times 1.0) \\ (\mathbf{4.0} \times 1.0) + (\mathbf{-1.0} \times 0.0) + (\mathbf{0.0} \times 0.5) + (\mathbf{5.0} \times 1.0) \end{bmatrix} = \begin{bmatrix} +9.5 \\ -4.5 \\ +9.0 \end{bmatrix}$$

Look at the arithmetic inside that calculation: matrix-vector multiplication is computing those exact three dot products simultaneously.

$$Wx = \begin{bmatrix} w_{\text{Hit}} \cdot x \\ w_{\text{Oscar}} \cdot x \\ w_{\text{Intl}} \cdot x \end{bmatrix} = \begin{bmatrix} +9.5 \\ -4.5 \\ +9.0 \end{bmatrix} \begin{matrix} \text{(Hit Raw Score)} \\ \text{(Oscar Raw Score)} \\ \text{(International Raw Score)} \end{matrix}$$

A matrix does not change the math. It is simply a clean, structured way to evaluate multiple vectors at once.

Look at what the three raw scores suggest about the script:

• $+9.5$ (Hit): High commercial appeal. Heavy Action and Sci-Fi produce a strong positive score, matching our single-neuron result from Topic 1.
• $-4.5$ (Oscar): Critical failure. Academy voters penalize explosive spectacle, dragging the Oscar score deep into negative territory.
• $+9.0$ (International): Strong overseas appeal. Visual action and sci-fi spectacle translate across languages and cultural barriers without needing dialogue translation.

One matrix multiplication evaluated the script from three completely different creative perspectives at the same time.

Evaluating 'The Notebook 2'

Now we evaluate our second script, 'The Notebook 2', using the exact same Weight Matrix $W$:

$$x = \begin{bmatrix} 0.0 \\ 1.0 \\ 0.5 \\ 0.0 \end{bmatrix} \begin{matrix} \text{(Action)} \\ \text{(Romance)} \\ \text{(Comedy)} \\ \text{(Sci-Fi)} \end{matrix}$$

Multiplying $W$ by this input vector:

$$Wx = \begin{bmatrix} 5.0 & -2.0 & 1.0 & 4.0 \\ -3.0 & 4.0 & 1.0 & -2.0 \\ 4.0 & -1.0 & 0.0 & 5.0 \end{bmatrix} \begin{bmatrix} 0.0 \\ 1.0 \\ 0.5 \\ 0.0 \end{bmatrix} = \begin{bmatrix} -1.5 \\ +4.5 \\ -1.0 \end{bmatrix} \begin{matrix} \text{(Hit Raw Score)} \\ \text{(Oscar Raw Score)} \\ \text{(International Raw Score)} \end{matrix}$$

Look at what the three raw scores suggest about this script:

• $-1.5$ (Hit): Low commercial appeal. A romance feature with slight comedy does not match audience preference for action.
• $+4.5$ (Oscar): High critical appeal. Academy voters reward romance and comedy, resulting in a positive score.
• $-1.0$ (International): Weak overseas draw. Dialogue-heavy romance does not travel as easily into foreign markets without action or sci-fi spectacle.

---

Why Neural Networks Use Matrices

Matrix multiplication is the foundational operation of modern AI for two reasons:

1. Multiple Criteria on the Same Data: The input features ($x$) never change. The matrix $W$ simply evaluates those same features against different criteria at the same time.
2. Hardware Parallelism: Because each row calculation is completely independent of the others, modern hardware (like GPUs) calculates every row in parallel. Evaluating three neurons—or 3,000 neurons—takes virtually the same computational time as evaluating one.

---