Matrices and Multiple Neurons In Practice

Part 1: The Underlying Mechanics

Problem 1: Weight Matrix Construction and Transpose

A neural network layer has 2 neurons evaluating a 3-dimensional input feature vector $x$ .

Neuron A has weight vector: $w_A = \begin{bmatrix} 2.0 \\ -1.0 \\ 4.0 \end{bmatrix}$
Neuron B has weight vector: $w_B = \begin{bmatrix} 0.5 \\ 3.0 \\ -2.0 \end{bmatrix}$
Part A: Construct the Weight Matrix $W$ for this layer by stacking the transposed neuron weight vectors into horizontal rows.
Part B: State the dimensions of $W$ ( $\text{rows} \times \text{columns}$ ).
Part C: Which row of $W$ corresponds to Neuron B, and what do the columns represent?

Reveal Solution

Part A: Transposing each vertical weight vector into a horizontal row ( $w^T$ ) and stacking them produces the Weight Matrix $W$ :

W = \begin{bmatrix} w_A^T \\ w_B^T \end{bmatrix} = \begin{bmatrix} 2.0 & -1.0 & 4.0 \\ 0.5 & 3.0 & -2.0 \end{bmatrix}

Part B: The Weight Matrix $W$ has 2 rows and 3 columns (shape $2 \times 3$ ).

Part C: Neuron B corresponds to Row 2 ( $w_B^T$ ). The columns represent the 3 input features of $x$ . Because Position is Identity, column 1 pairs with feature 1, column 2 with feature 2, and column 3 with feature 3 for both neurons simultaneously.

Problem 2: Matrix Dimensions and Output Shapes

A neural network layer receives an input feature vector $x$ and multiplies it by a weight matrix $W$ with shape $4 \times 5$ .

Part A: How many features must the input vector $x$ contain for the matrix-vector multiplication $Wx$ to be mathematically valid?
Part B: What is the shape of the resulting output vector $Wx$ ?
Part C: How many distinct neurons operate in this layer?

Reveal Solution

Part A: The input vector $x$ must contain 5 features (shape $5 \times 1$ ). For matrix multiplication to be valid, the inner dimensions must match:

$(4 \times \mathbf{5}) \times (\mathbf{5} \times 1)$

Each row in $W$ has 5 weights, requiring exactly 5 corresponding feature values in $x$ .

Part B: The output vector $Wx$ has shape $4 \times 1$ (a 4-dimensional column vector), determined by the outer dimensions $(4 \times 1)$ .

Part C: There are 4 neurons in this layer. In any neural network layer, the number of rows in the weight matrix equals the number of neurons, with each neuron producing one scalar value in the $4 \times 1$ output vector.

Problem 3: Matrix-Vector Multiplication with Mixed Signs ( $3 \times 2$ )

Given weight matrix $W = \begin{bmatrix} 2.0 & -1.0 \\ 0.0 & 3.0 \\ -4.0 & 1.5 \end{bmatrix}$ and input vector $x = \begin{bmatrix} 3.0 \\ -2.0 \end{bmatrix}$ :

Calculate $Wx$ . Show the row-by-row calculations.

Reveal Solution

Matrix-vector multiplication follows the Row $\times$ Column rule:

Wx = \begin{bmatrix} 2.0 & -1.0 \\ 0.0 & 3.0 \\ -4.0 & 1.5 \end{bmatrix} \begin{bmatrix} 3.0 \\ -2.0 \end{bmatrix} = \begin{bmatrix} (2.0)(3.0) + (-1.0)(-2.0) \\ (0.0)(3.0) + (3.0)(-2.0) \\ (-4.0)(3.0) + (1.5)(-2.0) \end{bmatrix}

Row 1 (Neuron 1): $6.0 + 2.0 = \mathbf{+8.0}$
Row 2 (Neuron 2): $0.0 - 6.0 = \mathbf{-6.0}$
Row 3 (Neuron 3): $-12.0 - 3.0 = \mathbf{-15.0}$

Wx = \begin{bmatrix} +8.0 \\ -6.0 \\ -15.0 \end{bmatrix}

Notice that in Row 2, the weight for feature 1 is $0.0$ , completely erasing the impact of feature 1 on Neuron 2.

Problem 4: Position is Identity in Weight Matrices

A $3 \times 2$ weight matrix $W$ evaluates a 2-dimensional feature vector $x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$ .

Part A: If an engineer accidentally swaps Row 1 and Row 2 of $W$ , what changes in the output vector $Wx$ ? Does the mathematical value of each neuron's evaluation change?
Part B: If the engineer instead accidentally swaps Column 1 and Column 2 of $W$ (while leaving $x$ unchanged), what happens to the math? Why does this corrupt the evaluation for all three neurons?

Reveal Solution

Part A: Swapping Row 1 and Row 2 simply swaps the positions of output 1 and output 2 in the resulting vector:

\begin{bmatrix} (w_1 \cdot x) \\ (w_2 \cdot x) \\ (w_3 \cdot x) \end{bmatrix} \longrightarrow \begin{bmatrix} (w_2 \cdot x) \\ (w_1 \cdot x) \\ (w_3 \cdot x) \end{bmatrix}

The mathematical scores themselves do not change; only their designated positions in the output vector are exchanged.

Part B: Swapping Column 1 and Column 2 causes every neuron to multiply Feature 1 by Feature 2's weight, and Feature 2 by Feature 1's weight:

(w_{i,2} x_1) + (w_{i,1} x_2) \neq (w_{i,1} x_1) + (w_{i,2} x_2)

Because Position is Identity, weights are strictly bound to specific features. Column 1 must align with Feature 1, and Column 2 with Feature 2. Swapping columns scrambles the semantic pairing across every row, corrupting all three calculations.

Problem 5: Multi-Neuron Bias Offsets and Decision Thresholds

A 2-neuron layer receives input feature vector $x = \begin{bmatrix} 4.0 \\ 2.0 \end{bmatrix}$ with weight matrix $W = \begin{bmatrix} 1.5 & -2.0 \\ 0.5 & 1.0 \end{bmatrix}$ .

Part A: Calculate $Wx$ .
Part B: Suppose the layer has bias vector $b = \begin{bmatrix} -3.0 \\ +1.0 \end{bmatrix}$ . Compute $z = Wx + b$ . Explain why Neuron 1's bias acts as a "hurdle" while Neuron 2's bias acts as a "head start."
Part C: Find the exact bias vector $b^* = \begin{bmatrix} b_1 \\ b_2 \end{bmatrix}$ that places both neurons simultaneously on the decision threshold ( $z_1 = 0$ and $z_2 = 0$ , yielding $\sigma(z_1) = 0.50$ and $\sigma(z_2) = 0.50$ ).

Reveal Solution

Part A: $Wx$

Wx = \begin{bmatrix} (1.5)(4.0) + (-2.0)(2.0) \\ (0.5)(4.0) + (1.0)(2.0) \end{bmatrix} = \begin{bmatrix} 6.0 - 4.0 \\ 2.0 + 2.0 \end{bmatrix} = \begin{bmatrix} \mathbf{+2.0} \\ \mathbf{+4.0} \end{bmatrix}

Part B: Linear Step ( $z = Wx + b$ ) and Bias Intuition

Vector addition adds matching positions element-by-element:

z = Wx + b = \begin{bmatrix} +2.0 \\ +4.0 \end{bmatrix} + \begin{bmatrix} -3.0 \\ +1.0 \end{bmatrix} = \begin{bmatrix} 2.0 - 3.0 \\ 4.0 + 1.0 \end{bmatrix} = \begin{bmatrix} \mathbf{-1.0} \\ \mathbf{+5.0} \end{bmatrix}

Neuron 1 ( $b_1 = -3.0$ ): A negative bias acts as a hurdle. It pulls the score down, requiring the raw dot product to exceed $+3.0$ just to reach the neutral threshold ( $z = 0$ ). Because the raw score was only $+2.0$ , Neuron 1 ends up negative ( $z_1 = -1.0$ ).
Neuron 2 ( $b_2 = +1.0$ ): A positive bias acts as a head start. It boosts the score by $+1.0$ , pushing the already-positive raw score from $+4.0$ to $+5.0$ .

Part C: Decision Threshold Calibration

The decision threshold occurs where the linear score equals zero ( $z = Wx + b^* = 0$ ):

Neuron 1: $(Wx)_1 + b_1 = 0 \implies 2.0 + b_1 = 0 \implies \mathbf{b_1 = -2.0}$
Neuron 2: $(Wx)_2 + b_2 = 0 \implies 4.0 + b_2 = 0 \implies \mathbf{b_2 = -4.0}$

The required bias vector is:

b^* = \begin{bmatrix} -2.0 \\ -4.0 \end{bmatrix}

Because neurons in a layer are independent, each bias is calibrated strictly to its own neuron's raw score.

Problem 6: Complete Multi-Neuron Forward Pass Pipeline

A neural network layer with 2 neurons evaluates input vector $x = \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix}$ . The layer is defined by:

W = \begin{bmatrix} 3.0 & -1.0 \\ -2.0 & 2.0 \end{bmatrix} \qquad\qquad b = \begin{bmatrix} -2.0 \\ -1.0 \end{bmatrix}

Part A: Compute $z = Wx + b$ .
Part B: Using the Sigmoid formula $\sigma(z) = \frac{1}{1 + e^{-z}}$ and the reference values below, calculate the activation vector $a = \sigma(z)$ : $\text{Reference: } e^{-1.0} \approx 0.3679, \quad e^{0.0} = 1.0000, \quad e^{1.0} \approx 2.7183$
Part C: Apply the standard $0.50$ decision threshold ( $a_i \ge 0.50 \to 1$ , $a_i < 0.50 \to 0$ ) to state the layer's binary classification vector.

Reveal Solution

Part A: Linear Step ( $z = Wx + b$ )

Wx = \begin{bmatrix} (3.0)(1.0) + (-1.0)(2.0) \\ (-2.0)(1.0) + (2.0)(2.0) \end{bmatrix} = \begin{bmatrix} 3.0 - 2.0 \\ -2.0 + 4.0 \end{bmatrix} = \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix}

z = Wx + b = \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix} + \begin{bmatrix} -2.0 \\ -1.0 \end{bmatrix} = \begin{bmatrix} \mathbf{-1.0} \\ \mathbf{+1.0} \end{bmatrix}

Part B: Element-Wise Sigmoid ( $a = \sigma(z)$ )

Neuron 1 ( $z_1 = -1.0$ ): $\sigma(-1.0) = \frac{1}{1 + e^{-(-1.0)}} = \frac{1}{1 + e^{1.0}} = \frac{1}{1 + 2.7183} = \frac{1}{3.7183} \approx \mathbf{0.2689} \quad (\mathbf{26.9\%})$
Neuron 2 ( $z_2 = +1.0$ ): $\sigma(+1.0) = \frac{1}{1 + e^{-1.0}} = \frac{1}{1 + 0.3679} = \frac{1}{1.3679} \approx \mathbf{0.7311} \quad (\mathbf{73.1\%})$

a = \begin{bmatrix} \sigma(-1.0) \\ \sigma(1.0) \end{bmatrix} \approx \begin{bmatrix} \mathbf{0.269} \\ \mathbf{0.731} \end{bmatrix}

Part C: Binary Classification

Neuron 1: $a_1 \approx 0.269 < 0.50 \implies \mathbf{0 \text{ (Inactive / Negative)}}$
Neuron 2: $a_2 \approx 0.731 \ge 0.50 \implies \mathbf{1 \text{ (Active / Positive)}}$

The layer produces the binary classification vector:

\begin{bmatrix} 0 \\ 1 \end{bmatrix}

Part 2: Applied Scenario: The VC Investment Predictor

In Topics 1 and 2, our Venture Capital firm evaluated startup pitch decks across four features: [Team Experience, Market Size, Competition, Risk]. Two startups submitted pitch decks:

x_{\text{OmniFlow}} = \begin{bmatrix} 1.0 \\ 0.3 \\ 1.0 \\ 0.2 \end{bmatrix} \qquad\qquad x_{\text{Solaris}} = \begin{bmatrix} 0.3 \\ 1.0 \\ 0.0 \\ 0.8 \end{bmatrix}

In Topic 2, the firm evaluated them using only one question: Unicorn Potential ( $w_1 = [4.0, 2.0, -3.0, 1.0]^T$ ). Solaris AI passed ( $81.8\%$ ), but OmniFlow was rejected ( $33.2\%$ ).

However, evaluating on a single question misses critical opportunities. To evaluate startups from multiple angles, the firm asks three distinct questions:

Unicorn Potential (Neuron 1): Can this become a billion-dollar standalone giant?
Strategic Acquisition (Neuron 2): Would a tech giant acquire this product and team?
Category Leadership (Neuron 3): Can this company become the market leader in its space?

To evaluate these questions, the firm assigns three separate weight vectors:

w_{\text{Unicorn}} = \begin{bmatrix} 4.0 \\ 2.0 \\ -3.0 \\ 1.0 \end{bmatrix} \qquad\qquad w_{\text{Acquisition}} = \begin{bmatrix} 3.0 \\ 1.0 \\ -2.0 \\ -3.0 \end{bmatrix} \qquad\qquad w_{\text{Leadership}} = \begin{bmatrix} 2.0 \\ 1.0 \\ -1.0 \\ -2.0 \end{bmatrix}

Problem 7: Weight Matrix Assembly and Parallel Raw Scores

Part A: Stack the three criteria weight vectors into a single Weight Matrix $W$ . State the matrix shape ( $\text{rows} \times \text{columns}$ ).
Part B: Calculate $Wx_{\text{OmniFlow}}$ for OmniFlow.
Part C: Calculate $Wx_{\text{Solaris}}$ for Solaris AI.

Reveal Solution

Part A: Stacking the three transposed weight vectors as rows produces the $3 \times 4$ Weight Matrix $W$ :

W = \begin{bmatrix} w_{\text{Unicorn}}^T \\ w_{\text{Acquisition}}^T \\ w_{\text{Leadership}}^T \end{bmatrix} = \begin{bmatrix} 4.0 & 2.0 & -3.0 & 1.0 \\ 3.0 & 1.0 & -2.0 & -3.0 \\ 2.0 & 1.0 & -1.0 & -2.0 \end{bmatrix}

Shape: 3 rows and 4 columns ( $3 \times 4$ ), corresponding to 3 evaluation criteria and 4 startup features.

Part B: Multiplying $W$ by $x_{\text{OmniFlow}} = [1.0, 0.3, 1.0, 0.2]^T$ :

Wx_{\text{OmniFlow}} = \begin{bmatrix} (4.0)(1.0) + (2.0)(0.3) + (-3.0)(1.0) + (1.0)(0.2) \\ (3.0)(1.0) + (1.0)(0.3) + (-2.0)(1.0) + (-3.0)(0.2) \\ (2.0)(1.0) + (1.0)(0.3) + (-1.0)(1.0) + (-2.0)(0.2) \end{bmatrix}

Row 1 (Unicorn): $4.0 + 0.6 - 3.0 + 0.2 = \mathbf{+1.8}$
Row 2 (Strategic Acquisition): $3.0 + 0.3 - 2.0 - 0.6 = \mathbf{+0.7}$
Row 3 (Category Leadership): $2.0 + 0.3 - 1.0 - 0.4 = \mathbf{+0.9}$

Wx_{\text{OmniFlow}} = \begin{bmatrix} +1.8 \\ +0.7 \\ +0.9 \end{bmatrix}

Part C: Multiplying $W$ by $x_{\text{Solaris}} = [0.3, 1.0, 0.0, 0.8]^T$ :

Wx_{\text{Solaris}} = \begin{bmatrix} (4.0)(0.3) + (2.0)(1.0) + (-3.0)(0.0) + (1.0)(0.8) \\ (3.0)(0.3) + (1.0)(1.0) + (-2.0)(0.0) + (-3.0)(0.8) \\ (2.0)(0.3) + (1.0)(1.0) + (-1.0)(0.0) + (-2.0)(0.8) \end{bmatrix}

Row 1 (Unicorn): $1.2 + 2.0 - 0.0 + 0.8 = \mathbf{+4.0}$
Row 2 (Strategic Acquisition): $0.9 + 1.0 - 0.0 - 2.4 = \mathbf{-0.5}$
Row 3 (Category Leadership): $0.6 + 1.0 - 0.0 - 1.6 = \mathbf{0.0}$

Wx_{\text{Solaris}} = \begin{bmatrix} +4.0 \\ -0.5 \\ 0.0 \end{bmatrix}

Problem 8: Bias Vector Offsets and the Multi-Neuron Linear Step

The firm defines its baseline Bias Vector across the three criteria:

b = \begin{bmatrix} b_{\text{Unicorn}} \\ b_{\text{Acquisition}} \\ b_{\text{Leadership}} \end{bmatrix} = \begin{bmatrix} -2.5 \\ 0.0 \\ 0.0 \end{bmatrix}

Part A: Compute $z_{\text{OmniFlow}} = Wx_{\text{OmniFlow}} + b$ .
Part B: Compute $z_{\text{Solaris}} = Wx_{\text{Solaris}} + b$ .
Part C: Why is Neuron 1's bias set to $-2.5$ while Neurons 2 and 3 have a bias of $0.0$ ? What real-world market reality does this asymmetry reflect?

Reveal Solution

Part A:

z_{\text{OmniFlow}} = \begin{bmatrix} +1.8 \\ +0.7 \\ +0.9 \end{bmatrix} + \begin{bmatrix} -2.5 \\ 0.0 \\ 0.0 \end{bmatrix} = \begin{bmatrix} \mathbf{-0.7} \\ \mathbf{+0.7} \\ \mathbf{+0.9} \end{bmatrix}

Part B:

z_{\text{Solaris}} = \begin{bmatrix} +4.0 \\ -0.5 \\ 0.0 \end{bmatrix} + \begin{bmatrix} -2.5 \\ 0.0 \\ 0.0 \end{bmatrix} = \begin{bmatrix} \mathbf{+1.5} \\ \mathbf{-0.5} \\ \mathbf{0.0} \end{bmatrix}

Part C: A macroeconomic downturn specifically impairs the market for massive, speculative billion-dollar initial public offerings (raising a steep $-2.5$ hurdle on Unicorn Potential). However, corporate acquisitions of strategic talent ( $b_2 = 0.0$ ) and market demand for focused category leadership ( $b_3 = 0.0$ ) maintain steady baseline demand regardless of interest rate cycles. Independent biases allow the model to adjust thresholds for each question without altering the others.

Problem 9: Element-Wise Sigmoid and Independent Classifications

Using the Sigmoid formula $\sigma(z) = \frac{1}{1 + e^{-z}}$ and the reference table below:

$\text{Reference: } e^{-1.5} \approx 0.2231, \quad e^{-0.9} \approx 0.4066, \quad e^{-0.7} \approx 0.4966, \quad e^{-0.5} \approx 0.6065, \quad e^{0.0} = 1.0000, \quad e^{0.5} \approx 1.6487, \quad e^{0.7} \approx 2.0138$

Part A: Compute $a_{\text{OmniFlow}} = \sigma(z_{\text{OmniFlow}})$ .
Part B: Compute $a_{\text{Solaris}} = \sigma(z_{\text{Solaris}})$ .
Part C: Apply the standard $0.50$ decision threshold ( $a_i \ge 0.50 \to \text{Yes}$ , $a_i < 0.50 \to \text{No}$ ) to determine all three binary classifications for both startups.

Reveal Solution

Part A: OmniFlow Activations

Neuron 1 ( $z_1 = -0.7$ ): $\sigma(-0.7) = \frac{1}{1 + e^{0.7}} = \frac{1}{1 + 2.0138} = \frac{1}{3.0138} \approx \mathbf{0.332} \quad (\mathbf{33.2\%})$
Neuron 2 ( $z_2 = +0.7$ ): $\sigma(+0.7) = \frac{1}{1 + e^{-0.7}} = \frac{1}{1 + 0.4966} = \frac{1}{1.4966} \approx \mathbf{0.668} \quad (\mathbf{66.8\%})$
Neuron 3 ( $z_3 = +0.9$ ): $\sigma(+0.9) = \frac{1}{1 + e^{-0.9}} = \frac{1}{1 + 0.4066} = \frac{1}{1.4066} \approx \mathbf{0.711} \quad (\mathbf{71.1\%})$

a_{\text{OmniFlow}} \approx \begin{bmatrix} \mathbf{0.332} \\ \mathbf{0.668} \\ \mathbf{0.711} \end{bmatrix}

Part B: Solaris AI Activations

Neuron 1 ( $z_1 = +1.5$ ): $\sigma(+1.5) = \frac{1}{1 + e^{-1.5}} = \frac{1}{1 + 0.2231} = \frac{1}{1.2231} \approx \mathbf{0.818} \quad (\mathbf{81.8\%})$
Neuron 2 ( $z_2 = -0.5$ ): $\sigma(-0.5) = \frac{1}{1 + e^{0.5}} = \frac{1}{1 + 1.6487} = \frac{1}{2.6487} \approx \mathbf{0.378} \quad (\mathbf{37.8\%})$
Neuron 3 ( $z_3 = 0.0$ ): $\sigma(0.0) = \frac{1}{1 + e^{0.0}} = \frac{1}{1 + 1.0000} = \frac{1}{2} = \mathbf{0.500} \quad (\mathbf{50.0\%})$

a_{\text{Solaris}} \approx \begin{bmatrix} \mathbf{0.818} \\ \mathbf{0.378} \\ \mathbf{0.500} \end{bmatrix}

Part C: Independent Classifications

Startup	Unicorn Potential (Neuron 1)	Strategic Acquisition (Neuron 2)	Category Leadership (Neuron 3)
OmniFlow	🔴 No (33.2%)	🟢 Yes (66.8%)	🟢 Yes (71.1%)
Solaris AI	🟢 Yes (81.8%)	🔴 No (37.8%)	🟢 Yes (50.0% Break-Even)

Notice how the classifications operate in complete independence: neither startup is universally accepted or universally rejected across all three neurons.

Problem 10: Layer Synthesis and The Investment Dilemma

Part A: State the single unified formula that summarizes this layer's forward pass, and specify the dimensions of every component for this VC screening model.
Part B: Compare the resulting strategic profiles of OmniFlow and Solaris AI.
Part C: The Investment Committee must decide: "Which single startup should receive the firm's lead investment check?" Can this multi-neuron layer make that final choice on its own? Explain why or why not, and identify what architectural structure must be added to solve the dilemma.

Reveal Solution

Part A: The unified multi-neuron forward pass formula:

a = \sigma(Wx + b)

Component dimensions for this model:

Input vector $x$ : $4 \times 1$
Weight matrix $W$ : $3 \times 4$
Bias vector $b$ : $3 \times 1$
Linear score vector $z$ : $3 \times 1$
Activation vector $a$ : $3 \times 1$

Part B:

OmniFlow is rejected as a standalone unicorn ( $33.2\%$ ) due to heavy competition and a small market, but qualifies strongly as a Strategic Acquisition ( $66.8\%$ ) and a Category Leader ( $71.1\%$ ) because seasoned repeat founders execute reliably and dominate their specific vertical.
Solaris AI is a classic Unicorn Moonshot ( $81.8\%$ ) due to an immense untapped market with zero competition, but fails as a strategic acquisition ( $37.8\%$ ) because unproven first-time founders and high technical failure risk scare off corporate acquirers. It lands right on the break-even tipping point for category leadership ( $50.0\%$ ).

Part C: No, this layer cannot decide which startup gets the investment check.

Neurons within a single layer operate in total isolation: each neuron computes its score independently and has no awareness of the other two neurons. The layer produces three parallel facts, but cannot weigh the trade-offs between competing outcomes (e.g., Is a high-risk 81.8% unicorn moonshot better or worse than a safe 71.1% vertical category leader?).

To automate the final binary decision (Fund vs. Pass), the 3 activation outputs of this layer must become the inputs to a subsequent layer whose weights learn how to prioritize and synthesize those competing criteria into a single decision. Parallel neurons gave the network width; automating the final verdict requires depth—stacking layers into a Multi-Layer Neural Network in Topic 4.