Multi-Layer Networks In Practice

Part 1: The Underlying Mechanics Drill

Work through each problem on paper before revealing the step-by-step solution.

Problem 1: Chained Matrix Evaluation & Linear Collapse

Given an input vector $x = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$ , a first-layer weight matrix $W^{(1)} = \begin{bmatrix} 2 & -1 \\ 3 & 1 \end{bmatrix}$ , and a second-layer weight matrix $W^{(2)} = \begin{bmatrix} 1 & -2 \end{bmatrix}$ (with zero biases and no activation functions):

Part A: Calculate the intermediate vector $z^{(1)} = W^{(1)} x$ .
Part B: Calculate the scalar output $z^{(2)} = W^{(2)} z^{(1)}$ .
Part C: Calculate the composite product matrix $W_{\text{composite}} = W^{(2)} W^{(1)}$ , and compute $W_{\text{composite}} x$ directly. Explain what this reveals about multi-layer linear networks without activation functions.

Reveal Solution

Part A: Calculate $z^{(1)} = W^{(1)} x$

z^{(1)} = \begin{bmatrix} 2 & -1 \\ 3 & 1 \end{bmatrix} \begin{bmatrix} 2 \\ 3 \end{bmatrix} = \begin{bmatrix} (2)(2) + (-1)(3) \\ (3)(2) + (1)(3) \end{bmatrix} = \begin{bmatrix} 4 - 3 \\ 6 + 3 \end{bmatrix} = \begin{bmatrix} 1 \\ 9 \end{bmatrix}

Part B: Calculate $z^{(2)} = W^{(2)} z^{(1)}$

z^{(2)} = \begin{bmatrix} 1 & -2 \end{bmatrix} \begin{bmatrix} 1 \\ 9 \end{bmatrix} = (1)(1) + (-2)(9) = 1 - 18 = \mathbf{-17}

Part C: Composite Matrix Multiplication & Linear Collapse

W_{\text{composite}} = W^{(2)} W^{(1)} = \begin{bmatrix} 1 & -2 \end{bmatrix} \begin{bmatrix} 2 & -1 \\ 3 & 1 \end{bmatrix}

W_{\text{composite}} = \begin{bmatrix} (1)(2) + (-2)(3) & (1)(-1) + (-2)(1) \end{bmatrix} = \begin{bmatrix} 2 - 6 & -1 - 2 \end{bmatrix} = \begin{bmatrix} -4 & -3 \end{bmatrix}

Now compute $W_{\text{composite}} x$ :

W_{\text{composite}} x = \begin{bmatrix} -4 & -3 \end{bmatrix} \begin{bmatrix} 2 \\ 3 \end{bmatrix} = (-4)(2) + (-3)(3) = -8 - 9 = \mathbf{-17}

Pedagogical Insight: Both calculations yield the exact same scalar ( $-17$ ). Without non-linear activation functions between layers, chaining multiple weight matrices mathematically collapses into a single matrix multiplication ( $W_{\text{composite}} = W^{(2)} W^{(1)}$ ). No matter how many linear layers are stacked, a linear network can never compute non-linear feature interactions.

Problem 2: Hidden Activation Vector Calculation with ReLU

A 5-neuron hidden layer receives an input vector and produces the following pre-activation linear sum vector $z^{(1)}$ :

z^{(1)} = \begin{bmatrix} 3.5 \\ -2.4 \\ 0.0 \\ 5.1 \\ -0.8 \end{bmatrix}

Calculate the activated hidden representation vector $a^{(1)} = \text{ReLU}\left(z^{(1)}\right)$ , and state which neuron channels are active vs. dormant.

Reveal Solution

Apply the piecewise definition $\text{ReLU}(z_i) = \max(0, z_i)$ to each coordinate independently:

$a_1^{(1)} = \max(0, 3.5) = \mathbf{3.5}$ (Active Channel)
$a_2^{(1)} = \max(0, -2.4) = \mathbf{0.0}$ (Dormant / Silenced Channel)
$a_3^{(1)} = \max(0, 0.0) = \mathbf{0.0}$ (Dormant / Boundary Channel)
$a_4^{(1)} = \max(0, 5.1) = \mathbf{5.1}$ (Active Channel)
$a_5^{(1)} = \max(0, -0.8) = \mathbf{0.0}$ (Dormant / Silenced Channel)

Assemble the resulting activation vector:

a^{(1)} = \begin{bmatrix} 3.5 \\ 0.0 \\ 0.0 \\ 5.1 \\ 0.0 \end{bmatrix}

Channel Status: Neurons $1$ and $4$ are actively firing and transmitting signals to the downstream layer. Neurons $2$ , $3$ , and $5$ are completely silenced to $0.0$ , generating a sparse representation where only positive evidence propagates forward.

Problem 3: Multi-Layer Dimension Compatibility & Parameter Counting

A deep learning engineer designs a 3-layer neural network with an architecture of $3 \to 4 \to 2 \to 1$ :

Input dimension $n_0 = 3$
Hidden Layer 1 width $n_1 = 4$
Hidden Layer 2 width $n_2 = 2$
Output Layer width $n_3 = 1$
Part A: State the required matrix dimensions for weight matrices $W^{(1)}$ , $W^{(2)}$ , and $W^{(3)}$ .
Part B: State the required vector dimensions for bias vectors $b^{(1)}$ , $b^{(2)}$ , and $b^{(3)}$ .
Part C: State the vector dimensions of the activation states $x$ , $a^{(1)}$ , $a^{(2)}$ , and $a^{(3)}$ .
Part D: Calculate the total number of learnable parameters (weights + biases) across the entire network.

Reveal Solution

Part A: Weight Matrix Dimensions ( $W^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}$ )

Layer 1: $W^{(1)} \in \mathbb{R}^{4 \times 3}$ ( $4$ neurons receiving $3$ inputs)
Layer 2: $W^{(2)} \in \mathbb{R}^{2 \times 4}$ ( $2$ neurons receiving $4$ hidden inputs)
Layer 3: $W^{(3)} \in \mathbb{R}^{1 \times 2}$ ( $1$ neuron receiving $2$ hidden inputs)

Part B: Bias Vector Dimensions ( $b^{(l)} \in \mathbb{R}^{n_l \times 1}$ )

Layer 1: $b^{(1)} \in \mathbb{R}^{4 \times 1}$ ( $4$ baseline hurdles)
Layer 2: $b^{(2)} \in \mathbb{R}^{2 \times 1}$ ( $2$ baseline hurdles)
Layer 3: $b^{(3)} \in \mathbb{R}^{1 \times 1}$ ( $1$ baseline hurdle)

Part C: Activation Vector Dimensions

Input Vector: $x \in \mathbb{R}^{3 \times 1}$
Hidden Layer 1 Activation: $a^{(1)} \in \mathbb{R}^{4 \times 1}$
Hidden Layer 2 Activation: $a^{(2)} \in \mathbb{R}^{2 \times 1}$
Output Layer Activation: $a^{(3)} \in \mathbb{R}^{1 \times 1}$

Part D: Parameter Count Calculation

Layer 1: $(4 \times 3 \text{ weights}) + (4 \text{ biases}) = 12 + 4 = \mathbf{16\text{ parameters}}$
Layer 2: $(2 \times 4 \text{ weights}) + (2 \text{ biases}) = 8 + 2 = \mathbf{10\text{ parameters}}$
Layer 3: $(1 \times 2 \text{ weights}) + (1 \text{ bias}) = 2 + 1 = \mathbf{3\text{ parameters}}$

$\text{Total Parameters} = 16 + 10 + 3 = \mathbf{29\text{ learnable parameters}}$

Problem 4: Final Prediction from Hidden State

An output layer receives a 2-dimensional hidden activation vector $a^{(1)} = \begin{bmatrix} 2.0 \\ 1.5 \end{bmatrix}$ . The output layer is parameterized by weight matrix $W^{(2)} = \begin{bmatrix} 3.0 & -2.0 \end{bmatrix}$ and baseline bias $b^{(2)} = \begin{bmatrix} -1.0 \end{bmatrix}$ .

Part A: Calculate the pre-activation scalar $z^{(2)} = W^{(2)} a^{(1)} + b^{(2)}$ .
Part B: Calculate the Sigmoid probability $a^{(2)} = \sigma(z^{(2)})$ (use $e^{-2.0} \approx 0.1353$ ).
Part C: Explain how the output layer treats the hidden activations $a^{(1)}$ as if they were raw input features.

Reveal Solution

Part A: Calculate $z^{(2)}$

z^{(2)} = W^{(2)} a^{(1)} + b^{(2)} = \begin{bmatrix} 3.0 & -2.0 \end{bmatrix} \begin{bmatrix} 2.0 \\ 1.5 \end{bmatrix} + (-1.0)

z^{(2)} = (3.0 \times 2.0) + (-2.0 \times 1.5) + (-1.0) = 6.0 - 3.0 - 1.0 = \mathbf{+2.0}

Part B: Calculate $a^{(2)} = \sigma(z^{(2)})$

a^{(2)} = \sigma(2.0) = \frac{1}{1 + e^{-2.0}} = \frac{1}{1 + 0.1353} = \frac{1}{1.1353} \approx \mathbf{0.8808} \implies \mathbf{88.1\%}

Part C: Downstream Invariance To the output neuron, the vector $a^{(1)} = [2.0, 1.5]^T$ is structurally indistinguishable from an original input vector $x$ . The output layer has no awareness of how $a^{(1)}$ was created, how many input features originally entered Layer 1, or whether Layer 1 used ReLU or Sigmoid. It simply executes a standard dot product on the numbers presented to it.

Problem 5: Zero Hidden State / Inactive Neuron Propagation

Suppose an input sample produces strong negative pre-activations across all hidden neurons ( $z^{(1)} \le \mathbf{0}$ ), causing the entire hidden activation vector to be clamped to zero: $a^{(1)} = \begin{bmatrix} 0.0 \\ 0.0 \end{bmatrix}$ .

The output layer has weights $W^{(2)} = \begin{bmatrix} 4.0 & 5.0 \end{bmatrix}$ and baseline bias $b^{(2)} = \begin{bmatrix} -1.5 \end{bmatrix}$ .

Part A: Calculate the output pre-activation $z^{(2)} and the final Sigmoid probability$ a^{(2)} = \sigma(z^{(2)}) $(use$ e^{1.5} \approx 4.4817$).
Part B: Explain why the network's final output becomes completely unresponsive to the input features $x$ or Layer 1 weights $W^{(1)}$ under this condition.

Reveal Solution

Part A: Calculate Output Values

z^{(2)} = W^{(2)} a^{(1)} + b^{(2)} = (4.0 \times 0.0) + (5.0 \times 0.0) + (-1.5) = 0.0 + 0.0 - 1.5 = \mathbf{-1.5}

a^{(2)} = \sigma(-1.5) = \frac{1}{1 + e^{-(-1.5)}} = \frac{1}{1 + e^{1.5}} = \frac{1}{1 + 4.4817} = \frac{1}{5.4817} \approx \mathbf{0.1824} \implies \mathbf{18.2\%}

Part B: Mathematical Decoupling Because $a^{(1)} = \mathbf{0}$ , the dot product $W^{(2)} a^{(1)}$ evaluates to exactly zero regardless of the values in $W^{(2)}$ . The output pre-activation collapses strictly to the output bias: $z^{(2)} = 0 + b^{(2)} = b^{(2)}$ .

When all hidden units are dormant, no information from the input vector $x$ reaches the output layer. The network outputs a fixed default baseline probability ( $\sigma(b^{(2)}) = 18.2\%$ ) determined solely by its environmental bias.

Part 2: Applied Scenario: The VC 2-Layer Representation Network

In Topics 1, 2, and 3, our venture capital firm evaluated startup pitch profiles across four locked criteria [Team Experience, Market Size, Competition, Risk]:

x_{\text{OmniFlow}} = \begin{bmatrix} 1.0 \\ 0.3 \\ 1.0 \\ 0.2 \end{bmatrix} \qquad\qquad x_{\text{Solaris}} = \begin{bmatrix} 0.3 \\ 1.0 \\ 0.0 \\ 0.8 \end{bmatrix}

Rather than evaluating raw pitch metrics directly, the investment committee deploys a 2-layer representation network ( $4 \to 2 \to 1$ ).

[ Raw Pitch Features ]         [ Layer 1: Latent Factors ]         [ Layer 2: Decision ]
(Team, Market, Comp, Risk)     (Scalability & Defensibility)      (Term Sheet Probability)

Layer 1 Configuration (Latent Investment Factors):

Hidden Neuron 1 (Scalability Factor - $a_1^{(1)}$ ): Captures total market expansion potential. Heavily rewards Market Size ( $+4.0$ ) and Team Experience ( $+2.0$ ), mildly penalizes technical Risk ( $-1.0$ ), and ignores Competition ( $0.0$ ), with hurdle bias $b_1^{(1)} = -1.0$ .
Hidden Neuron 2 (Defensibility Factor - $a_2^{(1)}$ ): Captures competitive moats and barriers to entry. Heavily rewards Team Pedigree ( $+3.0$ ), severely penalizes crowded Competition ( $-3.0$ ), rewards technical barrier Risk ( $+1.0$ ), and ignores Market Size ( $0.0$ ), with hurdle bias $b_2^{(1)} = -0.5$ .

W^{(1)} = \begin{bmatrix} 2.0 & 4.0 & 0.0 & -1.0 \\ 3.0 & 0.0 & -3.0 & 1.0 \end{bmatrix}, \qquad b^{(1)} = \begin{bmatrix} -1.0 \\ -0.5 \end{bmatrix}, \qquad f^{(1)} = \text{ReLU}

Layer 2 Configuration (Investment Committee Conversion):

The investment committee evaluates the two hidden factors: it rewards Scalability ( $+1.0$ ) and places high premium on Defensibility ( $+1.5$ ), against a strict macro hurdle bias $b^{(2)} = -3.0$ :

W^{(2)} = \begin{bmatrix} 1.0 & 1.5 \end{bmatrix}, \qquad b^{(2)} = \begin{bmatrix} -3.0 \end{bmatrix}, \qquad f^{(2)} = \text{Sigmoid}

To visually trace how both startup pitch vectors propagate through intermediate latent representations into the final term sheet conversion probability, examine the comparative dataflow below:

VC 2-Layer Representation Network Dataflow — Comparative 2-layer forward pass for OmniFlow and Solaris AI: OmniFlow's heavy competition penalty silences its Defensibility factor to 0.0 via ReLU, leaving it with insufficient signal to clear the Layer 2 hurdle ($26.9\%$ Pass). Solaris AI's uncontested market generates both Scalability ($2.8$) and Defensibility ($1.2$), securing an $83.2\%$ Term Sheet Greenlight.

Problem 6: Complete 2-Layer Startup Forward Pass

Part A: Compute the Layer 1 pre-activation vector $z^{(1)} = W^{(1)} x + b^{(1)}$ and the ReLU hidden activation vector $a^{(1)} = \text{ReLU}\left(z^{(1)}\right)$ for both OmniFlow and Solaris AI.
Part B: Compute the Layer 2 scalar pre-activation $z^{(2)} = W^{(2)} a^{(1)} + b^{(2)}$ and the final Sigmoid probability $a^{(2)} = \sigma(z^{(2)})$ for both startups (use $e^{1.0} \approx 2.7183$ and $e^{-1.6} \approx 0.2019$ ).
Part C: State which startup earns the term sheet greenlight ( $\hat{y} \ge 50\%$ ).
Part D: Provide an investment committee synthesis explaining how representation learning in Layer 1 exposed OmniFlow's critical vulnerability compared to Solaris AI.

Reveal Solution

Part A: Layer 1 Forward Calculation

For OmniFlow ( $x = [1.0, 0.3, 1.0, 0.2]^T$ ):

Neuron 1 (Scalability): $z_1^{(1)} = (2.0)(1.0) + (4.0)(0.3) + (0.0)(1.0) + (-1.0)(0.2) + (-1.0)$ $z_1^{(1)} = 2.0 + 1.2 + 0.0 - 0.2 - 1.0 = 3.2 - 1.2 = \mathbf{+2.0}$ $a_1^{(1)} = \text{ReLU}(2.0) = \mathbf{2.0}$
Neuron 2 (Defensibility): $z_2^{(1)} = (3.0)(1.0) + (0.0)(0.3) + (-3.0)(1.0) + (1.0)(0.2) + (-0.5)$ $z_2^{(1)} = 3.0 + 0.0 - 3.0 + 0.2 - 0.5 = 0.2 - 0.5 = \mathbf{-0.3}$ $a_2^{(1)} = \text{ReLU}(-0.3) = \mathbf{0.0}$

z^{(1)}_{\text{OmniFlow}} = \begin{bmatrix} +2.0 \\ -0.3 \end{bmatrix}, \qquad a^{(1)}_{\text{OmniFlow}} = \begin{bmatrix} 2.0 \\ 0.0 \end{bmatrix} \begin{matrix} \text{(Scalability Factor)} \\ \text{(Defensibility Factor)} \end{matrix}

For Solaris AI ( $x = [0.3, 1.0, 0.0, 0.8]^T$ ):

Neuron 1 (Scalability): $z_1^{(1)} = (2.0)(0.3) + (4.0)(1.0) + (0.0)(0.0) + (-1.0)(0.8) + (-1.0)$ $z_1^{(1)} = 0.6 + 4.0 + 0.0 - 0.8 - 1.0 = 4.6 - 1.8 = \mathbf{+2.8}$ $a_1^{(1)} = \text{ReLU}(2.8) = \mathbf{2.8}$
Neuron 2 (Defensibility): $z_2^{(1)} = (3.0)(0.3) + (0.0)(1.0) + (-3.0)(0.0) + (1.0)(0.8) + (-0.5)$ $z_2^{(1)} = 0.9 + 0.0 - 0.0 + 0.8 - 0.5 = 1.7 - 0.5 = \mathbf{+1.2}$ $a_2^{(1)} = \text{ReLU}(1.2) = \mathbf{1.2}$

z^{(1)}_{\text{Solaris}} = \begin{bmatrix} +2.8 \\ +1.2 \end{bmatrix}, \qquad a^{(1)}_{\text{Solaris}} = \begin{bmatrix} 2.8 \\ 1.2 \end{bmatrix} \begin{matrix} \text{(Scalability Factor)} \\ \text{(Defensibility Factor)} \end{matrix}

Part B: Layer 2 Forward Calculation

For OmniFlow ( $a^{(1)} = [2.0, 0.0]^T$ ):

z^{(2)}_{\text{OmniFlow}} = W^{(2)} a^{(1)} + b^{(2)} = (1.0 \times 2.0) + (1.5 \times 0.0) + (-3.0) = 2.0 + 0.0 - 3.0 = \mathbf{-1.0}

a^{(2)}_{\text{OmniFlow}} = \sigma(-1.0) = \frac{1}{1 + e^{-(-1.0)}} = \frac{1}{1 + e^{1.0}} = \frac{1}{1 + 2.7183} = \frac{1}{3.7183} \approx \mathbf{0.2689} \implies \mathbf{26.9\%}

For Solaris AI ( $a^{(1)} = [2.8, 1.2]^T$ ):

z^{(2)}_{\text{Solaris}} = W^{(2)} a^{(1)} + b^{(2)} = (1.0 \times 2.8) + (1.5 \times 1.2) + (-3.0) = 2.8 + 1.8 - 3.0 = 4.6 - 3.0 = \mathbf{+1.6}

a^{(2)}_{\text{Solaris}} = \sigma(1.6) = \frac{1}{1 + e^{-1.6}} = \frac{1}{1 + 0.2019} = \frac{1}{1.2019} \approx \mathbf{0.8320} \implies \mathbf{83.2\%}

Part C: Investment Decision

Solaris AI achieves an $83.2\%$ Term Sheet Greenlight ( $\hat{y} \ge 50\%$ ).
OmniFlow is rejected with a $26.9\%$ Probability ( $\hat{y} < 50\%$ ).

Part D: Strategic Representation Synthesis

The 2-layer architecture reveals why representation learning is critical for robust decisions:

OmniFlow's Hidden Vulnerability: In Layer 1, OmniFlow's veteran team ( $x_1 = 1.0$ ) generated a solid Scalability score ( $a_1^{(1)} = 2.0$ ). However, because OmniFlow operates in an intensely crowded market ( $x_3 = 1.0$ ), its competitive Defensibility score went negative ( $z_2^{(1)} = -0.3$ ). ReLU clamped this channel strictly to $0.0$ . When Layer 2 evaluated OmniFlow, it had only one active factor ( $2.0$ ), which was insufficient to overcome the strict $-3.0$ committee hurdle ( $z^{(2)} = -1.0 \implies 26.9\%$ ).
Solaris AI's Dual-Factor Strength: Solaris AI entered with high technical execution risk ( $x_4 = 0.8$ ) and a modest team score ( $x_1 = 0.3$ ). However, its massive market ( $x_2 = 1.0$ ) and total absence of competition ( $x_3 = 0.0$ ) allowed both hidden neurons to fire strongly ( $a_1^{(1)} = 2.8, a_2^{(1)} = 1.2$ ). When Layer 2 multiplied these intermediate concepts by their committee weights ( $1.0(2.8) + 1.5(1.2) = 4.6$ ), it easily absorbed the $-3.0$ hurdle, securing a decisive $83.2\%$ greenlight.

By utilizing hidden layers, the network avoids making simplistic decisions on raw features, structuring intermediate insights to produce nuanced, defensible predictions.