Hidden Layers and Network Geometry

By stacking layers sequentially, we construct a Multi-Layer Perceptron (MLP).

When it comes to AI, a Multi-Layer Perceptron is a feedforward neural network consisting of at least three distinct stages:

1. An Input Layer that receives raw feature values.
2. One or more Hidden Layers that transform incoming signals into intermediate representations.
3. An Output Layer that computes the final prediction.

Architectural Anatomy of an MLP

Let's inspect the structural components of a 2-layer Multi-Layer Perceptron designed for our movie prediction task:

[ Input Layer: x ]          [ Hidden Layer 1: a^(1) ]          [ Output Layer: a^(2) ]
(4 Raw Genre Features)       (2 Intermediate Concepts)           (1 Final Hit Decision)

      (Action)    x1 ───────────► ( h1: Popcorn ) ────────────┐
      (Romance)   x2 ───────────►                         ├───► ( y_hat: Greenlight )
      (Comedy)    x3 ───────────► ( h2: Rom-Com ) ────────────┘
      (Sci-Fi)    x4 ───────────►

1. Input Layer ($x \in \mathbb{R}^4$): Holds the raw feature values [Action, Romance, Comedy, Sci-Fi]. The input layer contains no weights, no biases, and performs no arithmetic; it is simply the entry vector of data.
2. Hidden Layer ($a^{(1)} \in \mathbb{R}^2$): A layer of $2$ parallel neurons. Each hidden neuron receives all $4$ input features, computes an affine sum ($z^{(1)} = W^{(1)}x + b^{(1)}$), and squashes the result through an activation function ($a^{(1)} = f(z^{(1)})$).
3. Output Layer ($a^{(2)} \in \mathbb{R}^1$): A single neuron that receives the $2$ hidden activations ($a^{(1)}$), computes a second affine sum ($z^{(2)} = W^{(2)}a^{(1)} + b^{(2)}$), and applies a final Sigmoid activation to produce the predicted Hit probability ($\hat{y} = a^{(2)}$).

Standard Layer Superscript Notation

To track parameters across multiple layers without mathematical confusion, we use parenthetical superscripts $(l)$ to designate the layer index:

• $W^{(l)}$: The Weight Matrix of Layer $l$.
• $b^{(l)}$: The Bias Vector of Layer $l$.
• $z^{(l)}$: The Pre-activation Linear Sum Vector of Layer $l$.
• $a^{(l)}$: The Post-activation Output Vector of Layer $l$.

For our 2-layer network ($4 \to 2 \to 1$):

$\mathbf{Layer\ 1\ (Hidden\ Layer):} \qquad z^{(1)} = W^{(1)} x + b^{(1)}, \qquad a^{(1)} = \text{ReLU}\left(z^{(1)}\right)$

$\mathbf{Layer\ 2\ (Output\ Layer):} \qquad z^{(2)} = W^{(2)} a^{(1)} + b^{(2)}, \qquad a^{(2)} = \sigma\left(z^{(2)}\right)$

Let's check the matrix dimensions at each layer:

• Layer 1 Weight Matrix ($W^{(1)} \in \mathbb{R}^{2 \times 4}$): $2$ rows (hidden neurons), $4$ columns (input features).
• Layer 1 Bias Vector ($b^{(1)} \in \mathbb{R}^{2 \times 1}$): $2$ baseline hurdle offsets.
• Layer 2 Weight Matrix ($W^{(2)} \in \mathbb{R}^{1 \times 2}$): $1$ row (output neuron), $2$ columns (hidden activation inputs).
• Layer 2 Bias Vector ($b^{(2)} \in \mathbb{R}^{1 \times 1}$): $1$ final output hurdle offset.

Notice the fundamental rule of multi-layer compatibility: the number of columns in $W^{(2)}$ must exactly equal the number of neurons in Layer 1 ($a^{(1)}$).

Why Are They Designated as "Hidden"?

The intermediate layer is called "hidden" for a simple reason: it does not interact directly with the external environment.

• The Input Layer is visible because the human engineer supplies the raw data ($x$).
• The Output Layer is visible because the user reads the final predicted score ($\hat{y}$).
• The Hidden Layer is internal to the model. Its pre-activations ($z^{(1)}$) and post-activations ($a^{(1)}$) exist strictly in the computer's memory registers during computation.

The network is never explicitly told what the hidden neurons must compute. During training, the optimization algorithm adjusts $W^{(1)}$ and $W^{(2)}$ automatically, allowing the hidden layer to discover internal representations that best help the output layer make accurate predictions.

To visualize how the 4-dimensional input features connect to the 2 hidden intermediate neurons and converge into the single output neuron, trace the structural architecture below: