Limitations of Single-Layer Networks

In Topic 3, we scaled our single neuron into a parallel dense layer. By stacking multiple weight vectors into a weight matrix $W \in \mathbb{R}^{m \times n}$ and adding a bias vector $b \in \mathbb{R}^m$, our neural network learned to evaluate multiple distinct perspectives on the same input vector at the exact same time:

$$z = W x + b \qquad \longrightarrow \qquad a = \sigma(z)$$

This granted our network Layer Width—the capacity to make several simultaneous decisions.

However, a single-layer network possesses a fundamental architectural limitation: it can only map raw input features directly to final output decisions.

In real-world data, decisions rarely depend on isolated features acting alone. To solve complex tasks, intelligent systems must recognize how individual features combine, interact, and modify one another.

---

The Isolated Feature Evaluation Constraint

To understand why a single layer struggles with complex decisions, examine the mathematical anatomy of a single neuron's linear sum:

$$z_i = \sum_{j=1}^n W_{ij} x_j + b_i = W_{i1} x_1 + W_{i2} x_2 + \dots + W_{in} x_n + b_i$$

Notice how each feature value $x_j$ enters the equation:

• Feature $x_1$ contributes exactly $W_{i1} x_1$, regardless of whether $x_2$ is $0.0$ or $100.0$.
• Feature $x_2$ contributes exactly $W_{i2} x_2$, regardless of whether $x_1$ is positive or negative.

Every feature value is evaluated in complete isolation. The neuron multiplies each feature by its dedicated weight and sums them together. It cannot evaluate conditionality, synergy, or feature combinations.

---

The Compound Feature Dilemma

In our running Netflix Greenlight Predictor, our screenplay feature vector $x \in \mathbb{R}^4$ captures four locked genre dimensions [Action, Romance, Comedy, Sci-Fi]:

$$x = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \end{bmatrix} \begin{matrix} \text{(Action)} \\ \text{(Romance)} \\ \text{(Comedy)} \\ \text{(Sci-Fi)} \end{matrix}$$

Consider how human audiences and studio executives actually evaluate movie scripts:

• High Action ($x_1 = 1.0$) on its own describes a standard action movie.
• High Sci-Fi ($x_4 = 1.0$) on its own describes a speculative science-fiction film.
• But when high Action ($x_1 = 1.0$) and high Sci-Fi ($x_4 = 1.0$) occur together, they form an entirely new compound concept: the Summer Popcorn Flick.
• Similarly, when high Romance ($x_2 = 1.0$) and high Comedy ($x_3 = 1.0$) occur together, they form a distinct compound concept: the Rom-Com.

A single linear layer cannot reward the specific combination of Action and Sci-Fi without also rewarding movies that contain only Action or only Sci-Fi. If we increase the Action weight ($W_{i1}$) and the Sci-Fi weight ($W_{i4}$), a script with high Action and zero Sci-Fi still receives a massive boost. The single layer cannot express the logical rule: "Boost the score if BOTH Action and Sci-Fi are high, but penalize or ignore scripts where only one is present."

---

The XOR Non-Linear Classification Boundary Failure

This limitation was mathematically formalized in 1969 by Marvin Minsky and Seymour Papert through the classic Exclusive-OR (XOR) problem.

Consider a binary classification task with two input features ($x_1, x_2 \in \{0, 1\}$). The target output $y$ should be $1$ if either $x_1$ or $x_2$ is active, but $0$ if neither or both are active:

Input $x_1$	Input $x_2$	Logical XOR Target ($y$)	Geometric Point $(x_1, x_2)$
$0$	$0$	$0$ (False)	Class $0$ at origin $(0, 0)$
$1$	$0$	$1$ (True)	Class $1$ at $(1, 0)$
$0$	$1$	$1$ (True)	Class $1$ at $(0, 1)$
$1$	$1$	$0$ (False)	Class $0$ at $(1, 1)$

Let's examine what happens when a single linear neuron attempts to solve XOR. A single neuron defines a linear decision boundary:

$$w_1 x_1 + w_2 x_2 + b = 0$$

To classify the data correctly, this single straight line must separate the two Class $1$ points from the two Class $0$ points.

To observe why a single straight line cannot separate the XOR data distribution, examine the geometric classification boundary below:

As the diagram proves, no single straight line in 2D space can separate $(0, 1)$ and $(1, 0)$ from $(0, 0)$ and $(1, 1)$. The positive examples sit on opposite diagonal corners from the negative examples.

---

The Architectural Necessity of Depth

A single-layer network fails on compound features and non-linear patterns because it lacks intermediate computational steps.

To solve non-linear classification problems like XOR and evaluate compound concepts like "Summer Popcorn Flick" or "Rom-Com", a neural network needs Depth.

We must insert intermediate layers of neurons between the raw input feature values and the final output prediction. These intermediate stages are called Hidden Layers.

---