The Rectified Linear Unit Activation

The Sigmoid function provides bounded probabilities between $0$ and $1$, making it ideal for the final output layer of binary classification networks.

However, inside deep neural networks with many hidden layers, Sigmoid has two major drawbacks:

1. Computational Cost: Calculating exponents ($e^{-z}$) and divisions for millions of neurons across billions of parameters is computationally expensive.
2. Saturation & Vanishing Gradients: When $z$ is large positive ($+8$) or large negative ($-8$), the Sigmoid curve becomes nearly flat. The slope (derivative) approaches zero, which stalls learning during backpropagation.

To solve this in hidden layers, modern AI architectures predominantly rely on the Rectified Linear Unit (ReLU).

---

Formal Definition of ReLU

The Rectified Linear Unit is defined piecewise:

$$\text{ReLU}(z) = \max(0, z) = \begin{cases} z & \text{if } z > 0 \\ 0 & \text{if } z \le 0 \end{cases}$$

---

How ReLU Operates in Neural Networks

The Rectified Linear Unit functions as an exact mathematical threshold gate with three critical properties:

1. Hard Sparsity for Negative Inputs ($z \le 0$): Any pre-activation score less than or equal to zero is clamped strictly to $0.0$. In multi-layer networks, this silences inactive or unhelpful feature channels, creating sparse representations that isolate distinct patterns.
2. Identity Pass-Through for Positive Inputs ($z > 0$): Positive pre-activations pass through completely unscaled ($\text{ReLU}(z) = z$). Because the slope for positive values is always constant ($\frac{d}{dz}\text{ReLU}(z) = 1$), ReLU completely eliminates the vanishing gradient saturation problem that affects deep Sigmoid networks.
3. Hardware Execution Efficiency: Evaluating $\max(0, z)$ requires only a single hardware comparison (if z > 0 return z else return 0), executing orders of magnitude faster on modern GPUs than exponential transcendental functions like $e^{-z}$.

Side-by-Side Comparison: Sigmoid vs. ReLU

Architectural Property	Sigmoid ($\sigma(z)$)	ReLU ($\text{ReLU}(z)$)
Mathematical Formula	$\frac{1}{1 + e^{-z}}$	$\max(0, z)$
Output Range	$(0, 1)$	$[0, +\infty)$
Negative Inputs ($z < 0$)	Smooth decay toward $0.0$	Clamped strictly to $0.0$ (hard sparsity)
Zero Input ($z = 0$)	Exactly $0.5$	Exactly $0.0$
Positive Inputs ($z > 0$)	Saturates asymptotically at $1.0$	Linear identity pass-through ($z$)
Primary Architectural Role	Binary Output Layer (Probabilities)	Hidden Representation Layers

To see why modern architectures deploy different activation functions across output and hidden layers, examine the mathematical curves of Sigmoid and ReLU side-by-side:

Sigmoid provides bounded probabilities strictly contained in $(0, 1)$, making it the standard choice for binary classification output layers. However, in deep hidden layers, its flat saturation regions cause vanishing gradients. In contrast, ReLU's constant unit slope ($\frac{d}{dz} = 1$) for all $z > 0$ preserves gradient flow during backpropagation, while its hard zero clamping ($z \le 0 \implies 0$) prunes inactive feature channels to produce sparse representations.

---