Sigmoid Activation Function

An Artificial Neuron executes its forward pass in two consecutive computational stages:

The Two Stages of an Artificial Neuron

1. Stage 1: The Linear Step ($z$) — It calculates the dot product of input features and weights, then adds the baseline bias: $z = w \cdot x + b$. (You already know how to do this from Lesson 1).
2. Stage 2: The Activation Function — Transforms the raw linear score into a standardized format.

In Lesson 1, our linear step produced scores of $z = +7.5$ for 'Die Hard in Space' and $z = -3.5$ for 'The Notebook 2'.

What Does a Score of +7.5 Mean?

If you hand $+7.5$ to a studio executive, their immediate question is: "7.5 out of what? " Is $+7.5$ a guaranteed blockbuster? What if the score in a larger model is $+10,000$? What if another film script evaluated with 50 features scores $+320$?

On its own, a raw number like $+7.5$ or $-3.5$ has no standardized scale:

• It has no upper or lower bound ($z \in (-\infty, +\infty)$).
• It cannot be interpreted as a percentage or probability.
• It cannot be compared across different models.

To solve this, the neuron passes the linear score ($z$) through an Activation Function—a mathematical transformation that converts the raw score into a standardized, usable format.

The Sigmoid Activation Function

For binary classification problems—where the model must predict one of the two possible outcomes (like Hit vs. Flop)—the widely used activation function is the Sigmoid function, denoted by $\sigma$ (sigma):

Where $e$ is the mathematical constant - Euler's number ($e \approx 2.718$).

Looking at the curve reveals four key mathematical properties:

1. Outputs a Valid Probability (The Primary Purpose): This is Sigmoid’s core job. No matter how large or small the input score $z$ is, $\sigma(z)$ always stays strictly between $0$ and $1$. This makes its output directly interpretable as a probability.
2. Symmetric Midpoint at $z = 0$: When the input score is exactly zero, $\sigma(0) = \mathbf{0.50}$ ($50\%$).
3. Directional Sensitivity: Positive inputs ($z > 0$) yield outputs above $0.50$ (climbing toward $1.0$). Negative inputs ($z < 0$) yield outputs below $0.50$ (falling toward $0.0$).
4. Saturation (Diminishing Returns): Near $z = 0$, the curve is steep—small changes in $z$ produce significant changes in probability. But beyond $\pm 4$, the curve flattens into plateaus. Once $|z|$ is large, extreme numbers produce diminishing returns, preventing outliers from blowing up downstream calculations.

The Binary Decision Rule

In binary classification, the model chooses between two possible outcomes: $1$ (like Hit) or $0$ (like Flop).

Because $\sigma(z)$ outputs a value between $0$ and $1$, its output is directly read as the predicted probability of a Hit (ranging from $0\%$ to $100\%$). The standard decision threshold is set at $0.50$ ($50\%$):

• If $\sigma(z) \ge 0.50$ (which corresponds to $z \ge 0$), predict $1$ (Hit).
• If $\sigma(z) < 0.50$ (which corresponds to $z < 0$), predict $0$ (Flop).

Applying Sigmoid to Our Movie Predictor

Now we pass our linear scores from Lesson 1 through $\sigma(z)$ to get probabilities:

• 🎬 'Die Hard in Space' ($z = +7.5$):

  $$\sigma(+7.5) = \frac{1}{1 + e^{-7.5}} \approx \mathbf{0.999} \quad (\mathbf{99.9\%})$$

  Because the probability is $99.9\% \ge 50\%$, the neuron classifies it as a Hit.

• 🎬 'The Notebook 2' ($z = -3.5$):

  $$\sigma(-3.5) = \frac{1}{1 + e^{-(-3.5)}} = \frac{1}{1 + e^{3.5}} \approx \mathbf{0.029} \quad (\mathbf{2.9\%})$$

  Because the probability is $2.9\% < 50\%$, the neuron classifies it as a Flop. An output of $0.029$ doesn't mean a $2.9\%$ chance of a Flop—it means there is only a $2.9\%$ chance of a Hit, leaving a $97.1\%$ chance of a Flop.

In two simple steps—a linear combination followed by a non-linear activation—the artificial neuron has converted raw script feature values into a Hit probability.