Learn
A4.3.8 | ARTIFICIAL NEURAL NETWORKS (HL ONLY)
01 | FROM INPUT NUMBERS TO A LEARNED PATTERN
A program receives an image and predicts a digit, or receives delivery features and estimates a duration. The relationship between the inputs and the answer may be too complicated for a single straight-line rule.
An artificial neural network (ANN) is a model made from connected computational units, often called artificial neurons or nodes. Units combine input values using weights and biases, then apply activation functions. Training adjusts parameters so the network can model useful patterns.
ANNs are inspired by aspects of connected biological neurons, but the model is a mathematical computation, not a literal brain or proof of human-like understanding. This HL lesson builds a single unit first, then a multi-layer network.
02 | INPUTS MUST BE REPRESENTED NUMERICALLY
An input is a numeric value supplied to the model. It could be a pixel intensity, a scaled distance or an encoded category. A vector is simply an ordered collection of those values.
The same input positions must retain the same meanings during training and later prediction. Swapping mass and distance columns, or changing the units silently, changes what the learned weights are applied to.
Data cleaning, appropriate scaling and a meaningful representation still matter. A neural network does not automatically repair incorrect labels, missing context or an unrepresentative dataset.
03 | A SINGLE UNIT HAS A SMALL COMPUTATION
For two inputs, begin with z = w₁x₁ + w₂x₂ + b. Each x is an input, each w is a weight and b is the bias. The value z is the pre-activation weighted sum.
Next compute a = f(z), where f is the activation function and a is the unit’s output. In a hidden layer, that output becomes an input to later units.
A classic perceptron uses a threshold or step activation to output a class decision. Modern network units often use other activations, such as ReLU or sigmoid. The weighted-sum-plus-activation structure connects these examples.
04 | SKETCH THE INPUTS, WEIGHTS, BIAS, ACTIVATION AND OUTPUT
In an exam sketch, label the connections as weights and identify the bias separately. Do not draw input values directly becoming an output without the combination and activation steps.
05 | WEIGHTS CONTROL EACH CONNECTION’S CONTRIBUTION
A weight multiplies an input before it contributes to the sum. A positive weight increases z as that input increases, holding the other terms fixed; a negative weight decreases it. A zero weight removes that input’s direct contribution to this unit.
For x₁ = 2 with w₁ = 0.6, the contribution is 1.2. For x₂ = 1 with w₂ = −0.4, the contribution is −0.4. Add contributions rather than treating a negative weight as a missing connection.
Weight magnitude must be interpreted in context. Different feature scales and paths through a network mean the largest individual weight is not automatically a global importance or causal score.
06 | BIAS SHIFTS THE UNIT’S RESPONSE
A bias is an added parameter, separate from the measured inputs. It shifts the pre-activation sum and therefore changes when or how the unit responds.
With contributions 1.2 and −0.4, the sum before bias is 0.8. Adding b = −0.3 gives z = 0.5. If the bias were −1.3 instead, z would become −0.5.
Bias here is a mathematical offset. It is not the same meaning as social bias, although a trained network can also produce unfair outcomes for other reasons. In a network, each ordinary hidden or output unit can have its own bias.
07 | A STEP ACTIVATION MAKES A THRESHOLD DECISION
For this lesson’s step function, f(z) = 1 if z ≥ 0, otherwise 0. At z = 0.5, the perceptron outputs 1. At z = −0.5, it outputs 0. At exactly zero, this stated convention outputs 1.
The output labels 0 and 1 could encode two categories. They are not automatically calibrated probabilities. A step output of 1 is a decision under the rule, not evidence of certainty.
The equality boundary is a design convention. State it clearly when tracing calculations. A different threshold convention can change an example exactly on the boundary.
08 | OTHER ACTIVATIONS PRODUCE DIFFERENT OUTPUTS
| Activation | Rule or behaviour | Typical role and limit |
|---|---|---|
| Step | Outputs 0 or 1 at a defined threshold | Transparent perceptron illustration; not normally the hidden activation used with ordinary gradient-based training |
| ReLU | max(0,z) | A common hidden-layer activation; output is not a probability |
| Sigmoid | 1/(1+e^(−z)), giving a value between 0 and 1 | Often a binary-output score; probability interpretation needs suitable training and evaluation |
| Linear / identity | f(z)=z | Useful for an unrestricted numerical regression output |
For z = 0.5, ReLU gives 0.5 and sigmoid gives approximately 0.622. The step function gives 1. The same weighted sum can therefore lead to different outputs under different activations.
An activation does more than simply decide whether all units are “on” or “off”. Many networks pass continuous activation values between layers. Activation choice should fit the model and task.
09 | TRACE A SINGLE UNIT INTERACTIVELY
Change the inputs, weights, bias and activation. The activity shows each multiplication, the weighted sum and the output. It performs a forward calculation with values you choose; it does not train or update weights automatically.
Start with the defaults and compare activations without changing z. Then change only the bias or the sign of a weight. A single traced example cannot establish how accurately a fitted network performs on a dataset.
10 | ONE PERCEPTRON HAS A LIMITED BOUNDARY
For two inputs, the condition w₁x₁ + w₂x₂ + b = 0 describes a straight boundary when the weights are not both zero. A threshold perceptron separates the two sides into its output categories.
Some patterns can be separated by one straight line. For binary inputs, AND can use weights 1 and 1 with bias −1.5: only (1,1) produces a nonnegative sum.
Other patterns cannot. The XOR pattern requires output 1 for (0,1) and (1,0), but 0 for (0,0) and (1,1). A single straight boundary cannot separate those two diagonal pairs in the original input plane.
11 | CONNECT UNITS INTO LAYERS
A multi-layer perceptron (MLP) is a feedforward neural network with an input layer, one or more hidden layers and an output layer. Information moves through successive computations toward the output.
The input layer presents the feature values. A hidden layer computes intermediate representations. The output layer produces the task’s final score or numerical prediction.
“Hidden” means intermediate rather than directly supplied as input or observed as the target. It does not mean the values are impossible to inspect. The XOR activity below shows every hidden value.
12 | SKETCH A MULTI-LAYER PERCEPTRON
In this fully connected example, every unit in one layer connects to each unit in the next. Other network architectures use different connectivity. The drawing shows the computation structure, not the number of records in the dataset.
13 | A HIDDEN OUTPUT BECOMES A NEW INPUT
The hidden unit h₁ receives x₁ and x₂, uses its own weights and bias, then produces an activation value. Hidden unit h₂ does the same with a different parameter set.
The output unit then treats h₁ and h₂ as its inputs. It does not normally recompute the original input-layer formula unchanged. Its own weights, bias and activation combine the hidden representation.
This is forward propagation: pass input values through the fitted computations. Forward propagation can happen during training to calculate a loss, and during inference to make a prediction. By itself it does not adjust the parameters.
14 | NONLINEAR ACTIVATIONS MAKE THE COMPOSITION MORE EXPRESSIVE
Stacking weighted sums and biases without nonlinear activations still gives an overall linear/affine mapping. For example, h = 2x + 1 followed by y = 3h + 4 simplifies to y = 6x + 7. Adding those two linear stages has not created a curved relationship.
Nonlinear hidden activations allow layers to combine intermediate patterns into relationships that a single linear boundary cannot express. ReLU is piecewise linear but nonlinear overall; step and sigmoid are also nonlinear.
More layers do not automatically discover useful structure. The architecture must be trained effectively with suitable data, and excessive capacity can overfit. The key is the composition of learned transformations and nonlinear activations, not simply a larger diagram.
15 | A SMALL NETWORK CAN REPRESENT XOR
| x₁ | x₂ | Wanted XOR output |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
Use two hidden step units: h₁ = step(x₁ + x₂ − 0.5) detects at least one input being 1. h₂ = step(x₁ + x₂ − 1.5) detects both inputs being 1.
The output is ŷ = step(h₁ − 2h₂ − 0.5). It combines “at least one” with a negative contribution for “both”, giving exactly one input on.
These parameter values are deliberately supplied to illustrate representation. The next activity calculates them; it does not claim to learn XOR from data or demonstrate gradient training through step activations.
16 | TRACE THE HIDDEN LAYER FOR ONE CASE
| Stage for x₁=1, x₂=1 | Pre-activation sum | Step output |
|---|---|---|
| Hidden h₁ | 1 + 1 − 0.5 = 1.5 | 1 |
| Hidden h₂ | 1 + 1 − 1.5 = 0.5 | 1 |
| Output | 1 − 2×1 − 0.5 = −1.5 | 0 |
Both hidden units output 1, but the final output is 0 because the second hidden value has output weight −2. A network does not simply vote among neurons or count how many are active.
For (1,0), h₁ is 1 and h₂ is 0, so the output sum is 1 − 0 − 0.5 = 0.5 and the result is 1. The hidden representation has made the desired distinction possible.
17 | INSPECT ALL FOUR XOR CASES
Select the two binary inputs and follow every sum. The full table shows how the fixed two-unit hidden layer produces the target pattern.
The purpose is to see a multi-layer computation model a pattern a single threshold unit cannot separate in the original inputs. It is not evidence that every larger network is accurate or that the supplied weights are the only solution.
18 | TRAINING CHANGES WEIGHTS AND BIASES
For supervised training, examples pair inputs with known outcomes. First perform a forward pass, then use a loss function to measure how the prediction differs from the target.
A training method adjusts weights and biases to reduce the training objective. In many modern ANNs, backpropagation computes how parameters influence the loss, and an optimiser uses that information to update them.
The loss function is not the activation function. The activation transforms a unit’s sum; the loss evaluates predictions against targets. Training uses many examples and repeated updates, not a guarantee that each update improves every example. No calculus or backpropagation derivation is needed for this outline objective.
19 | MATCH THE OUTPUT TO THE TASK
| Task | Possible network output | Interpretation |
|---|---|---|
| Binary classification | One sigmoid output plus a defined decision threshold | A score used to choose between two categories; calibration needs checking |
| Mutually exclusive multiclass classification | One score per class, often with softmax | Relative class probabilities under the model; choose using an appropriate rule |
| Regression | One or more numerical outputs, often linear | Estimated measured quantities, with task-appropriate units and constraints |
| Pattern recognition | Outputs or representations linked to recognised patterns | The task may involve classification or another prediction objective |
A network with one output is not limited to one class: one binary score can distinguish two categories. Conversely, several outputs can represent multiple numerical targets rather than classes. Define the task and encoding.
Softmax makes a vector of nonnegative values summing to 1 for a mutually exclusive class model. That mathematical format does not guarantee that the network’s confidence matches real-world correctness.
20 | COUNT THE PARAMETERS, NOT THE INPUT RECORDS
In the illustrated 2–2–1 fully connected network, input-to-hidden connections give 2 × 2 = 4 weights. Hidden-to-output connections give 2 × 1 = 2 more. There are also two hidden biases and one output bias.
Total trainable parameters = 4 + 2 + 2 + 1 = 9, under this stated architecture. The input nodes simply present values and do not each have an extra learned bias here.
Layer widths and counts are architecture choices or hyperparameters; weights and biases are normally fitted parameters. More units can increase capacity, memory and computation, but the number of training examples is a separate quantity.
21 | COMPLEX PATTERNS COME FROM COMBINED REPRESENTATIONS
Hidden layers can learn combinations useful for a task rather than being explicitly given every intermediate concept. Later layers combine earlier activations, allowing a compact computation to model complicated relationships.
For an image task, intermediate features may respond to local or combined patterns, depending on the architecture and data. Do not assume every hidden unit has a simple human-readable meaning or that an ordinary MLP preserves spatial layout automatically.
The next lesson introduces CNNs, whose structure is designed to exploit spatial relationships. The present outline concerns the common input, learned transformations, nonlinear activations and output structure.
22 | GENERALISATION STILL NEEDS EVIDENCE
A large ANN can fit training patterns, including incorrect labels or accidental correlations. Strong training performance alone does not show useful performance on genuinely new inputs.
Use suitable validation to choose architecture and training settings, preserve an independent final test, and fit preprocessing within the training process. Examine task metrics, data coverage and important errors.
Networks also have limits in interpretability, compute use and sensitivity to changed conditions. A high class score is not certainty, and biological inspiration does not remove the need to check fairness, reliability or the meaning of the objective.
23 | CHECK THE PARTS AND THE PROCESS
Choose the concept that best matches each statement. Distinguish values flowing forward from parameters being fitted.
24 | OUTLINE THE NETWORK CLEARLY
For a single unit, outline the sequence: numerical inputs → weighted contributions → added bias → activation function → output. Label each element in the sketch.
For an MLP, identify the input layer, one or more hidden layers and the output layer. Explain that hidden activations become inputs to the next layer, and that training adjusts connection weights and biases to model useful patterns.
Connect structure to function: nonlinear hidden transformations allow combinations more expressive than a single linear boundary, supporting classification, regression and pattern recognition. Include the limit that architecture alone does not prove accurate generalisation.
Further reading: weighted sums, biases and activations in a dense layer and feedforward composition and nonlinear hidden layers.