IB COMPUTER SCIENCE | A4.3.8 Artificial Neural Networks (HL)

Lesson objective

Outline single-neuron computation and multi-layer network structure, including inputs, weights, bias, activations, hidden layers and outputs for complex patterns.

Learn

A4.3.8 | ARTIFICIAL NEURAL NETWORKS (HL ONLY)

01 | FROM INPUT NUMBERS TO A LEARNED PATTERN

A program receives an image and predicts a digit, or receives delivery features and estimates a duration. The relationship between the inputs and the answer may be too complicated for a single straight-line rule.

An artificial neural network (ANN) is a model made from connected computational units, often called artificial neurons or nodes. Units combine input values using weights and biases, then apply activation functions. Training adjusts parameters so the network can model useful patterns.

ANNs are inspired by aspects of connected biological neurons, but the model is a mathematical computation, not a literal brain or proof of human-like understanding. This HL lesson builds a single unit first, then a multi-layer network.

02 | INPUTS MUST BE REPRESENTED NUMERICALLY

An input is a numeric value supplied to the model. It could be a pixel intensity, a scaled distance or an encoded category. A vector is simply an ordered collection of those values.

The same input positions must retain the same meanings during training and later prediction. Swapping mass and distance columns, or changing the units silently, changes what the learned weights are applied to.

Data cleaning, appropriate scaling and a meaningful representation still matter. A neural network does not automatically repair incorrect labels, missing context or an unrepresentative dataset.

03 | A SINGLE UNIT HAS A SMALL COMPUTATION

For two inputs, begin with z = w₁x₁ + w₂x₂ + b. Each x is an input, each w is a weight and b is the bias. The value z is the pre-activation weighted sum.

Next compute a = f(z), where f is the activation function and a is the unit’s output. In a hidden layer, that output becomes an input to later units.

A classic perceptron uses a threshold or step activation to output a class decision. Modern network units often use other activations, such as ReLU or sigmoid. The weighted-sum-plus-activation structure connects these examples.

04 | SKETCH THE INPUTS, WEIGHTS, BIAS, ACTIVATION AND OUTPUT

Single artificial neuron with its labelled computational parts Inputs x1 and x2 travel along weighted connections w1 and w2 into a weighted sum. Bias b is added to form z. The activation function f takes z and produces output a equals f of z. A step activation makes this a classic threshold perceptron. x₁ x₂ weight w₁ weight w₂ Add bias b Weighted sum z = w₁x₁ + w₂x₂ + b Activation a = f(z) a Inputs Output
Follow the arrows: inputs are weighted, the bias is added, and the activation produces the output. The bias is a learned offset, not another measured feature.

In an exam sketch, label the connections as weights and identify the bias separately. Do not draw input values directly becoming an output without the combination and activation steps.

05 | WEIGHTS CONTROL EACH CONNECTION’S CONTRIBUTION

A weight multiplies an input before it contributes to the sum. A positive weight increases z as that input increases, holding the other terms fixed; a negative weight decreases it. A zero weight removes that input’s direct contribution to this unit.

For x₁ = 2 with w₁ = 0.6, the contribution is 1.2. For x₂ = 1 with w₂ = −0.4, the contribution is −0.4. Add contributions rather than treating a negative weight as a missing connection.

Weight magnitude must be interpreted in context. Different feature scales and paths through a network mean the largest individual weight is not automatically a global importance or causal score.

06 | BIAS SHIFTS THE UNIT’S RESPONSE

A bias is an added parameter, separate from the measured inputs. It shifts the pre-activation sum and therefore changes when or how the unit responds.

With contributions 1.2 and −0.4, the sum before bias is 0.8. Adding b = −0.3 gives z = 0.5. If the bias were −1.3 instead, z would become −0.5.

Bias here is a mathematical offset. It is not the same meaning as social bias, although a trained network can also produce unfair outcomes for other reasons. In a network, each ordinary hidden or output unit can have its own bias.

07 | A STEP ACTIVATION MAKES A THRESHOLD DECISION

For this lesson’s step function, f(z) = 1 if z ≥ 0, otherwise 0. At z = 0.5, the perceptron outputs 1. At z = −0.5, it outputs 0. At exactly zero, this stated convention outputs 1.

The output labels 0 and 1 could encode two categories. They are not automatically calibrated probabilities. A step output of 1 is a decision under the rule, not evidence of certainty.

The equality boundary is a design convention. State it clearly when tracing calculations. A different threshold convention can change an example exactly on the boundary.

08 | OTHER ACTIVATIONS PRODUCE DIFFERENT OUTPUTS

Activation Rule or behaviour Typical role and limit
Step Outputs 0 or 1 at a defined threshold Transparent perceptron illustration; not normally the hidden activation used with ordinary gradient-based training
ReLU max(0,z) A common hidden-layer activation; output is not a probability
Sigmoid 1/(1+e^(−z)), giving a value between 0 and 1 Often a binary-output score; probability interpretation needs suitable training and evaluation
Linear / identity f(z)=z Useful for an unrestricted numerical regression output

For z = 0.5, ReLU gives 0.5 and sigmoid gives approximately 0.622. The step function gives 1. The same weighted sum can therefore lead to different outputs under different activations.

An activation does more than simply decide whether all units are “on” or “off”. Many networks pass continuous activation values between layers. Activation choice should fit the model and task.

09 | TRACE A SINGLE UNIT INTERACTIVELY

Change the inputs, weights, bias and activation. The activity shows each multiplication, the weighted sum and the output. It performs a forward calculation with values you choose; it does not train or update weights automatically.

Start with the defaults and compare activations without changing z. Then change only the bias or the sign of a weight. A single traced example cannot establish how accurately a fitted network performs on a dataset.

10 | ONE PERCEPTRON HAS A LIMITED BOUNDARY

For two inputs, the condition w₁x₁ + w₂x₂ + b = 0 describes a straight boundary when the weights are not both zero. A threshold perceptron separates the two sides into its output categories.

Some patterns can be separated by one straight line. For binary inputs, AND can use weights 1 and 1 with bias −1.5: only (1,1) produces a nonnegative sum.

Other patterns cannot. The XOR pattern requires output 1 for (0,1) and (1,0), but 0 for (0,0) and (1,1). A single straight boundary cannot separate those two diagonal pairs in the original input plane.

11 | CONNECT UNITS INTO LAYERS

A multi-layer perceptron (MLP) is a feedforward neural network with an input layer, one or more hidden layers and an output layer. Information moves through successive computations toward the output.

The input layer presents the feature values. A hidden layer computes intermediate representations. The output layer produces the task’s final score or numerical prediction.

“Hidden” means intermediate rather than directly supplied as input or observed as the target. It does not mean the values are impossible to inspect. The XOR activity below shows every hidden value.

12 | SKETCH A MULTI-LAYER PERCEPTRON

A two-input, two-hidden-unit, one-output MLP Two input nodes connect to both nodes in one hidden layer. Both hidden nodes connect to a single output node. Each connection has its own weight, and hidden and output units have their own biases and activations. Information flows from left to right. INPUT LAYER HIDDEN LAYER OUTPUT LAYER x₁ x₂ h₁ h₂ ŷ Feature values Intermediate values Final prediction Each line represents a weight; each computed unit also has a bias and activation.
A 2–2–1 feedforward MLP: two inputs, one hidden layer of two units and one output unit. More hidden layers or units are possible, but not automatically better.

In this fully connected example, every unit in one layer connects to each unit in the next. Other network architectures use different connectivity. The drawing shows the computation structure, not the number of records in the dataset.

13 | A HIDDEN OUTPUT BECOMES A NEW INPUT

The hidden unit h₁ receives x₁ and x₂, uses its own weights and bias, then produces an activation value. Hidden unit h₂ does the same with a different parameter set.

The output unit then treats h₁ and h₂ as its inputs. It does not normally recompute the original input-layer formula unchanged. Its own weights, bias and activation combine the hidden representation.

This is forward propagation: pass input values through the fitted computations. Forward propagation can happen during training to calculate a loss, and during inference to make a prediction. By itself it does not adjust the parameters.

14 | NONLINEAR ACTIVATIONS MAKE THE COMPOSITION MORE EXPRESSIVE

Stacking weighted sums and biases without nonlinear activations still gives an overall linear/affine mapping. For example, h = 2x + 1 followed by y = 3h + 4 simplifies to y = 6x + 7. Adding those two linear stages has not created a curved relationship.

Nonlinear hidden activations allow layers to combine intermediate patterns into relationships that a single linear boundary cannot express. ReLU is piecewise linear but nonlinear overall; step and sigmoid are also nonlinear.

More layers do not automatically discover useful structure. The architecture must be trained effectively with suitable data, and excessive capacity can overfit. The key is the composition of learned transformations and nonlinear activations, not simply a larger diagram.

15 | A SMALL NETWORK CAN REPRESENT XOR

x₁ x₂ Wanted XOR output
0 0 0
0 1 1
1 0 1
1 1 0

Use two hidden step units: h₁ = step(x₁ + x₂ − 0.5) detects at least one input being 1. h₂ = step(x₁ + x₂ − 1.5) detects both inputs being 1.

The output is ŷ = step(h₁ − 2h₂ − 0.5). It combines “at least one” with a negative contribution for “both”, giving exactly one input on.

These parameter values are deliberately supplied to illustrate representation. The next activity calculates them; it does not claim to learn XOR from data or demonstrate gradient training through step activations.

16 | TRACE THE HIDDEN LAYER FOR ONE CASE

Stage for x₁=1, x₂=1 Pre-activation sum Step output
Hidden h₁ 1 + 1 − 0.5 = 1.5 1
Hidden h₂ 1 + 1 − 1.5 = 0.5 1
Output 1 − 2×1 − 0.5 = −1.5 0

Both hidden units output 1, but the final output is 0 because the second hidden value has output weight −2. A network does not simply vote among neurons or count how many are active.

For (1,0), h₁ is 1 and h₂ is 0, so the output sum is 1 − 0 − 0.5 = 0.5 and the result is 1. The hidden representation has made the desired distinction possible.

17 | INSPECT ALL FOUR XOR CASES

Select the two binary inputs and follow every sum. The full table shows how the fixed two-unit hidden layer produces the target pattern.

The purpose is to see a multi-layer computation model a pattern a single threshold unit cannot separate in the original inputs. It is not evidence that every larger network is accurate or that the supplied weights are the only solution.

18 | TRAINING CHANGES WEIGHTS AND BIASES

For supervised training, examples pair inputs with known outcomes. First perform a forward pass, then use a loss function to measure how the prediction differs from the target.

A training method adjusts weights and biases to reduce the training objective. In many modern ANNs, backpropagation computes how parameters influence the loss, and an optimiser uses that information to update them.

The loss function is not the activation function. The activation transforms a unit’s sum; the loss evaluates predictions against targets. Training uses many examples and repeated updates, not a guarantee that each update improves every example. No calculus or backpropagation derivation is needed for this outline objective.

19 | MATCH THE OUTPUT TO THE TASK

Task Possible network output Interpretation
Binary classification One sigmoid output plus a defined decision threshold A score used to choose between two categories; calibration needs checking
Mutually exclusive multiclass classification One score per class, often with softmax Relative class probabilities under the model; choose using an appropriate rule
Regression One or more numerical outputs, often linear Estimated measured quantities, with task-appropriate units and constraints
Pattern recognition Outputs or representations linked to recognised patterns The task may involve classification or another prediction objective

A network with one output is not limited to one class: one binary score can distinguish two categories. Conversely, several outputs can represent multiple numerical targets rather than classes. Define the task and encoding.

Softmax makes a vector of nonnegative values summing to 1 for a mutually exclusive class model. That mathematical format does not guarantee that the network’s confidence matches real-world correctness.

20 | COUNT THE PARAMETERS, NOT THE INPUT RECORDS

In the illustrated 2–2–1 fully connected network, input-to-hidden connections give 2 × 2 = 4 weights. Hidden-to-output connections give 2 × 1 = 2 more. There are also two hidden biases and one output bias.

Total trainable parameters = 4 + 2 + 2 + 1 = 9, under this stated architecture. The input nodes simply present values and do not each have an extra learned bias here.

Layer widths and counts are architecture choices or hyperparameters; weights and biases are normally fitted parameters. More units can increase capacity, memory and computation, but the number of training examples is a separate quantity.

21 | COMPLEX PATTERNS COME FROM COMBINED REPRESENTATIONS

Hidden layers can learn combinations useful for a task rather than being explicitly given every intermediate concept. Later layers combine earlier activations, allowing a compact computation to model complicated relationships.

For an image task, intermediate features may respond to local or combined patterns, depending on the architecture and data. Do not assume every hidden unit has a simple human-readable meaning or that an ordinary MLP preserves spatial layout automatically.

The next lesson introduces CNNs, whose structure is designed to exploit spatial relationships. The present outline concerns the common input, learned transformations, nonlinear activations and output structure.

22 | GENERALISATION STILL NEEDS EVIDENCE

A large ANN can fit training patterns, including incorrect labels or accidental correlations. Strong training performance alone does not show useful performance on genuinely new inputs.

Use suitable validation to choose architecture and training settings, preserve an independent final test, and fit preprocessing within the training process. Examine task metrics, data coverage and important errors.

Networks also have limits in interpretability, compute use and sensitivity to changed conditions. A high class score is not certainty, and biological inspiration does not remove the need to check fairness, reliability or the meaning of the objective.

23 | CHECK THE PARTS AND THE PROCESS

Choose the concept that best matches each statement. Distinguish values flowing forward from parameters being fitted.

24 | OUTLINE THE NETWORK CLEARLY

For a single unit, outline the sequence: numerical inputs → weighted contributions → added bias → activation function → output. Label each element in the sketch.

For an MLP, identify the input layer, one or more hidden layers and the output layer. Explain that hidden activations become inputs to the next layer, and that training adjusts connection weights and biases to model useful patterns.

Connect structure to function: nonlinear hidden transformations allow combinations more expressive than a single linear boundary, supporting classification, regression and pattern recognition. Include the limit that architecture alone does not prove accurate generalisation.

Further reading: weighted sums, biases and activations in a dense layer and feedforward composition and nonlinear hidden layers.

MATCH THE NETWORK IDEA TO ITS ROLE

Terminology

Terminology

Artificial neural network

A model of connected computational units using learned parameters and activation functions.

Artificial neuron

A unit that combines weighted inputs, adds a bias and applies an activation.

Perceptron

A classic threshold unit producing a decision from a weighted sum and bias.

Input feature

A numerical value supplied in a defined position of the model input.

Weight

A parameter multiplying a value on a connection.

Bias

An added parameter shifting a unit’s pre-activation sum.

Weighted sum z

The sum of weighted contributions plus bias before activation.

Activation function

A transformation applied to a unit’s pre-activation sum.

Activation value

The output produced by a unit after applying its activation function.

Step function

A threshold function giving discrete outputs under a stated boundary convention.

ReLU

The activation max(0,z), nonlinear overall despite its linear pieces.

Sigmoid

An activation mapping a real sum smoothly to a value between 0 and 1.

Linear output

An output using identity activation, passing the computed sum through.

Input layer

The part of a network presenting the feature values.

Hidden layer

An intermediate layer computing representations between inputs and outputs.

Output layer

The final computed layer providing task predictions or scores.

Multi-layer perceptron

A feedforward network with an input layer, hidden layer(s) and output layer.

Fully connected layer

A layer where each unit receives values from every unit of the preceding layer.

Forward propagation

Computing successive activations from inputs toward outputs.

Inference

Applying a fitted model to obtain outputs for supplied inputs.

Parameter

A fitted model quantity such as a connection weight or bias.

Architecture

The chosen network structure, including connectivity, layer count and widths.

Loss function

A measure of prediction mismatch used in a training objective.

Backpropagation

Computing how parameters influence a training loss through a differentiable network.

Optimiser

The method using training information to update parameters.

Nonlinear transformation

A transformation that is not just an affine weighted-sum mapping.

XOR

A binary pattern that is 1 when exactly one input is 1.

Softmax

A transformation producing nonnegative class values that sum to 1.

Generalisation

Performance on genuinely new examples from the intended population.

Overfitting

Capturing training-specific detail that does not transfer reliably.

Questions

Questions

CHECK YOUR UNDERSTANDING

Select all correct choices. Each exact set earns one point.

1. Which describe an ANN?
2. Which identify the single-unit computation correctly?
3. For x₁=2, x₂=1, w₁=0.6, w₂=−0.4, b=−0.3, which are correct?
4. With step(z)=1 for z≥0, which are correct?
5. Which are true of other activations?
6. Which describe an MLP?
7. Which explain expressive structure?
8. For the supplied XOR network and (1,1), which are correct?
9. Which distinguish training and inference?
10. Which are correct architecture or evaluation statements?

EXPLAIN AND COMPARE

Write your answers before revealing each sample.

1. Outline the structure and function of a single artificial neuron.

2. Calculate the pre-activation sum for x₁=2, x₂=1, w₁=0.6, w₂=−0.4 and b=−0.3.

3. Outline the roles of weights and bias.

4. Calculate the output at z=0.5 for step, ReLU and sigmoid.

5. Outline an MLP using the required layers.

6. Outline why nonlinear hidden activations matter.

7. Trace the supplied XOR network for inputs (1,1).

8. Trace the supplied XOR network for inputs (1,0), then state its purpose.

9. Outline the difference between forward propagation and training.

10. Outline suitable outputs for classification and regression.

11. Calculate the number of weights and biases in the fully connected 2–2–1 example.

12. Outline two limitations to check when using an ANN for complex patterns.

OUTLINE IT | TWO LABELLED SKETCHES

Draw and label a single perceptron with inputs, weights, bias, activation and output. Then sketch an MLP with an input layer, hidden layer(s) and output layer. Outline how values flow through it, why nonlinear hidden activations matter and how training differs from merely calculating a prediction.

Flashcards

Flashcards

Click to flip. Select the ideas you need to revisit.

0 cards selected for revision.

Selections are kept while this page is open.

Workbook

Workbook

COMING SOON

The workbook for A4.3.8 Artificial Neural Networks (HL) is coming soon. Complete the matching task and written questions in your exercise book.