IB COMPUTER SCIENCE | A4.3.1 Linear Regression (HL)

Lesson objective

Explain how linear regression predicts continuous outcomes, interpret the predictor, response, slope and intercept, and assess fit using residuals and R².

Learn

A4.3.1 | LINEAR REGRESSION (HL ONLY)

01 | PREDICT A NUMBER, NOT A CATEGORY

A delivery service wants to estimate how many minutes a journey will take before a van leaves. Predicting “late” or “on time” would choose a category. Predicting 14.6 minutes estimates a numerical quantity.

Linear regression is a supervised learning approach that fits a linear relationship between input variables and a numerical response. It can then use that relationship to predict an outcome for a new input.

This lesson begins with one input: journey distance. You will build the meaning of the equation, see how a line is fitted and judge its fit without assuming that every prediction is correct. All delivery values are fictional teaching data.

02 | WHAT DOES CONTINUOUS MEAN?

A continuous outcome represents a measured quantity that can, in principle, take values between other values. Duration, temperature and mass are examples. A journey could take 11 minutes, 11.5 minutes or 11.53 minutes, even if the recording system rounds it.

The display format does not determine the task. A duration recorded as an integer number of minutes is still a rounded measurement. A numeric postcode or category code is not automatically a meaningful continuous target.

Regression estimates quantities; classification predicts discrete categories. Linear regression produces numeric predictions and does not automatically constrain them to physically valid values such as positive durations. Check predictions in context.

03 | NAME THE INPUT AND THE RESPONSE

The independent variable, also called the predictor, is the input used to make a prediction. Here it is distance x, measured in kilometres. The dependent variable, or response, is the outcome being estimated: duration y, measured in minutes.

These names describe roles in the model. Calling distance independent does not prove that we experimentally controlled it or that distance alone causes every change in duration.

During supervised training, historical examples contain both known inputs and observed outcomes. At prediction time, provide the new distance; the model calculates an estimated duration without seeing that journey’s eventual outcome.

04 | BEGIN WITH PAIRED OBSERVATIONS

Journey Distance x (km) Observed duration y (min)
A 1 7
B 2 8
C 3 11
D 4 12
E 5 17

Each row pairs the distance of one journey with its actual recorded duration. Longer journeys generally take longer here, although the increases are not identical. Stops, traffic and other factors can affect duration too.

Plot distance on the horizontal x-axis and duration on the vertical y-axis. Each row becomes one observed point. A fitted line summarises a pattern in those points; it does not join every observation in sequence.

This tiny dataset is for transparent calculations, not a sufficient operational evaluation of a delivery service.

05 | READ THE EQUATION ONE PART AT A TIME

For simple linear regression with one input, write ŷ = mx + b. The symbol ŷ, read “y-hat”, is the predicted response. The actual observed response is y.

The slope m controls how steeply the prediction changes as x increases. The intercept b sets the prediction at x = 0. These learned quantities are the model’s parameters.

You may also see ŷ = β₀ + β₁x. Here β₀ is the intercept and β₁ is the slope. The notation differs but the roles are the same. Keep the input, actual outcome and predicted outcome distinct.

06 | THE SLOPE IS A CHANGE PER INPUT UNIT

Consider the candidate rule ŷ = 2x + 5. Its slope is 2 minutes per kilometre. When distance increases by 1 km, the predicted duration increases by 2 minutes.

For a change of 3 km, the predicted increase is 2 × 3 = 6 minutes. The slope describes a change, not the entire prediction. At x = 3, you must still include the intercept.

The sign also matters. A positive slope means the fitted prediction increases with x; a negative slope means it decreases; a zero slope means the prediction does not vary with that input. A negative slope is possible in other tasks, such as a fitted relationship between altitude and temperature over an appropriate range.

07 | THE INTERCEPT LOCATES THE LINE

In ŷ = 2x + 5, the intercept is 5 minutes. Substituting x = 0 gives ŷ = 5. Geometrically, it is where the line crosses the vertical axis.

One possible interpretation is a fixed component of journey time, but this dataset does not prove that interpretation. Our observed distances range from 1 to 5 km; a zero-distance delivery is not represented.

An intercept can be useful for positioning the line even when x = 0 is outside the meaningful or observed range. Do not claim it is a real measured waiting time merely because the fitted value is positive.

08 | MAKE A PREDICTION BY SUBSTITUTION

Step Working for a 4.5 km journey
Write the rule ŷ = 2x + 5
Insert the known input ŷ = 2 × 4.5 + 5
Multiply ŷ = 9 + 5
Add and interpret ŷ = 14 minutes, a prediction rather than a known outcome

The model takes a known input and produces an estimated numerical response. It does not look up an identical journey or require the new outcome to be supplied.

Multiplication happens before addition. Omitting the intercept would give 9 minutes, which belongs to a different model. Include units so that the result has a clear meaning.

09 | PREDICTIONS DO NOT NEED TO LAND ON THE POINTS

At distance 2 km, the candidate rule predicts 2 × 2 + 5 = 9 minutes. Journey B actually took 8 minutes. That difference is normal evidence to examine, not a reason to replace the observed duration with the prediction.

A single straight line usually cannot pass through every noisy observation. The aim is to capture a useful overall relationship while allowing unexplained variation.

A line that passes through every training example is not automatically reliable on new examples. Judge performance with independent data and context, rather than demanding perfect training fit.

10 | RESIDUALS SHOW WHICH WAY A PREDICTION MISSES

A residual is observed minus predicted: e = y − ŷ. At journey B, e = 8 − 9 = −1 minute. The prediction is 1 minute too high, so the residual is negative.

At journey E, the rule predicts 15 minutes but the actual duration is 17. Its residual is 17 − 15 = +2 minutes: the prediction is too low.

On a scatter plot, the residual is the vertical separation between an observed point and the line at the same x-coordinate. It is not the shortest perpendicular distance to the line.

11 | SQUARE THE RESIDUALS TO COMPARE LINES

x Observed y Predicted 2x + 5 Residual y − ŷ Squared residual
1 7 7 0 0
2 8 9 -1 1
3 11 11 0 0
4 12 13 -1 1
5 17 15 2 4

The signed residuals add to zero here: 0 − 1 + 0 − 1 + 2 = 0. That does not mean there is no error, because positive and negative misses cancel.

Squaring makes each contribution nonnegative and penalises larger misses more strongly. The sum of squared errors, SSE, is 0 + 1 + 0 + 1 + 4 = 6 min². It is also called the residual sum of squares.

SSE is a combined measure for these particular observations. It is not a percentage and is not expressed in ordinary minutes. Comparing SSE between different-sized datasets needs care.

12 | WHAT DOES TRAINING THE LINE MEAN?

Ordinary least squares selects the slope and intercept that minimise the sum of squared residuals on the training data. The software adjusts the line’s parameters to reduce that objective.

For the five supplied observations, the least-squares line with an intercept is ŷ = 2.4x + 3.8. Its SSE is 4.4, lower than 6 for the candidate 2x + 5 line.

Training learns the parameters from paired examples. Inference then applies the fitted equation to new inputs. Some implementations solve the least-squares problem directly; others use iterative optimisation. You do not need to assume the model repeatedly guesses random lines.

13 | FOLLOW THE FITTED LINE THROUGH THE DATA

x Observed y Predicted 2.4x + 3.8 Residual Squared residual
1 7 6.2 0.8 0.64
2 8 8.6 -0.6 0.36
3 11 11 0 0
4 12 13.4 -1.4 1.96
5 17 15.8 1.2 1.44

The squared residuals sum to 0.64 + 0.36 + 0 + 1.96 + 1.44 = 4.4. The best line under this objective still misses individual observations.

At x = 4.5, the fitted prediction is 2.4 × 4.5 + 3.8 = 14.6 minutes. Its slope indicates an extra 2.4 predicted minutes for each additional kilometre within this fitted relationship.

The coefficients are supported by this small training set. They are not a guarantee that all future journeys take exactly that time.

14 | ADJUST THE LINE AND WATCH THE RESIDUALS

Change the slope and intercept, or load the least-squares fit and the mean baseline. The green line shows predictions, pink dots show observed durations and dashed pink segments show vertical residuals. The outlined marker is your new prediction.

The control ranges are chosen to keep this graph readable; linear regression itself can use negative slopes and intercepts.

Try the initial 2x + 5 line, the least-squares fit, and then the mean baseline. The displayed SSE and R² are calculated on these five teaching observations, not an independent test set. Try predicting at 7 km and notice the extrapolation warning.

15 | COMPARE WITH A SIMPLE BASELINE

Before judging how useful a fitted model is, compare it with an uninformative reference. The mean observed duration in this dataset is (7 + 8 + 11 + 12 + 17) / 5 = 11 minutes.

A constant line ŷ = 11 ignores distance and always predicts the mean. The squared deviations from this mean are 16, 9, 0, 1 and 36, summing to 62. This is the total sum of squares, SST.

R² uses this mean reference for the same evaluated outcomes. In a real held-out comparison, a deployable mean-only baseline would be learned from training data; the evaluation-set mean used in the R² denominator is a mathematical reference, not permission to learn predictions from test outcomes.

16 | R²: HOW MUCH BETTER THAN THE MEAN REFERENCE?

The coefficient of determination, R², compares residual squared error with variation around the evaluated outcomes’ mean: R² = 1 − SSE / SST, provided SST is nonzero.

For the fitted line, R² = 1 − 4.4 / 62 ≈ 0.929. On these five training observations, the line accounts for approximately 92.9% of their squared variation relative to the mean reference.

For the candidate 2x + 5 line, R² = 1 − 6 / 62 ≈ 0.903. The mean line has R² = 1 − 62 / 62 = 0. A perfect prediction of every evaluated value would have SSE = 0 and R² = 1.

17 | WHAT R² DOES NOT TELL YOU

R² = 0.929 does not mean 92.9% of predictions are exactly correct, or that every prediction is within 7.1% of the answer. It is not classification accuracy and does not establish causation.

An evaluated model can have a negative R² if its SSE exceeds SST: it is worse under this comparison than the evaluated mean reference. A negative slope does not automatically produce negative R²; slope direction and goodness of fit are different ideas.

If every evaluated outcome is identical, SST = 0 and this formula has a zero denominator. The ordinary formula is not defined; libraries may apply a special convention. R² also does not give an individual prediction’s uncertainty or error in minutes.

18 | LOOK AT ERRORS IN USEFUL UNITS TOO

For the fitted line, absolute residuals are 0.8, 0.6, 0, 1.4 and 1.2 minutes. Their average is 0.8 minutes, the mean absolute error, MAE, on these observations.

MAE answers a different question from R². It describes average absolute miss in the target’s units, while R² compares squared error with the evaluated mean reference. Both need the evaluation context to be meaningful.

A high R² can coexist with errors too large for a particular application. Inspect individual errors, relevant groups and rare cases rather than relying on a single headline number.

19 | INTERPOLATION AND EXTRAPOLATION

The observed distances span 1 to 5 km. Predicting at 4.5 km is interpolation, because it is inside that observed range. Predicting at 7 km is extrapolation, because it is outside it.

The fitted equation can calculate 2.4 × 7 + 3.8 = 20.6 minutes, but calculating a number is not evidence that the relationship continues there. Longer journeys might involve a motorway, different stops or different operating conditions.

Interpolation is not guaranteed correct either. A new journey can differ in important unrecorded ways. Treat both as estimates, with particular caution when the model is applied beyond the training range or population.

20 | CHECK WHETHER A STRAIGHT LINE IS REASONABLE

A straight line represents a constant predicted change per unit increase in the input. If duration grows rapidly after a congestion threshold, one line may miss a systematic pattern.

Inspect a scatter plot and residuals. Curved residual patterns can suggest that a straight-line relationship is inadequate. A growing spread of residuals can indicate that errors are less stable for some input ranges. Neither pattern is resolved simply by quoting a high training R².

Least squares is sensitive to large squared residuals, and unusual high-leverage inputs can influence a fitted line. Investigate context before removing observations: a genuine long journey is not an error merely because it changes the fit.

21 | MORE THAN ONE PREDICTOR

Simple linear regression uses one predictor. Multiple linear regression can use several, for example ŷ = b₀ + b₁ × distance + b₂ × stops. The output remains a numerical response.

In such a model, the distance coefficient describes the change in predicted duration for an extra kilometre while holding the other included predictor values fixed. It is not automatically the same as the slope fitted using distance alone.

Additional inputs can capture useful differences but also introduce redundancy, leakage or overfitting. Apply the earlier lessons on cleaning, selection and validation. A more complicated equation is not automatically a better deployed predictor.

22 | TRAIN, EVALUATE, THEN PREDICT

Collect representative paired examples and define when the prediction is needed. Split data appropriately, fitting any learned cleaning or scaling on training data only. Fit the regression parameters on training examples.

Use validation procedures for model and pipeline choices, then evaluate the chosen pipeline on independent test examples. Report which set each metric describes. Training R² alone does not demonstrate future performance.

At deployment, supply the same input meanings, units and preprocessing used by the fitted pipeline. Monitor errors and changed conditions. A 4.5-mile input supplied as 4.5 kilometres silently changes the prediction’s meaning.

23 | CHECK YOUR INTERPRETATION

Choose the concept that best explains each statement. Focus on the meanings of the equation and evaluation results rather than memorising isolated definitions.

24 | EXPLAIN THE PROCESS WITH CONNECTED REASONS

A complete explanation can follow the chain: paired input and response examples reveal a relationship; least-squares fitting chooses a line that minimises training squared residuals; substituting a new input produces a continuous numerical prediction; independent evaluation checks how useful that fitted relationship is.

Use connectives to make the reasoning clear: “The slope is 2.4 minutes per kilometre, therefore increasing distance by 1 km raises the predicted duration by 2.4 minutes. Actual journeys vary because distance does not capture every influence.”

Link fit and limits: “R² compares squared residuals with variation around the mean, so a value near 1 indicates a close fit on the evaluated data. However, training fit alone does not establish accurate predictions for new journeys.”

Further reading: ordinary least-squares fitting and R² interpretation and edge cases.

MATCH THE REGRESSION IDEA TO ITS ROLE

Terminology

Terminology

Regression

Prediction of a numerical quantity from input information.

Continuous outcome

A measured quantity that can in principle take intermediate values.

Classification

Prediction of discrete categories rather than a measured numerical response.

Supervised learning

Fitting a relationship using examples paired with known target outcomes.

Independent variable

The input or predictor variable in the model.

Dependent variable

The response or target outcome being modelled.

Simple linear regression

A linear regression with one predictor and a numerical response.

Predicted response ŷ

The numerical outcome estimated by the fitted model.

Observed response y

The actual outcome recorded for an example.

Slope

Change in predicted response per unit increase in the predictor.

Intercept

The model prediction at a predictor value of zero.

Parameter

A model quantity learned during fitting, such as slope or intercept.

Residual

Observed minus predicted response, y − ŷ.

Squared residual

The residual multiplied by itself, giving a nonnegative contribution to squared error.

SSE

Sum of squared residuals on the evaluated examples.

Ordinary least squares

Fitting by minimising the sum of squared residuals.

Mean baseline

A constant reference prediction based on a mean response.

SST

Sum of squared deviations of evaluated outcomes from their mean.

R²

Coefficient of determination: 1 − SSE/SST when SST is nonzero.

MAE

Mean absolute error, expressed in the response variable’s units.

Interpolation

Prediction within the observed predictor range.

Extrapolation

Prediction outside the observed predictor range.

Inference

Applying a fitted model to obtain predictions for supplied inputs.

Generalisation

Performance on genuinely new examples from the intended population.

Residual pattern

A systematic structure in prediction misses that can reveal model inadequacy.

Outlier

An unusual observation that may be a genuine event or an error.

Multiple linear regression

Linear regression using more than one predictor.

Training fit

How well the fitted model describes the examples used to learn its parameters.

Causation

A cause-and-effect relationship not established merely by a fitted association.

Data leakage

Use of information unavailable or illegitimate for fitting or a genuine new prediction.

Questions

Questions

CHECK YOUR UNDERSTANDING

Select all correct choices. Each exact set earns one point.

1. Which are numerical regression targets?
2. For duration predicted from distance, which roles are correct?
3. In ŷ = 2x + 5, which are correct?
4. Which predictions follow from ŷ = 2.4x + 3.8?
5. Which are correct residual statements?
6. Which describe least squares?
7. Which interpret R² correctly?
8. Which are true of extrapolation?
9. Which support honest evaluation?
10. Which are correct limitations?

EXPLAIN AND COMPARE

Write your answers before revealing each sample.

1. Explain how linear regression predicts a continuous outcome from paired examples.

2. Explain the roles of the independent and dependent variables in the delivery example.

3. Explain the slope and intercept in ŷ = 2.4x + 3.8.

4. Calculate the predicted duration at 4.5 km, then explain what the result means.

5. Calculate and interpret the residual when observed duration is 12 and the fitted prediction is 13.4.

6. Explain why least squares uses squared residuals rather than their signed sum.

7. Calculate R² when SSE = 4.4 and SST = 62, then explain its meaning.

8. Explain the meanings of R² = 1, R² = 0 and a negative R².

9. Explain why predicting at 7 km requires extra caution for this dataset.

10. Explain why R² and MAE can both be useful.

11. Explain how to evaluate the line for deployment rather than only fit the training examples.

12. Explain two limitations of predicting duration from distance alone.

EXPLAIN IT | FROM DATA TO DEPLOYMENT

A service fits duration = 2.4 × distance + 3.8 using five journeys from 1 to 5 km. Explain how paired examples produce the model, what its parameters mean and how it predicts at 4.5 km. Explain why training R² near 0.929 does not validate a prediction at 7 km, and describe an independent evaluation procedure.

Flashcards

Flashcards

Click to flip. Select the ideas you need to revisit.

0 cards selected for revision.

Selections are kept while this page is open.

Workbook

Workbook

COMING SOON

The workbook for A4.3.1 Linear Regression (HL) is coming soon. Complete the matching task and written questions in your exercise book.