IB COMPUTER SCIENCE | A4.3.10 Model Selection and Comparison (HL)

Lesson objective

Explain why model suitability depends on the task and data, and how fair validation, error costs, resource limits and desired outcomes guide comparison and selection.

Learn

A4.3.10 | MODEL SELECTION AND COMPARISON (HL ONLY)

01 | THERE IS NO SINGLE WINNER FOR EVERY TASK

A model with the highest score in an image experiment may be unsuitable for predicting delivery time from a small table. Even two classifiers for the same task can differ in missed cases, speed, memory and how easily their decisions can be explained.

Model selection means choosing a suitable model and configuration for a defined problem using evidence. Model comparison is the process of evaluating alternatives under meaningful and reasonably comparable conditions.

This HL lesson brings together the earlier approaches. The central question is not “Which algorithm is best?” but “Which validated pipeline meets this task’s requirements, on these data, within these constraints?”

02 | DEFINE THE OUTCOME BEFORE CHOOSING THE ALGORITHM

First describe the required output. A measured duration is a regression target; a defective/not-defective label is a classification target. Discovering purchasing groups without known segment labels is clustering.

A rule about products appearing together is an association-mining output. Learning a sequence of actions from rewards is a reinforcement-learning formulation. Searching for a shorter complete route can be an optimisation task.

These approaches do not all answer the same question, so their headline scores cannot be ranked as if they did. An algorithm that solves the wrong problem cannot become suitable merely by being more complex.

03 | START WITH THE DATA YOU REALLY HAVE

Ask what one example represents, how many useful examples exist, which inputs are available at prediction time and whether reliable targets are known. Check missing values, duplicates, class balance and coverage of the intended population.

For images, spatial organisation can make convolutional structure a useful starting point. For a small structured table, a simpler regression or tree may be easier to fit and evaluate. These are reasons to investigate candidates, not automatic winners.

Exclude illegitimate inputs before chasing a score. A value recorded only after the outcome can create leakage even if every candidate algorithm handles it perfectly.

04 | DATA CHARACTERISTICS CHANGE THE COMPARISON

Data characteristic Why it matters Possible implication to investigate
Approximately linear relationship A simple mapping may already capture the signal A linear baseline can be competitive and easy to inspect
Nonlinear interactions An effect may depend on combinations of inputs A tree or nonlinear ANN may capture useful structure
Many irrelevant dimensions Distance and fitting can be distracted by noise Feature selection or reduction may help the pipeline
Different numeric scales Distance- or gradient-based computation may be affected Use candidate-appropriate fitted preprocessing
Few examples or noisy labels Flexible models can fit accidents Simpler or constrained models may generalise better
Uneven classes Overall accuracy can conceal missed minority cases Inspect class-specific metrics and error counts
Spatial image structure Nearby values contain useful relationships A CNN can exploit local shared patterns

A property changes what is plausible, not what is guaranteed. For example, ordinary trees are often less sensitive to numerical scaling than K-NN, but they still require consistent feature meanings and valid data.

05 | WHY TWO ALGORITHMS CAN REACH DIFFERENT ANSWERS

Each approach makes different representational choices. Linear regression fits an affine numerical relationship. K-NN relies on nearby examples under its metric. A tree partitions feature space using tests. A nonlinear ANN composes learned transformations.

Those choices affect which patterns are easy to capture, how noise influences fitting and what happens in sparsely represented regions. Hyperparameters and preprocessing further change the behaviour.

The resulting differences are not necessarily implementation errors. Compare predictions and failure cases to understand whether each method’s assumptions and flexibility match the task.

06 | A WORKED EXAMPLE OF A CHANGING RANKING

Candidate Illustrative validation MAE: mostly linear data Illustrative validation MAE: nonlinear-interaction data
Linear regression 2.1 minutes 7.2 minutes
Decision-tree regressor 3.0 minutes 3.2 minutes
Nonlinear ANN regressor 2.4 minutes 2.8 minutes

These supplied fictional results illustrate two different regression datasets. Within each dataset, candidates use comparable development splits and the same target and metric. Lower MAE is better.

The linear model leads in the first setting, while the ANN leads in the second. This demonstrates a possible change in ranking when the relationship changes; it is not a measured general rule about these algorithms.

Do not compare the numbers across the two datasets as proof that one dataset is universally easier. Different outcome distributions and sampling conditions affect the meaning of a score. The useful comparison is between candidates within the stated evaluation context.

07 | USE A SIMPLE BASELINE

A baseline is a simple reference. A classification baseline might always predict the training set’s most common class. A regression baseline might always predict the training mean or median under an appropriate metric.

If 80% of development examples are good parcels, always predicting good can achieve 80% accuracy while detecting no defects. Beating that accuracy is useful evidence, but defect-specific errors still matter.

Fit a deployable baseline from training data, then evaluate it on the same held-out examples as the other candidates. A complex model should justify its additional burden through meaningful benefit, not merely a complicated architecture.

08 | COMPARE WHOLE PIPELINES

A pipeline includes preprocessing, feature selection or reduction and the fitted model. K-NN with inappropriate unscaled features is not a fair representation of what a carefully prepared K-NN candidate can do.

Fair comparison does not require identical transformations for every algorithm. It requires compatible input information, legitimate preprocessing fitted inside training, and a transparent procedure suitable for each candidate.

Record units, feature choices, missing-value handling and class encoding. If one model has extra information unavailable to another, make that difference explicit rather than attributing the whole performance gap to algorithm choice.

09 | KEEP THE EVALUATION EXAMPLES COMPARABLE

Use common held-out examples or suitably matched cross-validation folds when comparing candidates for the same task. Otherwise one candidate can appear better simply because it was evaluated on easier cases.

Choose splits that reflect the intended deployment. Keep related records together when needed, preserve time order for future prediction tasks and prevent duplicates from crossing fitting and evaluation boundaries.

Fit learned scaling, imputation and feature selection only on the relevant training partition. Splitting after those transformations may let held-out data influence the fitted pipeline.

10 | MODEL SELECTION ALSO INCLUDES TUNING

A tree’s maximum depth or K-NN’s k can make a large difference. Comparing one carefully tuned algorithm against an arbitrary poor setting of another can be misleading.

Declare appropriate tuning ranges, objectives and search budgets. Equal wall-clock time and equal configuration counts are not the same form of fairness; describe the budget and justify it for the purpose.

Selecting the model family, features, thresholds and hyperparameters are all development choices. Keep the final test data outside those decisions. Trying more candidates also creates more opportunities to overfit validation evidence.

11 | VALIDATION IS FOR CHOICES; TESTING IS FOR THE FINAL ESTIMATE

Use training examples to fit candidates and validation procedures to guide choices. After selection, a common procedure is to refit the chosen configuration on the allowed training-plus-validation development data.

Then evaluate the chosen pipeline on an independent final test set. If test results lead you to redesign and retune, that test is no longer an untouched final check. Report the limitation and obtain an appropriate independent evaluation.

When extensive tuning is combined with cross-validation, nested evaluation can separate the inner selection process from outer evaluation. The important principle is independence, not memorising one compulsory split percentage or advanced procedure.

12 | CHOOSE METRICS FOR THE OUTCOME AND ERROR COSTS

Task Useful comparison evidence Limit to remember
Classification Confusion counts, accuracy, precision, recall and F1 under stated class conventions Accuracy can hide rare-class failure; F1 does not encode every error cost
Regression MAE or other numerical errors; R² with its reference context A unitless fit score is not the same as error in useful units
Clustering Structure, stability, suitable internal measures and downstream usefulness No known target groups means ordinary accuracy is not directly available
Sequential decisions or optimisation Return/completion or objective value with validity and resource checks A large proxy score can miss the real objective

Do not compare 0.9 classification accuracy with 0.8 regression R² and conclude the classifier is better. They answer different questions on different target tasks.

13 | DEFINE THE PARCEL COMPARISON

Consider a fictional binary classifier screening parcels. Positive means defective. Each candidate is evaluated on the same 200 validation parcels: 40 defective and 160 not defective.

The task’s measured inputs are tabular descriptors, not raw images. The candidates are a constant baseline, a constrained decision tree, a scaled K-NN pipeline and a small nonlinear ANN. The supplied values illustrate selection reasoning; no candidates were trained or timed to produce this teaching table.

Candidate TP FP FN TN Illustrative latency Illustrative model/pipeline memory
Baseline: always good 0 0 40 160 1 ms 1 MiB
Decision tree 30 10 10 150 2 ms 4 MiB
K-NN 34 14 6 146 18 ms 32 MiB
ANN 36 12 4 148 5 ms 12 MiB

The time and memory figures are fictional measurements under one stated teaching setup, not product benchmarks or general claims about these model families. A real comparison must measure the relevant pipeline on the actual target hardware and workload.

14 | READ PERFORMANCE BEFORE CHOOSING

Candidate Accuracy Precision for defects Recall for defects F1 for defects
Baseline 80% Undefined: no flags 0% 0
Decision tree 90% 75% 75% 0.750
K-NN 90% ≈70.8% 85% ≈0.773
ANN 92% 75% 90% ≈0.818

The tree and K-NN have equal accuracy but different errors. The tree produces fewer false alarms, while K-NN finds more actual defects. Equal accuracy does not make them operationally interchangeable.

The ANN has the strongest supplied F1 and accuracy, but it is not automatically the deployable choice. A strict memory limit, explanation requirement or different error cost can change what is suitable.

15 | HARD REQUIREMENTS CAN RULE OUT A HIGH-SCORING MODEL

Suppose a device has an 8 MiB budget for the measured model/pipeline state. In the teaching figures, the tree fits while the ANN and K-NN do not. A high validation score cannot make an oversized pipeline fit the device.

If recall must be at least 85%, the tree does not meet the requirement. Combining that recall target with the 8 MiB limit leaves none of these candidates eligible. The correct conclusion is not to invent a winner.

Revisit requirements where authorised, gather better evidence, redesign the pipeline or investigate another candidate. Requirements must describe the real need rather than be changed merely to legitimise a preferred model.

16 | ERROR COSTS CAN CHANGE THE RANKING

Use a simple teaching cost: FP × cost per false positive + FN × cost per false negative. These weights represent a declared operational assumption, not probabilities.

If each false positive costs 10 units and each false negative costs 1, the tree costs 10×10 + 10×1 = 110. K-NN costs 14×10 + 6×1 = 146; ANN costs 12×10 + 4×1 = 124. The tree is cheapest despite the ANN’s higher F1.

If missed defects cost more, the result can reverse. This small formula assumes errors add independently and ignores other costs unless explicitly included. Real utility can be more complex, so justify any cost model and check its sensitivity.

17 | SELECT UNDER EXPLICIT REQUIREMENTS

Change the teaching objective and constraints. The activity recomputes metrics from the supplied confusion counts, filters ineligible candidates and reports its reasoning. It is a decision exercise, not a live training or hardware benchmark.

Try highest F1 with the defaults, then lowest latency. For a cost example, set FP cost to 10 and FN cost to 1. Next set minimum recall to 85% and memory to 8 MiB to see a valid “no eligible candidate” outcome.

The tree-path option is a specific teaching requirement, not a claim that other algorithms cannot be explained or that every tree is transparent. Costs affect ranking only when the cost objective is selected. Exact objective ties use lower latency, then the listed candidate order; tiny differences still need uncertainty checks.

18 | EXPLAINABILITY IS A REQUIREMENT TO DEFINE

A short tree path can show which tests produced a result. K-NN can expose neighbours and votes. A linear model can expose coefficients, while a larger nonlinear model may need additional analysis to describe its behaviour.

These explanations have different meanings. A visible path or coefficient does not establish causation, fairness or correctness. A very deep tree can also be difficult to inspect.

Specify what an explanation must support: user understanding, error investigation, a review process or a particular operational decision. Then check that evidence rather than assigning a universal “interpretable/uninterpretable” label from the model name alone.

19 | DEPLOYMENT COST IS MORE THAN TRAINING TIME

Training cost, inference latency, throughput, memory and energy can lead to different choices. A model trained remotely may still need to fit and run within an edge device’s response budget.

Measure the end-to-end path: input collection, preprocessing, transfers, model execution and output handling. A fast model call does not guarantee a fast service if its inputs or network connection dominate.

Use comparable hardware, load and measurement conventions. Average latency can hide occasional slow responses; include relevant distributions or deadlines. The lesson’s fictional numbers illustrate constraints, not universal timing relationships between algorithms.

20 | CHECK VARIATION RATHER THAN OVERREADING A SMALL GAP

Performance changes with data splits, random initialisation, sampling noise and changes in the population. A small validation difference may not repeat on new examples.

Compare appropriate repeated folds or runs, inspect the variation and relevant error cases, and use independent evaluation. A fixed seed makes a run reproducible; it does not prove the selected model is reliably better.

Summary variation across folds can be useful, but folds are not automatically independent experiments and a simple displayed spread is not itself a significance test. Report what was measured and avoid claims stronger than the evidence.

21 | LOOK BEYOND THE OVERALL AVERAGE

A model can perform well overall yet fail on a rare condition, a particular group or images from a different camera. Examine relevant slices and the consequences of the errors.

A change in deployment data can alter the ranking that held during development. Monitor inputs and outcomes, investigate drift and set a review or retraining process appropriate to the application.

Selecting once is not proof of permanent suitability. Deployment monitoring complements independent initial evaluation; it is not permission to train silently on every new outcome without considering quality, privacy and the evaluation design.

22 | RECORD THE CHOICE SO SOMEONE CAN REVIEW IT

Document the task, positive-class or target definition, candidate pipelines, data versions, splitting rules, tuning budget, metrics, constraints and the reasons for selecting or rejecting candidates.

State which figures are development validation results and which are final independent test results. Include the limits: missing coverage, uncertainty, resource measurements and important failures.

A useful justification connects evidence to the requirement: “The tree is selected under this cost model because it has fewer false alarms and meets the memory limit. Its lower recall is a known limitation.” Naming a fashionable algorithm is not a justification.

23 | CHECK THE COMPARISON PRINCIPLE

Choose the main principle illustrated by each scenario. Look for the reason a comparison is useful or misleading, not just its numerical winner.

24 | EXPLAIN WHY MODEL SELECTION MATTERS

Build the explanation from the problem: different algorithms represent patterns differently, so the nature of the target and the characteristics of the data affect their results. Appropriate preprocessing and complexity choices also affect the comparison.

Then connect the evidence: candidates should be compared through honest validation on suitable examples using task-relevant metrics. Error costs, resources and explanation needs can make a model with a lower headline score the more suitable choice.

Finish with the limit: the selected pipeline still needs independent final evaluation and monitoring. There is no universally best model, and a result from one dataset does not prove superiority for every task.

Further reading: model selection and evaluation and separating selection from independent evaluation.

MATCH THE MODEL-SELECTION IDEA TO ITS ROLE

Terminology

Terminology

Model selection

Choosing a suitable model and configuration using task-specific evidence.

Model comparison

Evaluating alternatives under meaningful and reasonably comparable conditions.

Pipeline

The connected preprocessing, feature handling and fitted model.

Baseline

A simple reference method evaluated to establish useful comparative value.

Target

The required outcome being predicted or otherwise modelled.

Data characteristic

A property such as structure, size, scale, noise or class balance affecting suitability.

Generalisation

Performance on genuinely new examples from the intended population.

Training set

Examples used to fit model parameters and learned preprocessing.

Validation

Development evaluation used to guide choices.

Test set

Independent examples reserved for final evaluation of the selected pipeline.

Hyperparameter tuning

Comparing configuration choices under a declared procedure and objective.

Data leakage

Use of information that should not legitimately influence fitting or evaluation.

Cross-validation

Repeated fitting and held-out evaluation within development data.

Nested evaluation

An outer evaluation separated from an inner model-selection procedure.

Class imbalance

Uneven numbers of examples across classes.

False positive

An actual negative predicted positive under the stated class convention.

False negative

An actual positive predicted negative under the stated class convention.

Precision

True positives divided by predicted positives, when defined.

Recall

True positives divided by actual positives, when defined.

F1

A harmonic combination of precision and recall under stated conventions.

Error cost

A declared consequence or weight assigned to a type of prediction mistake.

Hard requirement

A constraint that a candidate must meet to be eligible.

Latency

Time taken for an individual response under stated measurement conditions.

Throughput

Amount of work completed per unit time under stated conditions.

Memory budget

The allowed storage for the defined measured pipeline state.

Explainability

Evidence supporting an understanding of model behaviour for a defined purpose.

Reproducibility

Ability to repeat a described procedure under controlled conditions.

Sampling variability

Variation caused by which observations are present in fitting or evaluation samples.

Drift

A relevant change in deployment inputs, relationships or conditions over time.

Decision record

A documented justification including task, evidence, constraints and limitations.

Questions

Questions

CHECK YOUR UNDERSTANDING

Select all correct choices. Each exact set earns one point.

1. Which explain why different models can perform differently?
2. Which are fair comparison principles?
3. Which distinguish task outputs?
4. Which describe the parcel counts?
5. Under highest F1 and the default constraints, which are correct?
6. Which follow from recall ≥85% and memory ≤8 MiB in the supplied data?
7. If FP costs 10 and FN costs 1, which are correct?
8. Which preserve final evaluation independence?
9. Which are honest practical comparisons?
10. Which are true after selection?

EXPLAIN AND COMPARE

Write your answers before revealing each sample.

1. Explain why no algorithm is the best choice for every machine-learning problem.

2. Explain how data characteristics can change a model ranking.

3. Explain why a baseline is useful.

4. Explain how to compare pipelines fairly.

5. Calculate tree and K-NN accuracy and recall from the parcel counts, then explain the difference.

6. Calculate the tree and ANN weighted costs when FP costs 10 and FN costs 1.

7. Explain why no supplied candidate meets recall≥85% and memory≤8 MiB.

8. Explain why interpretability must be defined rather than assigned from an algorithm name.

9. Explain how validation and final testing should be separated.

10. Explain why a small validation-score advantage is not conclusive.

11. Explain why resource measures can change the deployment choice.

12. Explain what should happen after selecting a model.

EXPLAIN IT | DEFEND A REVIEWABLE CHOICE

Use the parcel evidence to explain why highest F1, lowest latency and a weighted error-cost objective can produce different choices. Include data characteristics, a suitable baseline, fair pipeline comparison, hard requirements and independent final testing. Explain what to report if no candidate satisfies the requirements.

Flashcards

Flashcards

Click to flip. Select the ideas you need to revisit.

0 cards selected for revision.

Selections are kept while this page is open.

Workbook

Workbook

COMING SOON

The workbook for A4.3.10 Model Selection and Comparison (HL) is coming soon. Complete the matching task and written questions in your exercise book.