Learn
A4.3.10 | MODEL SELECTION AND COMPARISON (HL ONLY)
01 | THERE IS NO SINGLE WINNER FOR EVERY TASK
A model with the highest score in an image experiment may be unsuitable for predicting delivery time from a small table. Even two classifiers for the same task can differ in missed cases, speed, memory and how easily their decisions can be explained.
Model selection means choosing a suitable model and configuration for a defined problem using evidence. Model comparison is the process of evaluating alternatives under meaningful and reasonably comparable conditions.
This HL lesson brings together the earlier approaches. The central question is not “Which algorithm is best?” but “Which validated pipeline meets this task’s requirements, on these data, within these constraints?”
02 | DEFINE THE OUTCOME BEFORE CHOOSING THE ALGORITHM
First describe the required output. A measured duration is a regression target; a defective/not-defective label is a classification target. Discovering purchasing groups without known segment labels is clustering.
A rule about products appearing together is an association-mining output. Learning a sequence of actions from rewards is a reinforcement-learning formulation. Searching for a shorter complete route can be an optimisation task.
These approaches do not all answer the same question, so their headline scores cannot be ranked as if they did. An algorithm that solves the wrong problem cannot become suitable merely by being more complex.
03 | START WITH THE DATA YOU REALLY HAVE
Ask what one example represents, how many useful examples exist, which inputs are available at prediction time and whether reliable targets are known. Check missing values, duplicates, class balance and coverage of the intended population.
For images, spatial organisation can make convolutional structure a useful starting point. For a small structured table, a simpler regression or tree may be easier to fit and evaluate. These are reasons to investigate candidates, not automatic winners.
Exclude illegitimate inputs before chasing a score. A value recorded only after the outcome can create leakage even if every candidate algorithm handles it perfectly.
04 | DATA CHARACTERISTICS CHANGE THE COMPARISON
| Data characteristic | Why it matters | Possible implication to investigate |
|---|---|---|
| Approximately linear relationship | A simple mapping may already capture the signal | A linear baseline can be competitive and easy to inspect |
| Nonlinear interactions | An effect may depend on combinations of inputs | A tree or nonlinear ANN may capture useful structure |
| Many irrelevant dimensions | Distance and fitting can be distracted by noise | Feature selection or reduction may help the pipeline |
| Different numeric scales | Distance- or gradient-based computation may be affected | Use candidate-appropriate fitted preprocessing |
| Few examples or noisy labels | Flexible models can fit accidents | Simpler or constrained models may generalise better |
| Uneven classes | Overall accuracy can conceal missed minority cases | Inspect class-specific metrics and error counts |
| Spatial image structure | Nearby values contain useful relationships | A CNN can exploit local shared patterns |
A property changes what is plausible, not what is guaranteed. For example, ordinary trees are often less sensitive to numerical scaling than K-NN, but they still require consistent feature meanings and valid data.
05 | WHY TWO ALGORITHMS CAN REACH DIFFERENT ANSWERS
Each approach makes different representational choices. Linear regression fits an affine numerical relationship. K-NN relies on nearby examples under its metric. A tree partitions feature space using tests. A nonlinear ANN composes learned transformations.
Those choices affect which patterns are easy to capture, how noise influences fitting and what happens in sparsely represented regions. Hyperparameters and preprocessing further change the behaviour.
The resulting differences are not necessarily implementation errors. Compare predictions and failure cases to understand whether each method’s assumptions and flexibility match the task.
06 | A WORKED EXAMPLE OF A CHANGING RANKING
| Candidate | Illustrative validation MAE: mostly linear data | Illustrative validation MAE: nonlinear-interaction data |
|---|---|---|
| Linear regression | 2.1 minutes | 7.2 minutes |
| Decision-tree regressor | 3.0 minutes | 3.2 minutes |
| Nonlinear ANN regressor | 2.4 minutes | 2.8 minutes |
These supplied fictional results illustrate two different regression datasets. Within each dataset, candidates use comparable development splits and the same target and metric. Lower MAE is better.
The linear model leads in the first setting, while the ANN leads in the second. This demonstrates a possible change in ranking when the relationship changes; it is not a measured general rule about these algorithms.
Do not compare the numbers across the two datasets as proof that one dataset is universally easier. Different outcome distributions and sampling conditions affect the meaning of a score. The useful comparison is between candidates within the stated evaluation context.
07 | USE A SIMPLE BASELINE
A baseline is a simple reference. A classification baseline might always predict the training set’s most common class. A regression baseline might always predict the training mean or median under an appropriate metric.
If 80% of development examples are good parcels, always predicting good can achieve 80% accuracy while detecting no defects. Beating that accuracy is useful evidence, but defect-specific errors still matter.
Fit a deployable baseline from training data, then evaluate it on the same held-out examples as the other candidates. A complex model should justify its additional burden through meaningful benefit, not merely a complicated architecture.
08 | COMPARE WHOLE PIPELINES
A pipeline includes preprocessing, feature selection or reduction and the fitted model. K-NN with inappropriate unscaled features is not a fair representation of what a carefully prepared K-NN candidate can do.
Fair comparison does not require identical transformations for every algorithm. It requires compatible input information, legitimate preprocessing fitted inside training, and a transparent procedure suitable for each candidate.
Record units, feature choices, missing-value handling and class encoding. If one model has extra information unavailable to another, make that difference explicit rather than attributing the whole performance gap to algorithm choice.
09 | KEEP THE EVALUATION EXAMPLES COMPARABLE
Use common held-out examples or suitably matched cross-validation folds when comparing candidates for the same task. Otherwise one candidate can appear better simply because it was evaluated on easier cases.
Choose splits that reflect the intended deployment. Keep related records together when needed, preserve time order for future prediction tasks and prevent duplicates from crossing fitting and evaluation boundaries.
Fit learned scaling, imputation and feature selection only on the relevant training partition. Splitting after those transformations may let held-out data influence the fitted pipeline.
10 | MODEL SELECTION ALSO INCLUDES TUNING
A tree’s maximum depth or K-NN’s k can make a large difference. Comparing one carefully tuned algorithm against an arbitrary poor setting of another can be misleading.
Declare appropriate tuning ranges, objectives and search budgets. Equal wall-clock time and equal configuration counts are not the same form of fairness; describe the budget and justify it for the purpose.
Selecting the model family, features, thresholds and hyperparameters are all development choices. Keep the final test data outside those decisions. Trying more candidates also creates more opportunities to overfit validation evidence.
11 | VALIDATION IS FOR CHOICES; TESTING IS FOR THE FINAL ESTIMATE
Use training examples to fit candidates and validation procedures to guide choices. After selection, a common procedure is to refit the chosen configuration on the allowed training-plus-validation development data.
Then evaluate the chosen pipeline on an independent final test set. If test results lead you to redesign and retune, that test is no longer an untouched final check. Report the limitation and obtain an appropriate independent evaluation.
When extensive tuning is combined with cross-validation, nested evaluation can separate the inner selection process from outer evaluation. The important principle is independence, not memorising one compulsory split percentage or advanced procedure.
12 | CHOOSE METRICS FOR THE OUTCOME AND ERROR COSTS
| Task | Useful comparison evidence | Limit to remember |
|---|---|---|
| Classification | Confusion counts, accuracy, precision, recall and F1 under stated class conventions | Accuracy can hide rare-class failure; F1 does not encode every error cost |
| Regression | MAE or other numerical errors; R² with its reference context | A unitless fit score is not the same as error in useful units |
| Clustering | Structure, stability, suitable internal measures and downstream usefulness | No known target groups means ordinary accuracy is not directly available |
| Sequential decisions or optimisation | Return/completion or objective value with validity and resource checks | A large proxy score can miss the real objective |
Do not compare 0.9 classification accuracy with 0.8 regression R² and conclude the classifier is better. They answer different questions on different target tasks.
13 | DEFINE THE PARCEL COMPARISON
Consider a fictional binary classifier screening parcels. Positive means defective. Each candidate is evaluated on the same 200 validation parcels: 40 defective and 160 not defective.
The task’s measured inputs are tabular descriptors, not raw images. The candidates are a constant baseline, a constrained decision tree, a scaled K-NN pipeline and a small nonlinear ANN. The supplied values illustrate selection reasoning; no candidates were trained or timed to produce this teaching table.
| Candidate | TP | FP | FN | TN | Illustrative latency | Illustrative model/pipeline memory |
|---|---|---|---|---|---|---|
| Baseline: always good | 0 | 0 | 40 | 160 | 1 ms | 1 MiB |
| Decision tree | 30 | 10 | 10 | 150 | 2 ms | 4 MiB |
| K-NN | 34 | 14 | 6 | 146 | 18 ms | 32 MiB |
| ANN | 36 | 12 | 4 | 148 | 5 ms | 12 MiB |
The time and memory figures are fictional measurements under one stated teaching setup, not product benchmarks or general claims about these model families. A real comparison must measure the relevant pipeline on the actual target hardware and workload.
14 | READ PERFORMANCE BEFORE CHOOSING
| Candidate | Accuracy | Precision for defects | Recall for defects | F1 for defects |
|---|---|---|---|---|
| Baseline | 80% | Undefined: no flags | 0% | 0 |
| Decision tree | 90% | 75% | 75% | 0.750 |
| K-NN | 90% | ≈70.8% | 85% | ≈0.773 |
| ANN | 92% | 75% | 90% | ≈0.818 |
The tree and K-NN have equal accuracy but different errors. The tree produces fewer false alarms, while K-NN finds more actual defects. Equal accuracy does not make them operationally interchangeable.
The ANN has the strongest supplied F1 and accuracy, but it is not automatically the deployable choice. A strict memory limit, explanation requirement or different error cost can change what is suitable.
15 | HARD REQUIREMENTS CAN RULE OUT A HIGH-SCORING MODEL
Suppose a device has an 8 MiB budget for the measured model/pipeline state. In the teaching figures, the tree fits while the ANN and K-NN do not. A high validation score cannot make an oversized pipeline fit the device.
If recall must be at least 85%, the tree does not meet the requirement. Combining that recall target with the 8 MiB limit leaves none of these candidates eligible. The correct conclusion is not to invent a winner.
Revisit requirements where authorised, gather better evidence, redesign the pipeline or investigate another candidate. Requirements must describe the real need rather than be changed merely to legitimise a preferred model.
16 | ERROR COSTS CAN CHANGE THE RANKING
Use a simple teaching cost: FP × cost per false positive + FN × cost per false negative. These weights represent a declared operational assumption, not probabilities.
If each false positive costs 10 units and each false negative costs 1, the tree costs 10×10 + 10×1 = 110. K-NN costs 14×10 + 6×1 = 146; ANN costs 12×10 + 4×1 = 124. The tree is cheapest despite the ANN’s higher F1.
If missed defects cost more, the result can reverse. This small formula assumes errors add independently and ignores other costs unless explicitly included. Real utility can be more complex, so justify any cost model and check its sensitivity.
17 | SELECT UNDER EXPLICIT REQUIREMENTS
Change the teaching objective and constraints. The activity recomputes metrics from the supplied confusion counts, filters ineligible candidates and reports its reasoning. It is a decision exercise, not a live training or hardware benchmark.
Try highest F1 with the defaults, then lowest latency. For a cost example, set FP cost to 10 and FN cost to 1. Next set minimum recall to 85% and memory to 8 MiB to see a valid “no eligible candidate” outcome.
The tree-path option is a specific teaching requirement, not a claim that other algorithms cannot be explained or that every tree is transparent. Costs affect ranking only when the cost objective is selected. Exact objective ties use lower latency, then the listed candidate order; tiny differences still need uncertainty checks.
18 | EXPLAINABILITY IS A REQUIREMENT TO DEFINE
A short tree path can show which tests produced a result. K-NN can expose neighbours and votes. A linear model can expose coefficients, while a larger nonlinear model may need additional analysis to describe its behaviour.
These explanations have different meanings. A visible path or coefficient does not establish causation, fairness or correctness. A very deep tree can also be difficult to inspect.
Specify what an explanation must support: user understanding, error investigation, a review process or a particular operational decision. Then check that evidence rather than assigning a universal “interpretable/uninterpretable” label from the model name alone.
19 | DEPLOYMENT COST IS MORE THAN TRAINING TIME
Training cost, inference latency, throughput, memory and energy can lead to different choices. A model trained remotely may still need to fit and run within an edge device’s response budget.
Measure the end-to-end path: input collection, preprocessing, transfers, model execution and output handling. A fast model call does not guarantee a fast service if its inputs or network connection dominate.
Use comparable hardware, load and measurement conventions. Average latency can hide occasional slow responses; include relevant distributions or deadlines. The lesson’s fictional numbers illustrate constraints, not universal timing relationships between algorithms.
20 | CHECK VARIATION RATHER THAN OVERREADING A SMALL GAP
Performance changes with data splits, random initialisation, sampling noise and changes in the population. A small validation difference may not repeat on new examples.
Compare appropriate repeated folds or runs, inspect the variation and relevant error cases, and use independent evaluation. A fixed seed makes a run reproducible; it does not prove the selected model is reliably better.
Summary variation across folds can be useful, but folds are not automatically independent experiments and a simple displayed spread is not itself a significance test. Report what was measured and avoid claims stronger than the evidence.
21 | LOOK BEYOND THE OVERALL AVERAGE
A model can perform well overall yet fail on a rare condition, a particular group or images from a different camera. Examine relevant slices and the consequences of the errors.
A change in deployment data can alter the ranking that held during development. Monitor inputs and outcomes, investigate drift and set a review or retraining process appropriate to the application.
Selecting once is not proof of permanent suitability. Deployment monitoring complements independent initial evaluation; it is not permission to train silently on every new outcome without considering quality, privacy and the evaluation design.
22 | RECORD THE CHOICE SO SOMEONE CAN REVIEW IT
Document the task, positive-class or target definition, candidate pipelines, data versions, splitting rules, tuning budget, metrics, constraints and the reasons for selecting or rejecting candidates.
State which figures are development validation results and which are final independent test results. Include the limits: missing coverage, uncertainty, resource measurements and important failures.
A useful justification connects evidence to the requirement: “The tree is selected under this cost model because it has fewer false alarms and meets the memory limit. Its lower recall is a known limitation.” Naming a fashionable algorithm is not a justification.
23 | CHECK THE COMPARISON PRINCIPLE
Choose the main principle illustrated by each scenario. Look for the reason a comparison is useful or misleading, not just its numerical winner.
24 | EXPLAIN WHY MODEL SELECTION MATTERS
Build the explanation from the problem: different algorithms represent patterns differently, so the nature of the target and the characteristics of the data affect their results. Appropriate preprocessing and complexity choices also affect the comparison.
Then connect the evidence: candidates should be compared through honest validation on suitable examples using task-relevant metrics. Error costs, resources and explanation needs can make a model with a lower headline score the more suitable choice.
Finish with the limit: the selected pipeline still needs independent final evaluation and monitoring. There is no universally best model, and a result from one dataset does not prove superiority for every task.
Further reading: model selection and evaluation and separating selection from independent evaluation.