Learn
A4.3.3 | HYPERPARAMETER TUNING (HL ONLY)
01 | THE SAME ALGORITHM CAN BEHAVE DIFFERENTLY
In the previous lesson, K-NN predicted a category by consulting nearby labelled examples. Changing k changed which examples voted. A decision tree could also change its predictions when its permitted depth changed.
These settings are hyperparameters: configuration choices that control fitting or prediction. Hyperparameter tuning compares suitable settings using a defined evaluation procedure, aiming to improve performance for the intended task.
The goal is useful performance on genuinely new examples, not the most impressive training score. This lesson connects configuration choices, underfitting and overfitting with accuracy, precision, recall and F1. All numerical scenarios are fictional.
02 | PARAMETERS AND HYPERPARAMETERS HAVE DIFFERENT ROLES
| Quantity | Role | Example |
|---|---|---|
| Model parameter or learned structure | Learned from the training examples during fitting | Regression slope; a tree’s fitted split thresholds |
| Hyperparameter | Configuration chosen for the fitting or prediction procedure | K-NN neighbour count; maximum tree depth; minimum leaf size |
In linear regression, fitting might learn slope 2.4 and intercept 3.8. In a tree, fitting learns which feature to test and where to split. Those are different from choosing a maximum allowed depth before a candidate fit.
A hyperparameter need not always be chosen manually: a search procedure can choose it using validation results. Its role remains a configuration choice rather than a fitted coefficient.
Basic K-NN retains labelled examples instead of learning a single equation. It still has hyperparameters such as k, the distance metric and voting weights.
03 | DEFINE WHAT SUCCESS MEANS BEFORE SEARCHING
Suppose a parcel-screening classifier predicts whether an item is defective. Missing a defect and unnecessarily inspecting a good item are different mistakes. A single count of correct predictions may hide that difference.
Decide which class matters for reporting, what errors cost and which metric or constraint will guide the search. For example, a screening stage might require recall of at least 90% while comparing how many unnecessary inspections are created.
Choose the evaluation rule before looking for a winning setting. Otherwise it is easy to change the objective until a preferred model appears best. A different application can reasonably require a different choice.
04 | KEEP TRAINING, VALIDATION AND TEST ROLES CLEAR
The training set provides the examples used to fit a candidate model. The validation set supplies independent examples used to compare settings. The test set is reserved for evaluating the final chosen pipeline.
For each candidate, fit using training examples, obtain predictions for the same validation examples and calculate the chosen metric. Comparing candidates on different convenient samples would confuse setting effects with data differences.
After selecting the configuration, a common procedure is to refit it using the combined training and validation data, then evaluate once on the untouched test set. Keep learned preprocessing and selection inside that fitting procedure too. There is no compulsory split percentage suitable for every dataset.
05 | WHAT A TEST SET CANNOT DO FOR YOU
If you try k = 1, 3, 5 and 9 on the test set and choose the best result, that set has become part of tuning. Its reported winning score is no longer a clean independent final estimate.
Repeatedly adapting choices to the same validation set can also overfit that set. A large search gives more opportunities to find a setting that looks good by chance.
Use appropriate validation, a controlled search and a final independent evaluation. If results lead you to redesign the model after a test, report that limitation and obtain a fresh independent evaluation where needed rather than calling the same test untouched.
06 | UNDERFITTING: TOO RESTRICTIVE TO CAPTURE THE PATTERN
An underfitted model does not capture enough of the useful relationship. A very shallow tree may ask too few questions to separate important groups, producing weak performance on both training and validation examples.
A very large k can also smooth away a small genuine class because a broad vote includes many less relevant examples. However, large k does not automatically cause underfitting in every dataset.
Weak scores alone are not proof of underfitting. Wrong labels, inadequate inputs, distribution changes or a bug can also cause poor results. Investigate the reason before simply making the model more complex.
07 | OVERFITTING: TOO MUCH ATTENTION TO THE TRAINING SAMPLE
An overfitted model captures training-specific noise or accidental details that do not transfer. A deep tree can isolate unusual training records, producing excellent training performance but poorer validation performance.
K-NN with k = 1 can be sensitive to individual noisy examples. Testing it against the identical stored training examples can be especially misleading because each example may be its own nearest neighbour.
A substantial training–validation gap can be evidence of overfitting, but it is not the only possible cause. Check leakage, representative splitting and differences in data collection. More complexity is not always more understanding.
08 | READ A COMPLEXITY COMPARISON
| Maximum tree depth | Illustrative training accuracy | Illustrative validation accuracy | Interpretation to investigate |
|---|---|---|---|
| 1 | 74% | 72% | May be too restrictive |
| 2 | 85% | 83% | Captures more useful structure |
| 3 | 94% | 89% | Best supplied validation accuracy |
| 5 | 100% | 84% | Training improves while validation falls |
| 10 | 100% | 79% | Further detail does not help here |
The numbers are supplied teaching results, not measurements from a live model. They illustrate why selecting depth 10 because it has perfect training accuracy would be a weak decision. Depth 3 has the strongest validation result under this particular metric.
The best depth is task-dependent. Changes in sample composition or label quality can change the pattern. Compare variation across suitable validation runs, not only a single ranking.
09 | NAME THE POSITIVE CLASS FIRST
For the rest of this lesson, positive means defective and negative means not defective. Positive does not mean good, approved or correct. It names the class whose detection we are examining.
Every evaluated parcel has an actual label and a predicted label. Combining them gives four possible outcomes: true positive, false positive, false negative and true negative.
“True” means the prediction matches the actual label; “false” means it does not. “Positive” or “negative” describes the prediction. Keeping these two parts separate makes the abbreviations easier to understand.
10 | BUILD THE CONFUSION MATRIX
| Actual class ↓ / Predicted class → | Predicted defective (+) | Predicted not defective (−) |
|---|---|---|
| Actually defective (+) | TP = 16: detected defects | FN = 4: missed defects |
| Actually not defective (−) | FP = 8: unnecessary defect flags | TN = 72: correctly passed good items |
There are 100 evaluated parcels: 20 actual defects and 80 actual good items. The model predicts defective for 24 parcels, of which 16 really are defective.
A false positive sends a good parcel for unnecessary inspection. A false negative misses a defective parcel. Which error is more costly depends on the application.
Matrix layouts vary between sources. Read the axis labels instead of memorising a square position. Here rows are actual and columns are predicted.
11 | ACCURACY: HOW MANY PREDICTIONS ARE CORRECT?
Accuracy = (TP + TN) / total. In the example, (16 + 72) / 100 = 0.88, or 88%. It counts correct predictions across both classes.
Accuracy is useful when an overall correctness rate reflects the task, but the class distribution and error consequences still matter. A model can perform well on a large common class while failing to find a small important class.
The denominator includes every evaluated example. Do not divide only by predicted positives or actual positives; those denominators answer different questions.
12 | PRECISION: CAN WE TRUST A POSITIVE FLAG?
Precision = TP / (TP + FP). It asks: among the examples predicted positive, how many were actually positive?
Here, 16 of 24 defect flags are correct, so precision = 16 / (16 + 8) ≈ 0.667, or 66.7%. The other eight flags concern good parcels.
Higher precision means fewer false positives relative to the positive predictions, which can matter when reviewing each alert is costly. It does not tell us how many actual defects the model missed.
13 | RECALL: HOW MANY REAL POSITIVES DID WE FIND?
Recall = TP / (TP + FN). It asks: among the actual positive examples, how many did the model identify?
Here, 16 of 20 actual defects are detected, so recall = 16 / (16 + 4) = 0.80, or 80%. Four actual defects were missed.
Higher recall means fewer missed positives relative to the actual positives. A model could achieve recall of 100% by flagging every parcel, but that would create many false positives. Recall must be interpreted with the rest of the matrix.
14 | DO NOT SWAP THE TWO DENOMINATORS
| Metric | Question | Which group is the denominator? |
|---|---|---|
| Accuracy | How many predictions were correct overall? | All evaluated examples |
| Precision | Of the defect flags, how many were correct? | Predicted positives: TP + FP |
| Recall | Of the actual defects, how many were found? | Actual positives: TP + FN |
A memory aid is to read the population aloud before calculating. “Of everything we flagged” points to precision. “Of everything actually defective” points to recall.
Precision and recall can move together or trade off as the model changes. There is no law that every increase in one must lower the other; a better representation can improve both.
15 | F1: COMBINE PRECISION AND RECALL
F1 = 2 × precision × recall / (precision + recall). It is the harmonic mean, so a very weak component pulls the result down more than an ordinary arithmetic average would.
For this matrix, an equivalent convenient formula is F1 = 2TP / (2TP + FP + FN). Therefore F1 = 32 / (32 + 8 + 4) = 32 / 44 ≈ 0.727, or 72.7%.
F1 gives precision and recall balanced roles, but it does not use TN directly and does not encode the actual financial or safety cost of each error. It is not automatically the best objective for every task. If one error needs priority, use appropriate requirements or a different objective.
16 | HANDLE ZERO DENOMINATORS HONESTLY
If a classifier predicts no positives, TP + FP = 0 and precision is undefined under its ordinary formula. If there are no actual positives, TP + FN = 0 and recall is undefined.
The direct F1 formula has a zero denominator when TP, FP and FN are all zero. If actual positives exist but the model predicts all negative, F1 is 0 under the direct count formula even though precision itself is undefined.
Software may report zero, omit a result or warn under a chosen convention. State that convention instead of silently calling undefined precision perfect. The calculator labels zero-denominator metrics “not defined” and uses the direct count formula for F1.
17 | EXPLORE THE FOUR METRICS
Enter nonnegative whole-number confusion counts. The calculator shows the formulas, metrics and actual class totals. The two preset examples demonstrate how high accuracy can hide poor defect detection.
Try a matrix containing only true negatives. Accuracy is defined for a nonempty set, but precision, recall and the direct F1 formula have zero denominators. An empty evaluation set does not have an accuracy.
18 | THE MAJORITY-CLASS TRAP
Suppose only 10 of 100 parcels are defective. A classifier that always predicts “not defective” gets the other 90 correct: 90% accuracy. Yet it finds none of the real defects: recall = 0%.
If the model instead flags every parcel, recall becomes 100%, accuracy becomes 10% and precision becomes 10%. Neither extreme is a useful complete solution simply because one metric is high.
Class imbalance means class sizes are uneven. Inspect the confusion matrix and the relevant class metrics, and compare a simple baseline. For more than two classes, report per-class results and the averaging convention; one binary F1 value cannot describe every class automatically.
19 | A WORKED K-NN TUNING COMPARISON
| k | Training F1 (supplied) | Validation TP | FP | FN | TN | Validation F1 |
|---|---|---|---|---|---|---|
| 1 | 0.990 | 12 | 12 | 8 | 68 | 0.545 |
| 3 | 0.930 | 16 | 8 | 4 | 72 | 0.727 |
| 5 | 0.880 | 17 | 10 | 3 | 70 | 0.723 |
| 9 | 0.770 | 10 | 14 | 10 | 66 | 0.455 |
Each candidate was fitted using the same training split and evaluated on the same validation split of 20 actual defects and 80 good items. These are supplied fictional results; the page does not train the candidates.
If validation F1 is the predeclared objective, k = 3 has the best supplied result, narrowly ahead of k = 5. The largest training F1 at k = 1 is not the selection criterion.
The small F1 gap between k = 3 and k = 5 is not strong evidence that k = 3 is universally better. Further suitable validation can assess stability. If recall of at least 85% is an actual requirement, k = 5 meets it here while k = 3 does not.
20 | CHANGE THE OBJECTIVE, THEN EXPLAIN THE CHOICE
Choose a rule for the supplied validation results. Compare the candidates and see why an operational requirement can change the choice. In practice, declare the rule before searching; changing it here is only a teaching exercise.
The comparison recalculates validation metrics from the supplied counts. It does not expose any final test results, fit models or prove that one configuration will win on future data.
21 | GRID SEARCH AND RANDOM SEARCH
A grid search tries specified combinations of hyperparameter values. For example, four k values and two voting-weight choices give 4 × 2 = 8 candidate configurations.
A random search samples a declared number of configurations from specified choices or ranges. It can explore a broad space under a compute budget but does not guarantee that it will sample the best combination.
Each configuration requires fitting and evaluation. With five-fold cross-validation, the eight-combination grid requires 8 × 5 = 40 candidate fold fits, plus any final refit. Search cost and the risk of overusing validation both matter.
22 | CROSS-VALIDATION MAKES MORE USE OF THE DEVELOPMENT DATA
In five-fold cross-validation, divide the development data into five parts. For one candidate, fit on four parts and evaluate on the remaining part, repeating so every part is held out once. Summarise the five held-out scores.
Repeat the same fold procedure for each candidate. Fit imputation, scaling and feature selection separately inside each training fold, then apply those fitted rules to its held-out fold. Otherwise the pipeline can leak information.
Use splits appropriate to the task. Related records may need grouping, future prediction may need time-aware splits, and stratification can help preserve class proportions when appropriate. Cross-validation is not five independent final test sets; its results are being used to choose settings.
23 | FROM A WINNING SCORE TO A REVIEWABLE DECISION
Select a configuration using the declared metric, constraints and validation evidence. Consider variation across folds, useful baselines, inference speed, memory and the cost of errors. Report the data and procedure that produced the result.
Refit the selected pipeline using the allowed development data, evaluate on the independent test set and explain limitations. A strong average can still conceal weak performance for a particular group or rare condition.
If a classifier exposes a score or probability estimate, its decision threshold can also be tuned using development data. This can change precision and recall without retraining the underlying score model. Do not confuse a threshold choice with a guaranteed probability of correctness or a reason to tune on the final test.
24 | EXPLAIN WHY THE PROCESS MATTERS
Build a connected explanation: “Changing k changes the neighbourhood, therefore it can alter sensitivity to noise and minority groups. Candidates are compared on validation examples because training scores can reward memorisation. The metric reflects the task’s error costs, so the highest accuracy is not always the most suitable result.”
For precision and recall, name the positive class and the denominator. For model complexity, link underfitting to missed useful structure and overfitting to training-specific details. For tuning, explain why the final test must remain outside the search.
There is no universally best hyperparameter setting or metric. The purpose is to choose and evaluate a suitable pipeline using evidence that reflects its intended deployment.
Further reading: F1 and zero-denominator conventions, validation and data separation, and hyperparameter-search procedures.