Learn
A4.2.2 | FEATURE SELECTION (HL ONLY)
01 | WHICH INFORMATION SHOULD THE MODEL SEE?
Imagine predicting a delivery’s duration before a van leaves the depot. You can provide the model with distance, planned stops, parcel mass, a booking reference and the time the delivery actually finished. Which of these should it use?
More columns do not automatically mean better predictions. Some contain useful information, some repeat information and some reveal an answer that will not exist when a real prediction is needed. Feature selection identifies and retains useful input attributes while excluding unsuitable or unhelpful ones.
This HL lesson follows the delivery example from data cleaning. We will examine individual inputs, compare combinations and distinguish the three syllabus strategies: filter, wrapper and embedded methods.
02 | FEATURES ARE INPUTS, NOT THE ANSWER
A feature is an input attribute used by a model. DistanceKm and PlannedStops are possible features. The target is the outcome to predict, here DurationMinutes. One row represents one delivery.
Feature selection chooses columns for the model, not which delivery rows to keep. A selected feature can be useful without being sufficient on its own: distance may matter, but two equally long journeys can take different times because their stops differ.
Before evaluating statistical usefulness, define the prediction moment. Here it is immediately before departure. Every selected input must be available then, under the intended deployment process.
03 | LOOK AT THE CANDIDATE ATTRIBUTES
| Attribute | What it represents | Initial judgement |
|---|---|---|
| DistanceKm | Planned route length known before departure | Plausibly informative |
| PlannedStops | Number of stops on the planned route | Plausibly informative |
| DistanceMetres | The same route length in metres | Redundant with DistanceKm |
| BookingReference | Assigned event identifier | Keep for record management, normally exclude from predictor |
| DepotCode | Same value for every training delivery | No variation within this training set |
| VanColour | Paint colour with no justified link to duration here | Candidate irrelevant feature; investigate context |
| ActualFinishTime | Recorded after delivery | Unavailable at prediction time |
| DurationMinutes | Actual elapsed duration | Target, never include as an input to its own prediction |
These are starting judgements, not a proof that a particular subset is optimal. For example, a booking reference may accidentally encode a historical pattern that does not persist. Statistical evidence and domain understanding need to work together.
04 | INFORMATIVE, IRRELEVANT AND REDUNDANT
An informative feature helps the model predict the target for the intended task. Distance is a plausible example because travelling farther often takes longer. An irrelevant feature adds no useful predictive information in that setting.
A redundant feature repeats information already provided by another. DistanceMetres is exactly 1000 times DistanceKm in a correctly converted dataset, so it does not add a new route measurement.
Redundancy and irrelevance differ. Both distance columns may be strongly associated with duration, yet carrying both may add little. Some models tolerate correlated inputs well; others become less stable or harder to interpret. Select using evidence rather than a universal ban on correlation.
05 | WHY FEWER SUITABLE FEATURES CAN HELP
Removing distracting inputs can reduce opportunities for a model to fit accidental patterns. This can improve generalisation: performance on genuinely new examples from the intended population. It may also reduce training time, working memory and the cost of collecting inputs.
A smaller input set can make the model easier to explain and maintain. If measuring a feature requires an expensive sensor, removing it may reduce deployment cost even when its computational cost is small.
These are potential benefits, not guarantees. Removing a useful feature can reduce predictive quality. The goal is an appropriate informative subset, not the smallest number of columns at any cost.
06 | OVERFITTING WITHOUT A COMPLICATED FORMULA
Suppose by chance all the blue vans in a small training sample had fast deliveries. A flexible model might learn “blue means fast”, although paint colour is not a reliable guide to delivery duration in this service.
This is one route to overfitting: learning quirks of the training examples instead of a relationship that transfers. With many candidate features and few examples, there are more opportunities to find misleading coincidences.
Feature selection can help by removing unsuitable inputs, but a selector can itself overfit. Repeatedly trying subsets against the same validation data can eventually exploit chance in that data too. Evaluation design matters as much as the selection label.
07 | REMOVE LEAKAGE BEFORE CHASING A SCORE
ActualFinishTime is known only after the delivery. It may give an excellent score when predicting DurationMinutes, especially if departure time is also known, but it cannot support a genuine pre-departure prediction.
This is target leakage: an input gives access to outcome-related information that would not legitimately be available for a new prediction. Including DurationMinutes as its own feature is an even more direct example.
Exclude these inputs by understanding the process, even if a statistical selector ranks them first. Keep the booking reference separately for joining predictions back to records; excluding it from model inputs does not require deleting it from the application database.
08 | THREE STRATEGIES, THREE SOURCES OF EVIDENCE
| Strategy | How it judges features | Main trade-off |
|---|---|---|
| Filter | Uses a data-based criterion without repeatedly fitting the final predictor to compare subsets | Usually efficient, but a simple per-feature test can miss interactions |
| Wrapper | Fits and evaluates a chosen model on candidate feature subsets | Directly tests model/subset combinations, but repeated fitting costs time |
| Embedded | Uses selection or importance arising within model training | Connects selection to learning, but depends on the model and its assumptions |
The distinction is how selection is performed, not whether the final dataset has fewer columns. All three can produce the same subset for different reasons. None automatically proves causation, fairness or best future performance.
09 | FILTER METHODS: SCREEN THE DATA
A filter method applies a criterion to the data before choosing the final model’s inputs. Examples include removing constant features, scoring association with the target or screening strongly redundant pairs.
A numerical association score can rank how closely a feature varies with a numerical target. A constant DepotCode cannot distinguish deliveries within this training set, so a zero-variance filter can remove it.
Filters are often fast and useful when there are many columns. However, the criterion must suit the data. A linear correlation score does not capture every nonlinear relationship, and a feature weak on its own can still help in a combination.
10 | A WORKED FILTER RANKING
| Eligible input | Illustrative absolute correlation with duration | Decision under top-two rule |
|---|---|---|
| DistanceKm | 0.82 | Keep |
| PlannedStops | 0.64 | Keep |
| ParcelMassKg | 0.12 | Exclude under this rule |
| VanColourCode | 0.03 | Exclude; arbitrary numeric category coding also needs caution |
These fictional scores are supplied to explain a ranking, not calculated from the small table above. The rule keeps the two highest-scoring eligible inputs. This screening happens after excluding unavailable/leaking inputs and removing a duplicate distance representation.
An absolute correlation of 0.82 is not 82% accuracy, and it does not prove a causal effect. If a category is encoded with arbitrary numbers, ordinary numeric correlation may be misleading. Use a criterion appropriate to the feature and target types.
A different model or interaction may still benefit from parcel mass. The filter offers a candidate subset to validate, not a universal verdict that the discarded feature is worthless.
11 | A FEATURE CAN MATTER ONLY IN COMBINATION
Consider two binary switches A and B. The target is 1 when exactly one switch is on and 0 otherwise. This is the XOR pattern.
| A | B | Target |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
Across these balanced examples, seeing A alone does not change the proportion of targets that are 1; the same is true for B. Yet together they determine the answer exactly.
A simple individual association screen can miss such an interaction. A model and selection procedure capable of evaluating the combination may retain both. This does not mean every wrapper discovers every interaction: a greedy search that adds only individually helpful features may stop too early.
12 | WRAPPER METHODS: TRY SUBSETS WITH A MODEL
A wrapper method evaluates feature subsets by fitting a particular model and measuring its performance using appropriate training/validation procedures. The model acts as the way to judge the subset.
For a duration predictor, a score might be mean absolute error: the average absolute difference between predicted and actual minutes. A lower value is better for this metric. Fit on training examples and measure on validation examples; training error alone is not a reliable selection target.
Because it evaluates the chosen model, a wrapper can reflect how that model uses features together. The cost is repeated fitting, and the chosen subset can change when the model, metric or data split changes.
13 | FORWARD SELECTION, ONE STEP AT A TIME
| Step | Candidate subset | Illustrative validation MAE | Decision |
|---|---|---|---|
| 1 | Distance only | 8.0 min | Best single feature |
| 1 | Stops only | 10.0 min | Do not start with this |
| 1 | Mass only | 13.0 min | Do not start with this |
| 2 | Distance + stops | 5.0 min | Choose this addition |
| 2 | Distance + mass | 7.7 min | Do not choose this addition |
| 3 | Distance + stops + mass | 5.1 min | Stop: no improvement over 5.0 |
Forward selection starts with an empty selected set, tests candidate additions and retains the best allowed improvement. In this fictional walk-through, it finishes with distance and stops.
These supplied values illustrate the logic; the activity does not fit a model. In practice, account for variation between splits and a predeclared stopping rule rather than overinterpreting a tiny score difference. Greedy forward selection is not guaranteed to find the globally best subset.
14 | BACKWARD ELIMINATION AND SEARCH COST
Backward elimination starts with the candidate inputs and considers removing features, keeping removals according to the chosen validation rule. It can reach a different subset from forward selection because the search path differs.
Testing every possible subset is an alternative, but the number grows quickly. With 10 candidate features there are 2¹⁰ = 1024 subsets including the empty set, or 1023 non-empty subsets. Each may require several model fits if cross-validation is used.
The actual search cost depends on the strategy. “Wrapper” does not mean every subset must be tested. Greedy searches reduce the workload but may miss combinations, while exhaustive searches can become impractical.
15 | TRY THE FORWARD-SELECTION WALK-THROUGH
Reveal the supplied validation results step by step. At each stage, check which feature is added and why the process stops. These are fictional precomputed scores, not live training.
16 | EMBEDDED METHODS: LEARN AND SELECT TOGETHER
An embedded method uses feature selection arising within the model-fitting process. The model learns relationships while its training procedure favours or identifies particular inputs.
One example is a linear model trained with an L1 penalty. A coefficient is a learned weight linking an input to the prediction; an L1 penalty discourages large total absolute weights and can set some coefficients exactly to zero. A zero coefficient removes that input’s contribution in that fitted linear expression.
The penalty strength affects how many inputs remain. Too strong a penalty can discard useful information. Inputs need appropriate scaling before interpreting coefficient magnitudes or applying scale-sensitive penalties. This is a conceptual example, not a requirement to derive the optimisation mathematics.
17 | TREES AND FEATURE IMPORTANCE
A decision tree learns split rules such as “distance greater than 10 km?”. Only features chosen for its splits contribute to its predictions. Tree-based models can also supply importance measures used to retain selected inputs.
This connects selection to the fitted model, but importance is not a universal property of a feature. Correlated inputs can share or compete for importance, and some tree importance measures favour features with many possible split points.
A model can output importance scores without the application actually dropping any columns. To create a reduced input pipeline, apply an explicit selection rule and validate the resulting pipeline. Importance is not proof that an attribute causes the outcome.
18 | A WORKED EMBEDDED EXAMPLE
| Scaled input | Illustrative L1-fitted coefficient | Result under nonzero-coefficient rule |
|---|---|---|
| Distance | 0.70 | Retain |
| Stops | 0.25 | Retain |
| Mass | 0.00 | Exclude for this fitted linear predictor |
| Van colour indicator | 0.00 | Exclude for this fitted linear predictor |
The coefficients are illustrative supplied results, not estimates from the earlier dataset. Under this specified rule, the retained inputs are distance and stops. The fitting process produced the zero weights.
Do not describe this as a filter merely because a coefficient threshold is applied afterwards. The evidence was learned as part of fitting the model. Conversely, a wrapper could fit many models even if each model itself has embedded selection: practical pipelines can combine methods.
19 | KEEP THE EVALUATION INDEPENDENT
Split data appropriately before learning selection rules. Fit a target-based feature ranking, wrapper search or embedded selector on training data using internal validation as needed. Keep the final test data out of those choices.
In cross-validation, selection must be refitted inside each training fold. Selecting columns once using the targets from the whole dataset before cross-validation leaks information from its held-out folds.
Apply the selected feature names, order and compatible preprocessing consistently at validation, test and prediction time. Refitting a selector independently on each new batch can change what the model’s inputs mean.
20 | CHOOSE A STRATEGY FOR THE CONSTRAINTS
| Situation | Possible starting point | What to check |
|---|---|---|
| Thousands of candidate attributes; limited compute | A suitable efficient filter | Interactions and redundant retained features |
| Modest candidate set; final model already chosen | A wrapper with defined search and validation rules | Repeated fitting cost and validation overuse |
| Suitable learner supports training-based selection | An embedded method | Scaling, regularisation or importance biases |
| Expensive input measurement | Compare validated subsets and acquisition cost | Required accuracy, reliability and population coverage |
You can combine approaches: first remove unavailable inputs and constants, use a filter to reduce a huge candidate pool, then compare a smaller set of subsets. The choice should match the prediction task, data and available resources.
No strategy excuses retaining a leaking feature. No statistically selected subset guarantees fairness. Check coverage and subgroup performance, and investigate whether apparently harmless inputs act as proxies for sensitive information.
21 | CLASSIFY THE SELECTION PROCESS
Choose the method from what the procedure actually does. The activity includes leakage and deployment cases so you can distinguish selecting a method from selecting legitimate information.
22 | SELECTION, CLEANING AND DIMENSIONALITY
Cleaning addresses problems such as incorrect units, duplicate records and missing values. Feature selection chooses a subset of the original input attributes. Transforming several original attributes into new combined components is feature extraction rather than simply selecting original columns.
Selection reduces the number of input dimensions, so it is one way to reduce dimensionality. Other reduction approaches transform the representation. The next lesson explores why high-dimensional data creates difficulties; PCA and LDA calculations are not required by this syllabus.
For a useful exam description, identify how a method makes decisions and connect that to a benefit and limit: “A wrapper compares validation performance for candidate input subsets using the chosen model. This reflects that model’s behaviour, but repeatedly fitting it increases computation.”
Further reading: filter, wrapper and embedded strategies and feature-selection examples and pipeline guidance.