IB COMPUTER SCIENCE | A4.2.2 Feature Selection (HL)

Lesson objective

Describe how feature selection retains informative inputs and compare filter, wrapper and embedded strategies, including benefits, limitations and leakage prevention.

Learn

A4.2.2 | FEATURE SELECTION (HL ONLY)

01 | WHICH INFORMATION SHOULD THE MODEL SEE?

Imagine predicting a delivery’s duration before a van leaves the depot. You can provide the model with distance, planned stops, parcel mass, a booking reference and the time the delivery actually finished. Which of these should it use?

More columns do not automatically mean better predictions. Some contain useful information, some repeat information and some reveal an answer that will not exist when a real prediction is needed. Feature selection identifies and retains useful input attributes while excluding unsuitable or unhelpful ones.

This HL lesson follows the delivery example from data cleaning. We will examine individual inputs, compare combinations and distinguish the three syllabus strategies: filter, wrapper and embedded methods.

02 | FEATURES ARE INPUTS, NOT THE ANSWER

A feature is an input attribute used by a model. DistanceKm and PlannedStops are possible features. The target is the outcome to predict, here DurationMinutes. One row represents one delivery.

Feature selection chooses columns for the model, not which delivery rows to keep. A selected feature can be useful without being sufficient on its own: distance may matter, but two equally long journeys can take different times because their stops differ.

Before evaluating statistical usefulness, define the prediction moment. Here it is immediately before departure. Every selected input must be available then, under the intended deployment process.

03 | LOOK AT THE CANDIDATE ATTRIBUTES

Attribute What it represents Initial judgement
DistanceKm Planned route length known before departure Plausibly informative
PlannedStops Number of stops on the planned route Plausibly informative
DistanceMetres The same route length in metres Redundant with DistanceKm
BookingReference Assigned event identifier Keep for record management, normally exclude from predictor
DepotCode Same value for every training delivery No variation within this training set
VanColour Paint colour with no justified link to duration here Candidate irrelevant feature; investigate context
ActualFinishTime Recorded after delivery Unavailable at prediction time
DurationMinutes Actual elapsed duration Target, never include as an input to its own prediction

These are starting judgements, not a proof that a particular subset is optimal. For example, a booking reference may accidentally encode a historical pattern that does not persist. Statistical evidence and domain understanding need to work together.

04 | INFORMATIVE, IRRELEVANT AND REDUNDANT

An informative feature helps the model predict the target for the intended task. Distance is a plausible example because travelling farther often takes longer. An irrelevant feature adds no useful predictive information in that setting.

A redundant feature repeats information already provided by another. DistanceMetres is exactly 1000 times DistanceKm in a correctly converted dataset, so it does not add a new route measurement.

Redundancy and irrelevance differ. Both distance columns may be strongly associated with duration, yet carrying both may add little. Some models tolerate correlated inputs well; others become less stable or harder to interpret. Select using evidence rather than a universal ban on correlation.

05 | WHY FEWER SUITABLE FEATURES CAN HELP

Removing distracting inputs can reduce opportunities for a model to fit accidental patterns. This can improve generalisation: performance on genuinely new examples from the intended population. It may also reduce training time, working memory and the cost of collecting inputs.

A smaller input set can make the model easier to explain and maintain. If measuring a feature requires an expensive sensor, removing it may reduce deployment cost even when its computational cost is small.

These are potential benefits, not guarantees. Removing a useful feature can reduce predictive quality. The goal is an appropriate informative subset, not the smallest number of columns at any cost.

06 | OVERFITTING WITHOUT A COMPLICATED FORMULA

Suppose by chance all the blue vans in a small training sample had fast deliveries. A flexible model might learn “blue means fast”, although paint colour is not a reliable guide to delivery duration in this service.

This is one route to overfitting: learning quirks of the training examples instead of a relationship that transfers. With many candidate features and few examples, there are more opportunities to find misleading coincidences.

Feature selection can help by removing unsuitable inputs, but a selector can itself overfit. Repeatedly trying subsets against the same validation data can eventually exploit chance in that data too. Evaluation design matters as much as the selection label.

07 | REMOVE LEAKAGE BEFORE CHASING A SCORE

ActualFinishTime is known only after the delivery. It may give an excellent score when predicting DurationMinutes, especially if departure time is also known, but it cannot support a genuine pre-departure prediction.

This is target leakage: an input gives access to outcome-related information that would not legitimately be available for a new prediction. Including DurationMinutes as its own feature is an even more direct example.

Exclude these inputs by understanding the process, even if a statistical selector ranks them first. Keep the booking reference separately for joining predictions back to records; excluding it from model inputs does not require deleting it from the application database.

08 | THREE STRATEGIES, THREE SOURCES OF EVIDENCE

Strategy How it judges features Main trade-off
Filter Uses a data-based criterion without repeatedly fitting the final predictor to compare subsets Usually efficient, but a simple per-feature test can miss interactions
Wrapper Fits and evaluates a chosen model on candidate feature subsets Directly tests model/subset combinations, but repeated fitting costs time
Embedded Uses selection or importance arising within model training Connects selection to learning, but depends on the model and its assumptions

The distinction is how selection is performed, not whether the final dataset has fewer columns. All three can produce the same subset for different reasons. None automatically proves causation, fairness or best future performance.

09 | FILTER METHODS: SCREEN THE DATA

A filter method applies a criterion to the data before choosing the final model’s inputs. Examples include removing constant features, scoring association with the target or screening strongly redundant pairs.

A numerical association score can rank how closely a feature varies with a numerical target. A constant DepotCode cannot distinguish deliveries within this training set, so a zero-variance filter can remove it.

Filters are often fast and useful when there are many columns. However, the criterion must suit the data. A linear correlation score does not capture every nonlinear relationship, and a feature weak on its own can still help in a combination.

10 | A WORKED FILTER RANKING

Eligible input Illustrative absolute correlation with duration Decision under top-two rule
DistanceKm 0.82 Keep
PlannedStops 0.64 Keep
ParcelMassKg 0.12 Exclude under this rule
VanColourCode 0.03 Exclude; arbitrary numeric category coding also needs caution

These fictional scores are supplied to explain a ranking, not calculated from the small table above. The rule keeps the two highest-scoring eligible inputs. This screening happens after excluding unavailable/leaking inputs and removing a duplicate distance representation.

An absolute correlation of 0.82 is not 82% accuracy, and it does not prove a causal effect. If a category is encoded with arbitrary numbers, ordinary numeric correlation may be misleading. Use a criterion appropriate to the feature and target types.

A different model or interaction may still benefit from parcel mass. The filter offers a candidate subset to validate, not a universal verdict that the discarded feature is worthless.

11 | A FEATURE CAN MATTER ONLY IN COMBINATION

Consider two binary switches A and B. The target is 1 when exactly one switch is on and 0 otherwise. This is the XOR pattern.

A B Target
0 0 0
0 1 1
1 0 1
1 1 0

Across these balanced examples, seeing A alone does not change the proportion of targets that are 1; the same is true for B. Yet together they determine the answer exactly.

A simple individual association screen can miss such an interaction. A model and selection procedure capable of evaluating the combination may retain both. This does not mean every wrapper discovers every interaction: a greedy search that adds only individually helpful features may stop too early.

12 | WRAPPER METHODS: TRY SUBSETS WITH A MODEL

A wrapper method evaluates feature subsets by fitting a particular model and measuring its performance using appropriate training/validation procedures. The model acts as the way to judge the subset.

For a duration predictor, a score might be mean absolute error: the average absolute difference between predicted and actual minutes. A lower value is better for this metric. Fit on training examples and measure on validation examples; training error alone is not a reliable selection target.

Because it evaluates the chosen model, a wrapper can reflect how that model uses features together. The cost is repeated fitting, and the chosen subset can change when the model, metric or data split changes.

13 | FORWARD SELECTION, ONE STEP AT A TIME

Step Candidate subset Illustrative validation MAE Decision
1 Distance only 8.0 min Best single feature
1 Stops only 10.0 min Do not start with this
1 Mass only 13.0 min Do not start with this
2 Distance + stops 5.0 min Choose this addition
2 Distance + mass 7.7 min Do not choose this addition
3 Distance + stops + mass 5.1 min Stop: no improvement over 5.0

Forward selection starts with an empty selected set, tests candidate additions and retains the best allowed improvement. In this fictional walk-through, it finishes with distance and stops.

These supplied values illustrate the logic; the activity does not fit a model. In practice, account for variation between splits and a predeclared stopping rule rather than overinterpreting a tiny score difference. Greedy forward selection is not guaranteed to find the globally best subset.

14 | BACKWARD ELIMINATION AND SEARCH COST

Backward elimination starts with the candidate inputs and considers removing features, keeping removals according to the chosen validation rule. It can reach a different subset from forward selection because the search path differs.

Testing every possible subset is an alternative, but the number grows quickly. With 10 candidate features there are 2¹⁰ = 1024 subsets including the empty set, or 1023 non-empty subsets. Each may require several model fits if cross-validation is used.

The actual search cost depends on the strategy. “Wrapper” does not mean every subset must be tested. Greedy searches reduce the workload but may miss combinations, while exhaustive searches can become impractical.

15 | TRY THE FORWARD-SELECTION WALK-THROUGH

Reveal the supplied validation results step by step. At each stage, check which feature is added and why the process stops. These are fictional precomputed scores, not live training.

16 | EMBEDDED METHODS: LEARN AND SELECT TOGETHER

An embedded method uses feature selection arising within the model-fitting process. The model learns relationships while its training procedure favours or identifies particular inputs.

One example is a linear model trained with an L1 penalty. A coefficient is a learned weight linking an input to the prediction; an L1 penalty discourages large total absolute weights and can set some coefficients exactly to zero. A zero coefficient removes that input’s contribution in that fitted linear expression.

The penalty strength affects how many inputs remain. Too strong a penalty can discard useful information. Inputs need appropriate scaling before interpreting coefficient magnitudes or applying scale-sensitive penalties. This is a conceptual example, not a requirement to derive the optimisation mathematics.

17 | TREES AND FEATURE IMPORTANCE

A decision tree learns split rules such as “distance greater than 10 km?”. Only features chosen for its splits contribute to its predictions. Tree-based models can also supply importance measures used to retain selected inputs.

This connects selection to the fitted model, but importance is not a universal property of a feature. Correlated inputs can share or compete for importance, and some tree importance measures favour features with many possible split points.

A model can output importance scores without the application actually dropping any columns. To create a reduced input pipeline, apply an explicit selection rule and validate the resulting pipeline. Importance is not proof that an attribute causes the outcome.

18 | A WORKED EMBEDDED EXAMPLE

Scaled input Illustrative L1-fitted coefficient Result under nonzero-coefficient rule
Distance 0.70 Retain
Stops 0.25 Retain
Mass 0.00 Exclude for this fitted linear predictor
Van colour indicator 0.00 Exclude for this fitted linear predictor

The coefficients are illustrative supplied results, not estimates from the earlier dataset. Under this specified rule, the retained inputs are distance and stops. The fitting process produced the zero weights.

Do not describe this as a filter merely because a coefficient threshold is applied afterwards. The evidence was learned as part of fitting the model. Conversely, a wrapper could fit many models even if each model itself has embedded selection: practical pipelines can combine methods.

19 | KEEP THE EVALUATION INDEPENDENT

Split data appropriately before learning selection rules. Fit a target-based feature ranking, wrapper search or embedded selector on training data using internal validation as needed. Keep the final test data out of those choices.

In cross-validation, selection must be refitted inside each training fold. Selecting columns once using the targets from the whole dataset before cross-validation leaks information from its held-out folds.

Apply the selected feature names, order and compatible preprocessing consistently at validation, test and prediction time. Refitting a selector independently on each new batch can change what the model’s inputs mean.

20 | CHOOSE A STRATEGY FOR THE CONSTRAINTS

Situation Possible starting point What to check
Thousands of candidate attributes; limited compute A suitable efficient filter Interactions and redundant retained features
Modest candidate set; final model already chosen A wrapper with defined search and validation rules Repeated fitting cost and validation overuse
Suitable learner supports training-based selection An embedded method Scaling, regularisation or importance biases
Expensive input measurement Compare validated subsets and acquisition cost Required accuracy, reliability and population coverage

You can combine approaches: first remove unavailable inputs and constants, use a filter to reduce a huge candidate pool, then compare a smaller set of subsets. The choice should match the prediction task, data and available resources.

No strategy excuses retaining a leaking feature. No statistically selected subset guarantees fairness. Check coverage and subgroup performance, and investigate whether apparently harmless inputs act as proxies for sensitive information.

21 | CLASSIFY THE SELECTION PROCESS

Choose the method from what the procedure actually does. The activity includes leakage and deployment cases so you can distinguish selecting a method from selecting legitimate information.

22 | SELECTION, CLEANING AND DIMENSIONALITY

Cleaning addresses problems such as incorrect units, duplicate records and missing values. Feature selection chooses a subset of the original input attributes. Transforming several original attributes into new combined components is feature extraction rather than simply selecting original columns.

Selection reduces the number of input dimensions, so it is one way to reduce dimensionality. Other reduction approaches transform the representation. The next lesson explores why high-dimensional data creates difficulties; PCA and LDA calculations are not required by this syllabus.

For a useful exam description, identify how a method makes decisions and connect that to a benefit and limit: “A wrapper compares validation performance for candidate input subsets using the chosen model. This reflects that model’s behaviour, but repeatedly fitting it increases computation.”

Further reading: filter, wrapper and embedded strategies and feature-selection examples and pipeline guidance.

MATCH THE SELECTION IDEA TO ITS ROLE

Terminology

Terminology

Feature

An input attribute supplied to a model.

Target

The outcome a supervised model learns to predict.

Feature selection

Choosing a suitable subset of original input attributes.

Informative feature

An input that contributes useful predictive information in the intended setting.

Irrelevant feature

An input that contributes no useful predictive information for the task.

Redundant feature

An input repeating information already available from other inputs.

Generalisation

Performance on genuinely new examples from the intended population.

Overfitting

Learning training-data quirks that do not transfer reliably to new examples.

Target leakage

Access to outcome-related information unavailable or illegitimate at prediction time.

Prediction moment

The point in the real process when a prediction must be made.

Filter method

Selection using a data-based criterion without repeatedly comparing fitted final-model subsets.

Variance filter

A screen removing inputs with insufficient variation under a chosen rule.

Association

A statistical relationship between variables; not by itself evidence of causation.

Correlation

A numerical measure of association, commonly linear association for Pearson correlation.

Interaction

A contribution that depends on a combination of inputs rather than each considered alone.

Wrapper method

Selection by fitting and evaluating a chosen model on candidate subsets.

Forward selection

A search that starts with no selected features and considers additions.

Backward elimination

A search that starts with candidates and considers removals.

Greedy search

A search choosing a preferred next step without guaranteeing the globally best combination.

Mean absolute error

The average absolute difference between predicted and actual numerical outcomes.

Embedded method

Selection or selection evidence arising within model training.

Coefficient

A learned weight connecting an input to a model expression.

L1 penalty

A training penalty on absolute coefficient magnitudes that can produce zero coefficients.

Regularisation

Constraints or penalties intended to control fitting complexity or improve generalisation.

Feature importance

A model- and method-dependent measure of an input’s contribution.

Cross-validation

Evaluation using repeated train/held-out folds, fitting the pipeline within each training fold.

Pipeline

The connected preprocessing, selection and modelling operations applied consistently.

Feature acquisition cost

The resources needed to obtain an input at prediction time.

Feature extraction

Constructing transformed features, potentially combining original attributes.

Dimensionality

The number of input dimensions used to represent an example.

Questions

Questions

CHECK YOUR UNDERSTANDING

Select all correct choices. Each exact set earns one point.

1. Which describe feature selection?
2. Which inputs are legitimate for pre-departure prediction?
3. Which distinguish relevance and redundancy?
4. Which are filter examples?
5. Which limit simple individual filters?
6. Which describe wrappers?
7. Which describe embedded selection?
8. Which preserve independent evaluation?
9. Which are true of MAE and subsets?
10. Which preserve deployment meaning?

EXPLAIN AND COMPARE

Write your answers before revealing each sample.

1. Describe the role of feature selection in a delivery-duration model.

2. Describe the difference between irrelevant and redundant features.

3. Describe why actual finish time is unsuitable before departure.

4. Describe a filter strategy and a limitation.

5. Describe why the XOR example challenges an individual-feature filter.

6. Describe how forward selection works in the supplied example.

7. Calculate the number of non-empty subsets of ten features and describe a consequence.

8. Describe a wrapper’s benefit and limitation.

9. Describe an embedded approach using L1 regularisation.

10. Describe why tree importance needs careful interpretation.

11. Describe how to prevent leakage during cross-validation.

12. Describe how selection differs from cleaning and feature extraction.

DESCRIBE IT | SELECT INPUTS FOR A REAL TASK

A pre-departure duration predictor has planned distance, stops, duplicate distance units, an event identifier and actual finish time. Describe legitimate inputs, a suitable selection strategy and a validation procedure. Include how you would investigate redundancy, prevent leakage and keep prediction inputs consistent.

Flashcards

Flashcards

Click to flip. Select the ideas you need to revisit.

0 cards selected for revision.

Selections are kept while this page is open.

Workbook

Workbook

COMING SOON

The workbook for A4.2.2 Feature Selection (HL) is coming soon. Complete the matching task and written questions in your exercise book.