Underfitting, Optimal Fitting and Overfitting
Underfitting, optimal fitting and overfitting are broad subjects that can be explored in much greater depth, particularly within statistics, machine learning and quantitative research. Readers interested in the mathematical foundations or advanced techniques can easily find extensive material through books and online resources.
For our purposes, understanding the basic ideas is sufficient because these concepts appear repeatedly whenever historical data are analysed to build trading strategies.
The Distinction
Whenever a quantitative model or trading strategy is developed using historical data, its objective is not simply to describe the past, but to capture relationships that are likely to remain valid when new data become available.
The balance between learning too little and learning too much can be described through three different situations.
Underfitting
Underfitting occurs when a model is too simple to capture the underlying structure of the data. Because the model fails to learn meaningful relationships, its performance is poor not only on unseen data but also on the historical data used during development.
Typical causes include:
- an overly simple model;
- too few explanatory variables or features;
- insufficient training or optimisation.
Although underfitting rarely produces misleadingly impressive historical results, it generally leads to inefficient models with limited predictive power.
Optimal Fitting
Optimal fitting represents the desired balance. The model captures the genuine relationships contained in the historical data while ignoring most of the random fluctuations that naturally exist in financial markets.
An optimally fitted model usually performs well both on historical data and on new, unseen observations because it has learned the underlying behaviour rather than the accidental characteristics of a specific dataset.
In practice, perfect optimal fitting rarely exists, but it remains the objective of every quantitative research process.
Overfitting
Overfitting occurs when a model becomes excessively adapted to the historical dataset used during development. Instead of learning only meaningful market behaviour, it also memorises random fluctuations, isolated events and statistical noise.
As a result, the model may appear exceptionally profitable during backtesting while performing poorly once applied to new market data.
This makes overfitting one of the most important risks in quantitative trading, since it can create the false impression that a strategy possesses predictive power when it is actually exploiting patterns that occurred only by chance.
Why Underfitting and Overfitting Matter
Both situations should be avoided, although for different reasons.
Underfitting mainly represents an efficiency problem. Since the model has not extracted enough information from the available data, its forecasts remain weak and its practical usefulness is limited.
Overfitting is considerably more dangerous.
Financial markets contain a significant amount of randomness. Given enough parameters, almost any model can be adjusted to produce excellent historical performance by exploiting coincidences rather than genuine market behaviour.
Some of the main causes include:
- excessive parameter optimisation;
- overly complex models;
- limited historical data relative to model complexity;
- attempting to explain random market noise as if it were persistent information.
The consequence is a strategy that appears robust during development but rapidly deteriorates when exposed to live market conditions.
Moving Towards Optimal Fitting
There is no technique capable of completely eliminating overfitting. Instead, quantitative research focuses on reducing its probability.
Some widely adopted practices include:
- separating development data from validation and out-of-sample datasets;
- using cross-validation techniques whenever appropriate;
- favouring simpler models unless additional complexity provides clear evidence of improvement;
- limiting the number of optimised parameters;
- verifying that performance remains stable across different market periods rather than depending on a single historical sample;
- continuously monitoring live performance to detect potential deterioration over time.
The common objective of all these techniques is to improve the model’s ability to generalize, meaning its capacity to maintain similar behaviour on data that were not used during development.
