What is Overfitting?

Overfitting happens when an investment rule is adjusted so closely to past data that it ends up describing random noise rather than a real, repeatable pattern. Think of a student who memorises last year's exam answers instead of learning the subject. The score on that exact paper is perfect, but a new paper exposes the gap. In investing, overfitting usually shows up in a backtest, which is a test of how a set of rules would have performed on historical prices. The more adjustable settings a rule has, and the more combinations someone tries, the easier it becomes to find one that looks excellent on the past purely by luck.

Say a researcher tests 1,000 slightly different versions of a trading rule on 20 years of data, and none of them has any genuine edge. Chance alone spreads the results out, so a few look outstanding. Suppose the best version shows a hypothetical 18% a year against 10% for its index. On paper, $10,000 at 18% for 20 years grows to about $273,900, because 1.18 multiplied by itself 20 times is roughly 27.4. At 10% it grows to about $67,300, because 1.10 to the 20th power is roughly 6.7. The gap looks enormous, yet the winner was picked from 1,000 attempts. On fresh data the same rule could easily end up near the index return, minus trading costs, because its earlier result reflected luck in the sample rather than skill.

A common misunderstanding is that a more detailed backtest is more trustworthy. Detail often makes overfitting worse, since every extra condition or threshold is another way to fit the past. Another trap is testing many ideas and reporting only the best one, which hides how many attempts were made. Researchers guard against this with out-of-sample testing, meaning they check a rule on data that was held back and never used to design it. They also favour rules with few settings that work across different markets and periods. None of this removes the risk entirely, because markets change and even an honestly built rule can stop working.

For a private investor, overfitting is the main reason an impressive hypothetical backtest deserves scrutiny. Strategies and products are often shown with smooth historical charts, and the smoothest are frequently the most heavily tuned. Investors who ask how many variations were tried, how many settings the rule uses and whether results hold up out of sample are better placed to tell a durable idea from a statistical accident. A long-term portfolio depends on rules that keep working after money is committed, not only on the data used to build them.

By Dr. Harry Hamann, MBA

What is overfitting in a backtest?
It is when a strategy's rules are tuned so tightly to historical prices that the backtest captures chance patterns. The hypothetical results look strong, but the patterns do not repeat, so performance on new data is usually much weaker.

How can investors tell if a strategy is overfitted?
Warning signs include many adjustable settings, very smooth hypothetical returns, results that collapse when settings change slightly, and no out-of-sample test. Rules that hold up across different markets and time periods are less likely to be overfitted.

Is overfitting the same as data mining?
They are closely linked. Data mining means searching through lots of data or rule variations for patterns. Overfitting is the usual result when the search is extensive and only the best-looking outcome is kept, without testing it on unseen data.