What is Out-of-sample?

Out-of-sample describes data that was kept aside while a strategy was being built and only used afterwards to test it. Designing an investment approach usually involves a backtest, which means applying a set of rules to past market prices to see how they would have performed. The years used to design and fine-tune the rules are called in-sample. The years held back and examined only at the end are out-of-sample. It is a bit like a teacher setting exam questions the students have never seen: good answers then suggest real understanding rather than memorisation.

Say a researcher has 20 years of hypothetical price history. They use the first 15 years to design a rule and find that $10,000 would have grown at 12% a year, reaching about $54,700 (10,000 × 1.12 to the power of 15). They then run the unchanged rule on the final 5 years, which played no part in the design. There it earns 4% a year, so $10,000 becomes about $12,170 (10,000 × 1.04 to the power of 5). At the in-sample pace, the same $10,000 would have reached about $17,620 over those 5 years. The gap of roughly $5,450 suggests much of the original result came from fitting quirks of the design period rather than a lasting pattern.

Out-of-sample testing has limits. It only works if the held-back data stays untouched: once a researcher looks at those results, adjusts the rules and tests again, the out-of-sample years quietly become part of the design period. A short window can mislead in either direction, because a few years may contain only one kind of market, such as a long rise or a single sharp fall. A weaker out-of-sample result is not automatically a failure either, since some fading is common. The usual misunderstanding is to treat a good out-of-sample result as proof. It is evidence about the past, not a statement about future returns.

For a private investor building a long-term portfolio, the idea is a useful filter when reading about any backtested approach. Results shown only on the data used to create them deserve caution, because overfitting, meaning rules tailored so closely to past noise that they lose their edge on new data, is a frequent reason hypothetical returns look better than real ones. Investors who check whether a claim was tested out-of-sample, how long that period was and whether the rules stayed fixed are better placed to judge whether an approach reflects a durable idea or a lucky fit to history.

By Dr. Harry Hamann

What is the difference between in-sample and out-of-sample?
In-sample data is the period used to design and adjust a strategy. Out-of-sample data is held back and used only afterwards, with the rules unchanged, to check whether the results still hold on information the design never saw.

Why do out-of-sample results often look worse than backtests?
Designing on past data tends to pick up random patterns that do not repeat. When the rules meet fresh data, those accidental gains disappear, so returns are usually lower. A large drop suggests overfitting; a modest one is common.

Does a good out-of-sample result prove a strategy works?
No. It is stronger evidence than an in-sample backtest alone, but it still describes the past. A short or unusual test period can flatter a strategy, and results remain hypothetical.