Picture a magician at a party. He asks you to think of a card, makes a few hand gestures and suddenly guesses it. Your jaw drops. But here is a detail: that magician has been trying the same trick all night with a hundred different guests. With almost all of them he failed, and nobody noticed. With you he got it right once, and that single time is the one you remember. Is he a magic genius or did he just get lucky with you?
The exact same thing happens in quantitative trading, and it has a name: overfitting. The strategy that looks best in testing may really be the lucky magician who got it right once. PBO exists to measure how much you should distrust that champion.
What overfitting means
Overfitting is memorizing the past instead of understanding the market. It is like a student who memorizes the exact answers to last year's exam: he nails those specific questions but fails the moment you change the wording. An overfitted strategy knows the historical data it was built on perfectly, but collapses the moment the market shows it something new โ which is exactly what happens in real life.
The problem is that an overfitted strategy looks gorgeous in the backtest. Smooth equity curve, few losses, all perfect. That beauty is the trap. The more perfect a backtest looks, the more suspicious it should make you.
What PBO measures
PBO stands for Probability of Backtest Overfitting. It answers a very specific and very uncomfortable question: the strategy that came out best in testing, what is the chance it is actually worse than average when you use it for real?
It reads as a percentage and the intuition is direct:
- PBO close to 0%: excellent. Your champion also wins outside the tests. There are reasons to trust it.
- PBO close to 50%: red alarm. It is like flipping a coin. Your champion is almost as likely to be good as to be a fraud. You were fooled.
Put another way: PBO measures the chance that you were the guest the magician happened to guess correctly. A high PBO does not say the strategy is definitely bad; it says you have no way to tell it apart from pure luck, and in finance that is reason enough to discard it.
The idea behind it: shuffling the exam (CPCV)
How do you calculate something like this without a crystal ball? With a technique that, in plain words, means cutting your data into many pieces and shuffling which ones you use to choose and which to check. The technical name is CPCV (combinatorial purged cross-validation), but the idea is homely.
Imagine that instead of a single exam, you give your strategy many different exams, each with different questions. In each exam you pick the best strategy with one piece of data and then test it on another piece it never saw. You repeat this many, many times, shuffling the combinations. If your champion keeps being good when it faces new data again and again, great. If it only shone on the specific piece where you picked it, the PBO shoots up and exposes it.
The word purged simply means care is taken so the test piece is not contaminated with information from the training piece. It is like making sure the student cannot see the answers before the exam.
How AlphaLab handles this
AlphaLab, the quantitative lab that runs on your own PC, treats PBO as a gatekeeper. Good backtest numbers are not enough for it; it wants to know whether those numbers hold up when the exams are shuffled.
That is why AlphaLab puts every candidate through this kind of cross-validation and computes its PBO. A strategy with a high PBO โ near that 50% that equals flipping a coin โ is discarded, no matter how pretty its curve was. There are no cosmetic exceptions. It is better to reject a good strategy than to accept a fraud, because the fraud is the one that makes you lose real money.
And, true to its philosophy, AlphaLab also publishes the strategies that fail the PBO. Failure is information, not something to hide. The code is auditable and your data never leaves your computer.
Key takeaways
- The strategy that looks best in testing may be the magician who got it right once by luck.
- PBO measures the probability that your champion is actually worse than average in real life.
- PBO near 0% is a good sign; near 50% means it is almost a coin flip โ discard it.
- It is computed by shuffling many exams with data the strategy never saw (CPCV).
- AlphaLab uses PBO as a filter and rejects suspicious champions, even with a beautiful curve.
- Passing the PBO does not guarantee profits: trading always carries risk of loss.
If you want to check the PBO of your own strategies before risking money, you can explore AlphaLab with a 14-day free trial. A card is needed to activate it, but nothing is charged if you cancel before day 14. We do not promise you a champion; we give you a way to know whether your champion is real.