Prevent Backtest Overfitting

[slavakurilyak]

Goals

As a developer, I want to prevent backtest overfitting, so that I can find more True Positives and eliminate False Positives.

Consider

• Consider using pypbo library which calculates the Probability of Backtest Overfitting (PBO) and Deflated Sharpe Ratio (DSR)

Inspiration

Backtest overfitting is an industry-wide problem (Financial Times, 2014).

Source: The 7 Reasons Most Machine Learning Funds Fail (Presentation Slides), 2018

True Positive Technologies, lead by Marcos Lopez de Prado, developed multiple solutions to the backtest overfitting problem, including:

1. Probability of Backtest Overfitting (PBO): It computes the probability that the best in-sample strategy will underperform out-of-sample a randomly chosen alternative.
2. Deflated Sharpe Ratio (DSR): It deflates the predicted performance of a strategy by controlling for the number of trials involved in a discovery. In the example above, the “buy on Tuesday and sell on Friday” strategy would be discarded because of the large number of combinations tried.
3. Optimal Trading Rule (OTR): This empirical procedure determines the optimal trading rule on a large amount of synthetic data generated by the distribution that characterizes the observed data.
4. CPCV: It derives the full distribution of the strategy’s performance (rather than a single path) under thousands of alternative scenarios.

[slavakurilyak]

@bukosabino Let's use pypbo, the python library, to prevent backtest overfitting