By |Published On: August 20th, 2018|Categories: Basilico and Johnsen, Academic Research Insight|

Quantifying Backtest Overfitting in Alternative Beta Strategies

  • Antti Suhonen, Matthias Lennkh, and Fabrice Perez
  • Journal of Portfolio Management, Winter 2017
  • A version of this paper can be found here
  • Want to read our summaries of academic finance papers? Check out our Academic Research Insight category.

What are the Research Questions?

The paper investigates the following research question:
  1. Does persistence (out of sample performance) exist for alternative beta strategies sponsored by investment banks?
  2. Does adding complexity to a strategy increase the risk of backtesting overfitting?
  3. Do the strategies capture the factor exposure they seek to exploit? And, does the exposure remain consistent between backtesting and live periods?

What are the Academic Insights?

By analyzing a proprietary dataset of around 215 individual strategy indices across 17 sponsoring investment banks with both backtest and live data (the minimum length of the backtest is 3.3 years while the average is 10.7 years; the minimum length for the live period is 0.44 years while the average is 4.6 years), the authors find the following:

  1. Most of the strategies exhibit a substantial reduction in Sharpe ratios (SR) in the live period. This reduction is due to a falling in excess returns. The haircut is at 50% or more for all asset classes and at 60% or more for all strategy groups. Only 18 of 215 strategies ( or 8.4% of the sample) had a live SR equal to or greater than the backtested one. Also, 65 strategies (or 30% of the sample) had a negative SR during the live period. (similar to the analysis in “Chasing Returns and Avoiding Spaghetti Against the Wall Fund Companies“)
  2. YES, more complex trading strategies are more liable to overfitting, and deteriorated performance. After controlling for fixed effects, the most complex strategies suffer from an SR haircut that this more than 30% compared to simpler strategies. (see here for the reasons why)
  3. The equity value strategies display a positive and significant exposure to the value factor, which becomes insignificant and negative in the live period. Differently, the equity volatility strategy, the fixed income curve strategy, and FX carry strategy show consistent exposures both in the backtest and live periods.

Why does it matter?

This study contributes to an emerging body of research on backtest overfitting in alternative beta strategies. In fact, it focuses on a very particular dataset: strategies sold by investment banks. The findings highlight the importance of due diligence on quantitative strategies (which we also highlighted in the past here) and suggest that backtested performance need to be taken with a grain of salt when selecting a strategy. The authors conclude by suggesting two further readings: Principal for Financial Benchmarks (IOSCO, 2013) and Financial Benchmarks: Thematic Review of Oversights and Control (FCA, 2015).

The Most Important Chart from the Paper:

 

The results are hypothetical results and are NOT an indicator of future results and do NOT represent returns that any investor actually attained. Indexes are unmanaged, do not reflect management or trading fees, and one cannot invest directly in an index.


Abstract

The authors investigate the biases in the backtested performance of “alternative beta” strategies using a unique sample of 215 trading strategies developed and promoted by global investment banks. Their results lend support to the cautions in the recent literature regarding backtest overfitting and lack of robustness in trading strategy performance during the “live” period (out of sample). The authors report a median 73% deterioration in Sharpe ratios between backtested and live performance periods for the strategies, and they establish a link between performance deterioration and strategy complexity, with the realized reduction in live versus backtested Sharpe ratios of the most complex strategies exceeding those of the simplest ones by over 30 percentage points. The robustness of strategy exposure to risk factors varies between asset classes and strategies; it appears reasonable in equity volatility and FX carry strategies but quite weak in the equity value strategy in particular.

About the Author: Wesley Gray, PhD

Wesley Gray, PhD
After serving as a Captain in the United States Marine Corps, Dr. Gray earned an MBA and a PhD in finance from the University of Chicago where he studied under Nobel Prize Winner Eugene Fama. Next, Wes took an academic job in his wife’s hometown of Philadelphia and worked as a finance professor at Drexel University. Dr. Gray’s interest in bridging the research gap between academia and industry led him to found Alpha Architect, an asset management firm dedicated to an impact mission of empowering investors through education. He is a contributor to multiple industry publications and regularly speaks to professional investor groups across the country. Wes has published multiple academic papers and four books, including Embedded (Naval Institute Press, 2009), Quantitative Value (Wiley, 2012), DIY Financial Advisor (Wiley, 2015), and Quantitative Momentum (Wiley, 2016). Dr. Gray currently resides in Palmas Del Mar Puerto Rico with his wife and three children. He recently finished the Leadville 100 ultramarathon race and promises to make better life decisions in the future.