Looking at Alternatives? Avoid Complexity and Magical Backtests

//, Academic Research Insight/Looking at Alternatives? Avoid Complexity and Magical Backtests

Looking at Alternatives? Avoid Complexity and Magical Backtests

By |2018-09-17T15:49:19+00:00August 20th, 2018|Basilico and Johnsen, Academic Research Insight|

Quantifying Backtest Overfitting in Alternative Beta Strategies

  • Antti Suhonen, Matthias Lennkh, and Fabrice Perez
  • Journal of Portfolio Management, Winter 2017
  • A version of this paper can be found here
  • Want to read our summaries of academic finance papers? Check out our Academic Research Insight category.

What are the Research Questions?

The paper investigates the following research question:
  1. Does persistence (out of sample performance) exist for alternative beta strategies sponsored by investment banks?
  2. Does adding complexity to a strategy increase the risk of backtesting overfitting?
  3. Do the strategies capture the factor exposure they seek to exploit? And, does the exposure remain consistent between backtesting and live periods?

What are the Academic Insights?

By analyzing a proprietary dataset of around 215 individual strategy indices across 17 sponsoring investment banks with both backtest and live data (the minimum length of the backtest is 3.3 years while the average is 10.7 years; the minimum length for the live period is 0.44 years while the average is 4.6 years), the authors find the following:

  1. Most of the strategies exhibit a substantial reduction in Sharpe ratios (SR) in the live period. This reduction is due to a falling in excess returns. The haircut is at 50% or more for all asset classes and at 60% or more for all strategy groups. Only 18 of 215 strategies ( or 8.4% of the sample) had a live SR equal to or greater than the backtested one. Also, 65 strategies (or 30% of the sample) had a negative SR during the live period. (similar to the analysis in “Chasing Returns and Avoiding Spaghetti Against the Wall Fund Companies“)
  2. YES, more complex trading strategies are more liable to overfitting, and deteriorated performance. After controlling for fixed effects, the most complex strategies suffer from an SR haircut that this more than 30% compared to simpler strategies. (see here for the reasons why)
  3. The equity value strategies display a positive and significant exposure to the value factor, which becomes insignificant and negative in the live period. Differently, the equity volatility strategy, the fixed income curve strategy, and FX carry strategy show consistent exposures both in the backtest and live periods.

Why does it matter?

This study contributes to an emerging body of research on backtest overfitting in alternative beta strategies. In fact, it focuses on a very particular dataset: strategies sold by investment banks. The findings highlight the importance of due diligence on quantitative strategies (which we also highlighted in the past here) and suggest that backtested performance need to be taken with a grain of salt when selecting a strategy. The authors conclude by suggesting two further readings: Principal for Financial Benchmarks (IOSCO, 2013) and Financial Benchmarks: Thematic Review of Oversights and Control (FCA, 2015).

The Most Important Chart from the Paper:

The results are hypothetical results and are NOT an indicator of future results and do NOT represent returns that any investor actually attained. Indexes are unmanaged, do not reflect management or trading fees, and one cannot invest directly in an index.


Abstract

The authors investigate the biases in the backtested performance of “alternative beta” strategies using a unique sample of 215 trading strategies developed and promoted by global investment banks. Their results lend support to the cautions in the recent literature regarding backtest overfitting and lack of robustness in trading strategy performance during the “live” period (out of sample). The authors report a median 73% deterioration in Sharpe ratios between backtested and live performance periods for the strategies, and they establish a link between performance deterioration and strategy complexity, with the realized reduction in live versus backtested Sharpe ratios of the most complex strategies exceeding those of the simplest ones by over 30 percentage points. The robustness of strategy exposure to risk factors varies between asset classes and strategies; it appears reasonable in equity volatility and FX carry strategies but quite weak in the equity value strategy in particular.


  • The views and opinions expressed herein are those of the author and do not necessarily reflect the views of Alpha Architect, its affiliates or its employees. Our full disclosures are available here. Definitions of common statistics used in our analysis are available here (towards the bottom).
  • Join thousands of other readers and subscribe to our blog.
  • This site provides NO information on our value ETFs or our momentum ETFs. Please refer to this site.

Elisabetta Basilico, PhD, CFA
Dr. Elisabetta Basilico is a seasoned investment professional with an expertise in "turning academic insights into investment strategies." Research is her life's work and by combing her scientific grounding in quantitative investment management with a pragmatic approach to business challenges, she’s helped several institutional investors achieve stable returns from their global wealth portfolios. Her expertise spans from asset allocation to active quantitative investment strategies. Holder of the Charter Financial Analyst since 2007 and a PhD from the University of St. Gallen in Switzerland, she has experience in teaching and research at various international universities and co-author of articles published in peer-reviewed journals. She and co-author Tommi Johnsen are currently writing a book on research-backed investment ideas. You can find additional information at Academic Insights on Investing.
Yes No
This website uses cookies and third party services. Settings Ok

Cookies

We use “cookies” on this site. A cookie is a piece of data stored on a site visitor’s hard drive to help us improve your access to our site and identify repeat visitors to our site. For instance, when we use a cookie to identify you, you would not have to log in a password more than once, thereby saving time while on our site. Cookies can also enable us to track and target the interests of our users to enhance the experience on our site. Usage of a cookie is in no way linked to any personally identifiable information on our site. Some of our business partners may use cookies on our site (for example, advertisers). However, we have no access to or control over these cookies.

Embedded Content

Articles on this Site may include embedded content (e.g. videos, images, articles, etc.). Embedded content from other websites behaves in the exact same way as if the visitor has visited the other website.These websites may collect data about you, use cookies, embed additional third-party tracking, and monitor your interaction with that embedded content, including tracking your interaction with the embedded content if you have an account and are logged in to that website.