The Virtue of Complexity in Return Prediction

In the realm of investment strategies, simplicity has long been favored. Traditional models with a limited number of parameters are prized for their interpretability and ease of use. However, recent research challenges this convention, suggesting that embracing complexity can lead to more accurate return predictions. The study “The Virtue of Complexity in Return Prediction” by Bryan T. Kelly, Semyon Malamud, and Kangying Zhou explores how complex models, particularly those utilizing machine learning techniques, can outperform simpler counterparts in forecasting equity market returns.

The Virtue of Complexity in Return Prediction

• Bryan T. Kelly, Semyon Malamud, and Kangying Zhou
• The Journal of Finance, 2024
• A version of this paper can be found here
• Want to read our summaries of academic finance papers? Check out our Academic Research Insight category

Key Academic Insights

Complex Models Outperform Simpler Ones: The study demonstrates that models with a higher number of parameters can capture intricate patterns in financial data, leading to improved return predictions. This finding is contrary to the traditional belief that simpler models are more effective due to their lower risk of overfitting. Relatedly, recent work on refining value investing—for instance, by incorporating intangible assets—highlights how complexity can make even classic strategies smarter and more adaptive.

Machine Learning Enhances Predictive Power: By employing machine learning algorithms, the researchers show that complex models can adapt to the nuances of financial markets, adjusting to new information and changing conditions more effectively than static, simple models. This echoes developments in momentum investing, where attention to signal clarity (not just past returns) has helped improve trend-following approaches.

Improved Portfolio Performance: The enhanced predictive capabilities of complex models translate into better-informed investment decisions, potentially leading to higher Sharpe ratios and overall portfolio performance

Practical Applications for Investment Advisors

Incorporate Advanced Modeling Techniques: Advisors should consider integrating machine learning models into their investment analysis processes to leverage the predictive advantages these models offer.

Balance Complexity with Interpretability: While complex models provide improved predictions, it’s essential to maintain a balance to ensure that the models remain interpretable and actionable for both advisors and clients.

Continuous Model Evaluation: Regularly assess and update models to ensure they adapt to evolving market conditions, maintaining their predictive accuracy over time.

How to Explain This to Clients

“Traditionally, we’ve used straightforward models to predict market returns, valuing their simplicity and clarity. However, recent research indicates that more complex models, especially those using advanced techniques like machine learning, can better capture the complexities of the market. By embracing these sophisticated tools, we aim to enhance our investment strategies, potentially leading to improved returns for your portfolio.”

The Most Important Chart from the Paper

This table shows how using more complex models can improve our ability to predict stock returns — but only up to a point. At first, as models become more detailed (adding more “moving parts”), their predictions get better. But if they become too complex without proper controls, they start to “overthink” the data — picking up noise instead of useful patterns. That’s called overfitting.

The results are hypothetical results and are NOT an indicator of future results and do NOT represent returns that any investor actually attained. Indexes are unmanaged and do not reflect management or trading fees, and one cannot invest directly in an index.

Abstract

Much of the extant literature predicts market returns with “simple” models that use only a few parameters. Contrary to conventional wisdom, we theoretically prove that simple models severely understate return predictability compared to “complex” models in which the number of parameters exceeds the number of observations. We empirically document the virtue of complexity in U.S. equity market return prediction. Our findings establish the rationale for modeling expected returns through machine learning.