The article introduces a novel approach using Large Language Models (LLMs) to measure the information content in earnings conference call Q&A sessions.

Executives vs. Chatbots: Unmasking Insights through Human-AI Differences in Earnings Conference Q&A

  • Baj, Boyson, Cao, Liu and Wan
  • Working paper
  • A version of this paper can be found here
  • Want to read our summaries of academic finance papers? Check out our Academic Research Insight category

What are the Research Questions?

This paper acknowledges the pivotal role of earnings calls in disseminating value-relevant information, with particular emphasis on the Q&A segment. However, it confronts the inherent challenge posed by the unstructured nature of language in these calls, complicating quantitative analysis. In response, the authors innovate by introducing a novel measure designed to grasp the subtleties of information embedded in the dialogue. This measure, the Human AI Difference (HAID) is based on discerning semantic differences between responses to identical questions from corporate CEOs and CFOs, and those generated by advanced Large Language Models (LLMs), such as ChatGPT.

The authors ask the following questions:

  1. Are high HAID values more likely to convey positive news rather than negative news?
  2. Are high HAID values positively associated with earnings growth?
  3. Are high HAID values positively associated with abnormal future stock returns and abnormal trading volume?
  4. Does higher HAID lead to more accurate and consistent analyst forecasts?
  5. Is HAID negatively related to stock illiquidity measures?

What are the Academic Insights?

By studying a sample of 104,932 earnings conference calls for the period of 2004 – 2020, the authors find:

  1. YES – higher values of HAID are more likely to convey positive news rather than negative news, aligning with managers’ incentives to emphasize positive developments and conceal adverse information
  2. YES – higher values of HAID are indeed correlated with future earnings growth
  3. YES – higher HAID is associated with higher absolute cumulative abnormal stock returns and abnormal trading volume around earnings calls, indicating that investors respond more actively to calls with higher HAID
  4. YES – higher HAID is associated with smaller analyst forecast errors and less cross-analyst disagreement after earnings calls
  5. YES – HAID is negatively related to stock illiquidity measures, such as bid-ask spread and Amihud Ratio, indicating that higher HAID is linked to improved liquidity. This implies that HAID contributes to better price discovery and reduced information asymmetry

The authors conduct robustness tests by applying the methodology to alternative LLMs, Google Bard, and an open-source LLM, and find qualitatively similar results, reinforcing the generalizability of the findings.

Why does this study matter?

This study contributes to our understanding of the role of language and information in financial markets. In fact, it demonstrates that the HAID measure effectively quantifies the information content in earnings conference calls, and higher HAID values are associated with more positive market outcomes, improved analyst behavior, and enhanced liquidity.

The Most Important Chart from the Paper:

The results are hypothetical results and are NOT an indicator of future results and do NOT represent returns that any investor actually attained.  Indexes are unmanaged and do not reflect management or trading fees, and one cannot invest directly in an index.


A significant portion of information shared in earnings calls is conveyed through verbal communication by corporate managers. However, quantifying the extent of new information provided by managers poses challenges due to the unstructured nature of human language and the difficulty in gauging the market’s existing knowledge. In this study, we introduce a novel measure of information content (Human-AI Differences, HAID) by exploiting the discrepancy between answers to questions at earnings calls provided by corporate executives and those given by several context-preserving Large Language Models (LLM) such as ChatGPT, Google Bard, and an open source LLM. HAID strongly predicts stock liquidity, abnormal returns, number of analysts’ forecast revisions, analyst forecast accuracy following these calls, and propensity of managers to provide management guidance, consistent with HAID capturing new information conveyed by managers. Overall, our results highlight the importance of using LLM as a tool to help investors unveil the veiled – penetrating the information layers and unearthing hidden insights.

Print Friendly, PDF & Email
Elisabetta Basilico, PhD, CFA
Dr. Elisabetta Basilico is a seasoned investment professional with an expertise in "turning academic insights into investment strategies." Research is her life's work and by combing her scientific grounding in quantitative investment management with a pragmatic approach to business challenges, she’s helped several institutional investors achieve stable returns from their global wealth portfolios. Her expertise spans from asset allocation to active quantitative investment strategies. Holder of the Charter Financial Analyst since 2007 and a PhD from the University of St. Gallen in Switzerland, she has experience in teaching and research at various international universities and co-author of articles published in peer-reviewed journals. She and co-author Tommi Johnsen published a book on research-backed investment ideas, titled Smarte(er) Investing. How Academic Insights Propel the Savvy Investor. You can find additional information at Academic Insights on Investing.

Important Disclosures

For informational and educational purposes only and should not be construed as specific investment, accounting, legal, or tax advice. Certain information is deemed to be reliable, but its accuracy and completeness cannot be guaranteed. Third party information may become outdated or otherwise superseded without notice.  Neither the Securities and Exchange Commission (SEC) nor any other federal or state agency has approved, determined the accuracy, or confirmed the adequacy of this article.

The views and opinions expressed herein are those of the author and do not necessarily reflect the views of Alpha Architect, its affiliates or its employees. Our full disclosures are available here. Definitions of common statistics used in our analysis are available here (towards the bottom).

Join thousands of other readers and subscribe to our blog.

Print Friendly, PDF & Email