Executives vs. Chatbots in Earnings Conference Q&A

The article introduces a novel approach using Large Language Models (LLMs) to measure the information content in earnings conference call Q&A sessions.

Executives vs. Chatbots: Unmasking Insights through Human-AI Differences in Earnings Conference Q&A

• Baj, Boyson, Cao, Liu and Wan
• Working paper
• A version of this paper can be found here
• Want to read our summaries of academic finance papers? Check out our Academic Research Insight category

What are the Research Questions?

This paper acknowledges the pivotal role of earnings calls in disseminating value-relevant information, with particular emphasis on the Q&A segment. However, it confronts the inherent challenge posed by the unstructured nature of language in these calls, complicating quantitative analysis. In response, the authors innovate by introducing a novel measure designed to grasp the subtleties of information embedded in the dialogue. This measure, the Human AI Difference (HAID) is based on discerning semantic differences between responses to identical questions from corporate CEOs and CFOs, and those generated by advanced Large Language Models (LLMs), such as ChatGPT.

The authors ask the following questions:

1. Are high HAID values more likely to convey positive news rather than negative news?
2. Are high HAID values positively associated with earnings growth?
3. Are high HAID values positively associated with abnormal future stock returns and abnormal trading volume?
4. Does higher HAID lead to more accurate and consistent analyst forecasts?
5. Is HAID negatively related to stock illiquidity measures?

What are the Academic Insights?

By studying a sample of 104,932 earnings conference calls for the period of 2004 – 2020, the authors find:

1. YES – higher values of HAID are more likely to convey positive news rather than negative news, aligning with managers’ incentives to emphasize positive developments and conceal adverse information
2. YES – higher values of HAID are indeed correlated with future earnings growth
3. YES – higher HAID is associated with higher absolute cumulative abnormal stock returns and abnormal trading volume around earnings calls, indicating that investors respond more actively to calls with higher HAID
4. YES – higher HAID is associated with smaller analyst forecast errors and less cross-analyst disagreement after earnings calls
5. YES – HAID is negatively related to stock illiquidity measures, such as bid-ask spread and Amihud Ratio, indicating that higher HAID is linked to improved liquidity. This implies that HAID contributes to better price discovery and reduced information asymmetry

The authors conduct robustness tests by applying the methodology to alternative LLMs, Google Bard, and an open-source LLM, and find qualitatively similar results, reinforcing the generalizability of the findings.

Why does this study matter?

This study contributes to our understanding of the role of language and information in financial markets. In fact, it demonstrates that the HAID measure effectively quantifies the information content in earnings conference calls, and higher HAID values are associated with more positive market outcomes, improved analyst behavior, and enhanced liquidity.

The Most Important Chart from the Paper:

The results are hypothetical results and are NOT an indicator of future results and do NOT represent returns that any investor actually attained.  Indexes are unmanaged and do not reflect management or trading fees, and one cannot invest directly in an index.

Abstract

A significant portion of information shared in earnings calls is conveyed through verbal communication by corporate managers. However, quantifying the extent of new information provided by managers poses challenges due to the unstructured nature of human language and the difficulty in gauging the market’s existing knowledge. In this study, we introduce a novel measure of information content (Human-AI Differences, HAID) by exploiting the discrepancy between answers to questions at earnings calls provided by corporate executives and those given by several context-preserving Large Language Models (LLM) such as ChatGPT, Google Bard, and an open source LLM. HAID strongly predicts stock liquidity, abnormal returns, number of analysts’ forecast revisions, analyst forecast accuracy following these calls, and propensity of managers to provide management guidance, consistent with HAID capturing new information conveyed by managers. Overall, our results highlight the importance of using LLM as a tool to help investors unveil the veiled – penetrating the information layers and unearthing hidden insights.