Evolution and the Overconfidence Phenomenon

Many have questioned why overconfidence might exist in human populations when it can lead to suboptimal outcomes such as wars or, say, financial collapse.  Common sense suggests that an accurate assessment of one’s capabilities would lead to consistently better outcomes on an individual level, since you wouldn’t be doing things like making highly leveraged commodity futures bets, or getting into inadvisable bar fights.  So why is the phenomenon of overconfidence so well documented in studies of human behavior?  How did this trait persist in humans, if evolution favors individuals who are realistic about their skills and abilities?

We at Turnkey Analyst are closet evolutionary biologists, so we were fascinated to see some recent research in the field of evolution as it relates to overconfidence.  Earlier this month, researchers Johnson and Fowler published the results of a study in the journal Nature showing that overconfidence can make evolutionary sense under conditions when an opponent’s strength is unknown and when the benefits of success exceed the costs incurred.  They used a clever computer model and game theory to demonstrate why.

The model features two individuals who are fighting over resources.  The game conditions are as follows.  If they fight, there are costs to both, but the winner takes the resource.  If only one claims the resource, he gets it, cost-free.  If neither claims the resource, neither gets it.  Additionally, each individual may overestimate or underestimate his opponent’s strength.  Finally, there is an error rate associated with estimating an opponent’s strength.  Thus, each actor has to make assess his own perceived strength versus his opponent’s, with some uncertainty, and judge whether to fight or not.

The researchers ran a multi-generational simulation of populations of individuals possessing different degrees of overconfidence, and different error rates for estimating opponents.  The idea was to model evolution, identifying which traits would survive and be passed to successive generations.  The conclusion was that overconfidence is the best strategy when rewards exceed costs incurred in fighting for them, and when the relative strength of opponents is unclear.  So here we have what seems to be a reasonable explanation for why overconfidence might be an evolutionary advantage.

The literature of psychology and behavioral economics is replete with examples of our innate overconfidence.  In a famous example, Svenson (1981) asked a group of students whether they were safe drivers; 90% believed they were above average, and 82% ranked themselves in the top 30% of all drivers.  Other studies have shown that even drivers who have been in car accidents continue to rate themselves as above average drivers.

Kruger and Dunning (1999) gave an LSAT to a group of undergraduates, and then asked them to rank themselves versus their peers.  The group rated themselves on average as being in the 66%ile, versus the actual mean of 50%.  It turns out the bottom quartile, with an actual average score of 12%ile, felt they were way above average, believing they were in the 68%ile, and were guilty of the highest degree of overconfidence.  So it was the least competent students who were least able to recognize their own incompetence.

This is amazing to me.  Here you have people completely unaware of their ineptitude, and the more inept they are, the more overconfident they are.  Psychologists refer to this ability (or inability) to recognize one’s own mistakes as a “metagcognitive” skill.

There are plenty of studies of overconfidence in financial markets, but I just picked out a simple one to demonstrate a simple truth.  In a study by Yeoh and Wood (2011), students took part in an eight week market simulation.  Before the game, 63% of students thought they would achieve investment returns in excess of 15% over the eight week study.  During the exercise, only 3% actually succeeded in beating this return.  Okay, so maybe you don’t think you can earn 15% in eight weeks (or maybe you do?), but the principle of overestimation of stock picking ability by groups is clear.  This kind of overconfidence in one’s own investing prowess is widely replicated in numerous studies.

Now making good stock picks is inherently difficult.  I don’t have any studies to support this at my fingertips, but I’m going to guess that the vast majority of people think they can do it well and beat the averages, and by a lot.  After all, there is something in us that wants to be good at picking stocks.  It shows we’re smarter, more energetic, and have better skills and judgment.  This is the stuff of overconfidence.

If you are a non-professional, as some readers of this site are, and are trying to manage your own portfolio, from a practical standpoint you face some daunting obstacles.  Take trying to pick undervalued, liquid large capitalization stocks, for example.  Overall, the market prices these assets pretty efficiently, as there are armies of analysts, money managers, and other investment pros all constantly evaluating them.

Ask many hedge fund managers and they will tell you about their edge, some differentiating factor that allows them to consistently beat the markets.  So what is your edge?  Superior analysis?  Superior intuition or insight, which renders you better able to make sense of a huge volume of information and distill it down to its essence?  Raw smarts?  A temperament better suited for investing?  I could go on, since there are so many ways you can be better than the next guy.

Sure, maybe you can compete on some level, but it’s going to be a huge amount of work, and then consider trying to compete across a portfolio.  Even pros who study stocks all day long every day can only know, what, one or perhaps two dozen equities very well?  If you want reasonable diversification how can you hope to stay on top of so many companies, and hold down your day job?

Also, how do you measure how well you’re doing?  If you’re like many, you may move money in and out of your investment account, and because retail investors don’t generally pay for audits, you’re not going to have an accurate sense for how you stack up.  But that’s okay, since you just know you’re better, right?

Furthermore, if you’re not managing your own portfolio, how do you pick a hedge fund or mutual fund manager?  Are you seeing something that everyone else is missing?  Is the fact that they have outperformed for some period of time dispositive?  Studies suggest not.  Or are you just really good at evaluating who the best investors are?

It takes courage to say you may not be as good as the next guy.  To say that you know that you don’t know.  If we are genetically programmed to overestimate our abilities because these traits have been naturally selected in us, it takes significant cognitive effort to realistically appraise ourselves.  In fact, the phrase “Know thyself” was inscribed on the Temple of Apollo at Delphi, maybe at least partly as an exhortation to be honest with yourself and realistic about your abilities.  Perhaps even in ancient Greece, they had begun thinking about metacognition.

It is indisputable that many people would be better off investing using quantitative techniques.  It may be that you would be better off, and I encourage you to consider it.  There are many ways to leverage off the rich body of academic literature, and use strategies that have been shown to generate excess returns over long periods of time.  Read the papers.  Think about what would work for you.  Then use our site, and let it do the brute force work for you.

You might choose a strategy, and stick with it for the next ten years, letting the screens do all the work for you.  Or you might use quantitative techniques to enhance your own investment process.  Use the Company Reports analysis to see how a firm ranks within a strategy, or you can identify specific areas, such as accruals or earnings manipulation, for additional research.  Or use a screen to narrow the market universe to a manageable list of stocks, and go from there.  Even if you’re wrong sometimes, and you don’t pick the best of the group, the quant index may save you, since on average the deck will be stacked in your favor and you’re going to outperform.

As for me, I’m a believer.  Despite an expensive postsecondary education, and Wall Street experience, I’m not convinced I have the skills and time to beat the fantastic alpha generated in the academic studies you’ll find on this site.  So join me on the quant bandwagon.  There’s plenty of room!

And if you don’t believe me, take it from John Stuart Mill:

“Unfortunately for the good sense of mankind, the fact of their fallibility is far from carrying the weight in their practical judgment which is always allowed to it in theory; for while everyone well knows himself to be fallible, few think it necessary to take any precautions against their own fallibility, or admit the supposition that any opinion, of which they feel very certain may be one of the examples of error to which they acknowledge themselves to be liable.”
– John Stuart Mill