Models vs. Experts #12: Modeled Human Judgement beats Human Judgement. Huh?

/Models vs. Experts #12: Modeled Human Judgement beats Human Judgement. Huh?

Models vs. Experts #12: Modeled Human Judgement beats Human Judgement. Huh?

By | 2017-08-18T17:00:43+00:00 September 13th, 2013|Behavioral Finance|1 Comment
Print Friendly, PDF & Email
(Last Updated On: August 18, 2017)

Man versus Model of man: A rational, plus some evidence, for a method of improving on clinical inferences

  • Goldberg, L. R.
  • Psychological Bulletin, 73, 422-432
  • An online version of the paper can be found here
  • Want a summary of academic papers with alpha? Check out our free Academic Alpha Database!

Abstract:

Clinical psychologists, physicians, and other professionals are typically called upon to combine cues to arrive at some diagnostic or prognostic decision. Mathematical representations of such clinical judges can often be constructed to capture critical aspects of their judgmental strategies. An analysis of the characteristics of such models permits a specification of the conditions under which the model itself will be a more valid predictor than will the man from whom it was derived. To ascertain whether such conditions are met in natural clinical decision making, data were reanalyzed from a study of the judgments of 29 clinical psychologists attempting to differentiate psychotic from neurotic patients on the basis of their MMPI profiles. The results of these analyses indicate that for this diagnostic task models of the men are generally more valid than the men themselves. Moreover, the finding occurred even when the models were constructed on a small set of cases, and then man and model competed on a completely new set.

Prediction:

The author engaged 29 clinical psychologists of varying experience and training to make a diagnostic decision on the basis of each MMPI profile. The validity of each judge and his model was compared.

To compare the validity, here is how the test works (my layman interpretation)

  • Use a judge’s previous diagnosis to construct a model for the judge.
  • Table 3 highlights different time periods to calibrate the quantitative version of the judge’s judgement. For example, “(2)” means half of the data points were used to build the quant model.)

Here are the results from the test. See table 3.

  • When model were constructed on one-half of the cases, in 86% the model was more accurate than was the clinicians.

table3

The author next tests a sample where humans have the highest judgement capability. The author then creates a model of each judge’s judgement and uses the models prediction for each human and compares it to the actual human. Amazingly, the modeled version of the human’s judgement actually outperforms the human’s judgement!!!

table4

This study implies that we can look at the smartest stock-pickers, build a model for how they pick stocks (e.g., deep value stocks under 500mm with no debt), and that model–out of sample–will pick stocks better than the expert stock-picker will pick stocks.

Why?

Good stock-pickers identify what works, but they have a problem repeating the process that made them successful in the first place–boredom, fatigue, illness, etc. add noise to the original “good-idea” generating process. Fortunately, a computer-modeled version of a process don’t suffer from human conditions.

Wild.


  • The views and opinions expressed herein are those of the author and do not necessarily reflect the views of Alpha Architect, its affiliates or its employees. Our full disclosures are available here. Definitions of common statistics used in our analysis are available here (towards the bottom).
  • Join thousands of other readers and subscribe to our blog.
  • This site provides NO information on our value ETFs or our momentum ETFs. Please refer to this site.

About the Author:

After serving as a Captain in the United States Marine Corps, Dr. Gray earned a PhD, and worked as a finance professor at Drexel University. Dr. Gray’s interest in bridging the research gap between academia and industry led him to found Alpha Architect, an asset management that delivers affordable active exposures for tax-sensitive investors. Dr. Gray has published four books and a number of academic articles. Wes is a regular contributor to multiple industry outlets, to include the following: Wall Street Journal, Forbes, ETF.com, and the CFA Institute. Dr. Gray earned an MBA and a PhD in finance from the University of Chicago and graduated magna cum laude with a BS from The Wharton School of the University of Pennsylvania.