Models vs. Experts #7: Job Analysis

A comparison of holistic and decomposed judgment strategies in job analyses by job incumbents synthesis

• Cornelius, E. T., & Lyness, K. S.
• Journal of Applied Psychology, 65, 155-163
• An online version of the paper cannot be found
• Want a summary of academic papers with alpha? Check out our free Academic Alpha Database!

Abstract:

The purpose of this study was to compare three alternative judgment strategies used to aid job incumbents in making overall evaluations of worker requirements and motivational characteristics of their own jobs. One hundred fifteen job incumbents across several organizations evaluated their jobs using 13 job analysis scales on two occasions. The results indicated that a “decomposed” judgment strategy, using a mechanical algorithm to determine the overall evaluations, resulted in improved data from incumbents as measured by interrater agreement coefficients. Simple overall clinical, or holistic, judgments proved as effective as the fully decomposed judgments when assessing intrarater reliability across time and when comparing incumbent ratings to job analyst and job supervisor ratings. In all instances, a judgment strategy that involved decomposed estimates followed by holistic overall evaluations proved to be the least effective approach to developing summary evaluations by job incumbents. Other results indicate that the education level of the incumbent is directly related to success on the rating task, whereas length of job experience is not.

Prediction:

What is job analysis? There is a great primer at the following link: http://www.ou.edu/faculty/M/Jorge.L.Mendoza-1/job-analysis-criteria-reliability-validity.pdf . In a nutshell, job analysis helps human resources folks and managers determine the job description (what you gotta do), job specifications (the skills needed to accomplish the job), and performance standards (how to identify if the job is getting done). As one can imagine, job analysis is not a trivial task and theoretically involves some high-level thinking.

Here is how the tests go down:

1. Identify a variety of jobs with a large amount of workers (janitor, production worker, bank teller, etc.). These workers are called ‘incumbents’
2. Holistic job analysis by incumbents involves them rating their job along 13 dimensions based on a given list of task statements (human, low complexity)
3. Decomposed-clinical involves the workers rating each task on 13 scales and then rating the overall job on the same 13 scales (human, high complexity)
4. Decomposed-algorithm involves the workers rating each task on 13 scales and then a computer algorithm takes that information and generates a rating for the job as a whole (quant)
5. The authors then look at how these three sets of analysis compare over 2 job analysis sessions separated by 3-8 weeks. The tests look at consistency and reliability across time.

Alpha Highlight:

Here are the results:

• Quant assessment had an 82% correlation with original
• Low complex assessment had an 80% correlation with original
• High complex assessment had an 60% correlation with original

==> As it gets more complicated, computers win. If the inputs for decision making are relatively simple, humans and computers perform about the same.

Here are some more results. Humans aren’t consistent judgers, even when analyzing the same data. As complexity in the decision making process increases, consistency goes down further.

Buffett’s 40′ foot view can work; computer algorithms can work; but deep dive security analysis probably won’t work.

Thoughts on the paper?