Do We Use General Cognitive Ability Assessments Intelligently? 🧑‍🏫

Why I’m distrustful of the second-best predictor of candidate performance

One of the best predictors of candidate job performance is something called General Cognitive Ability (GCA).

It’s so good that GCA assessments are second only to Work Sample Tests when predicting candidate performance. Frank Schmidt, a pillar of industrial-organizational psychology, absolutely LOVES them.¹

But I’m a bit skeptical, not of cognitive ability itself (after all, who doesn’t want to work with smart people?) but of how most people, and more specifically, hiring managers, tend to take common proxies of cognitive ability at face value. I’m also disturbed that many measures of cognitive ability are usually the first filter that recruiting teams use to reject candidates, even for jobs where, let’s face it, the employee doesn’t have to be that intelligent.

In this post, I’ll define General Cognitive Ability and outline some of the risks associated with many of the ways we measure it.

What is General Cognitive Ability, and why should I care about it?

In general, we tend to think of GCA as a measure of someone’s intelligence, and that’s mostly correct. GCA assessments have clear right and wrong answers, which reduces subjectivity and allows scale evaluations. 

The SAT and ACT college entrance exams are two examples of general cognitive assessments. And while many employers are happy using the SAT/ACT or G.P.A. as a proxy for intelligence, there is also a cottage industry of career-specific assessment companies that also claim to accurately measure a candidate’s intelligence and their ability to learn.²

The appeal of GCA assessments and signals is their efficiency. As hiring methods go they offer the biggest bang for your buck. Additionally, they are less time-consuming to take and administer than Work Sample Tests. And, also unlike Work Sample Test, they can be used to measure candidates ranging from entry-level to executive-level roles.³ 

GCA Assessments sound great, right? Well, before you run off and ask all your candidates to retake the SAT, let’s look at some of the downsides.

The case against General Cognitive Ability assessments and their proxies

There are several risks associated with evaluating candidates based upon general cognitive ability assessments and their proxies, things like G.P.A, SAT/ACT scores, or IQ tests. In no specific order, they are:

Gary Larson’s The Far Side nails it (again).

Common measures of General Cognitive Ability have proven bias. (Oops)

Researchers have proved those common GCA assessments, like the SAT and ACT, underpredict the performance of non-male, non-white groups. That failure is why the SAT has been revamped several times and why some colleges and university’s no longer require it.

There’s also a growing awareness that money and time can “buy” better scores on GCA assessments. If you have the time to prep and can afford a test coach, you’re in much better shape than somewhat who doesn’t have the time to learn and practice the strategies that result in a higher score.

Courtesy of North by Northwestern. Note most test prep companies claim a test score bump of 150-350 points.

G.P.A is also used as a proxy for cognitive ability, and sometimes unfairly. Say you’re evaluating two candidates for an entry-level role. One of them worked 60 hours a week to pay for their mid-tier college, and they have a mediocre G.P.A. The other attended a higher tier school but didn’t have to work to support themselves and has an impressive G.P.A. Do you believe that G.P.A is a fair measure of each candidate’s cognitive ability? I’m suspect.

Standard measures of GCA have not kept pace with job requirements.

There are two types of GCA assessments: Tests of Achievement, which measure how much someone has already learned; and Tests of Ability which measure how well someone solves unfamiliar problems. As William Dickens of the Brookings Institution points out, the problem is that it is nearly impossible to test ability without also testing achievement.

Why is this problematic? Well, modern workplaces increasingly require employees to solve unfamiliar problems, often without prior knowledge or context. If current GCA measures rely, even partially, on prior knowledge, they won’t fairly gauge a person’s ability to work in an increasingly ambiguous environment.

Further, the internet is increasingly making Test of Achievement obsolete. Workers have whatever context they need at their fingertips. Test of Ability, being able to quickly teach yourself a skill, apply a framework, and gather the information you need to solve a problem, are now much better measures of general cognitive ability - not because they are more accurate, but because they’re more relevant.

I suspect that the inability to measure the ability to solve unfamiliar problems is the reason that Lazslo Bock, then Google’s SVP of People, said this in a 2014 interview:

G.P.A.’s are worthless as a criteria for hiring, and test scores are worthless. ... We found that they don’t predict anything.

Particularly in the context of Google’s work environment, Bock’s comment makes sense. Test scores and G.P.A. are a rollup of many different Tests of Achievement. In other words, they’re reflections of what someone already knows, not of their ability to learn and adapt.

Relying on GCA can mask more significant candidate issues

Some of the most intelligent people I’ve ever worked with were huge jerks who eventually alienated nearly everyone around them. I suspect that you’ve run across these people too.⁴

I’ve noticed a tendency for myself and my colleagues to forgive the shortcomings of intelligent candidates in the hopes that their brainpower will be worth whatever trade-offs we have to make. But, in my experience, this is never a smart bet. General cognitive ability is not a proxy for decency.

My own mixed feelings on General Cognitive Ability

Personally, I’ve had mixed feelings about this post. It’s true that General Cognitive Ability is a great predictor of candidate success. But I’ve just seen too many hiring managers overvalue a certain school, G.P.A., or assessment score - to the point where they excuse obvious candidate shortcomings, perpetuate existing inequities, and I suspect reduce the probability that we will hire the candidate we need.

We would do much better for our candidates and ourselves if we thought about cognitive ability critically and as a singular data point rather than the first hurdle for a candidate to clear.

Next week I’ll talk about some Do’s and Don’ts as they relate to evaluating General Cognitive Ability. Spoiler: your first filter on a candidate shouldn’t be their G.P.A.

And, as always, thanks again for reading! If you have any thoughts on GCA I’d love to hear them in the comments below.

1. Need proof? He wrote a paper entitled The Role of General Cognitive Ability and Job Performance: Why There Cannot Be a Debate. As I said, the dude LOVES them.

2. For the purposes of this post, when I refer to “GCA assessments” I’m referring to specific assessments from assessment companies as well as the use of SAT, ACT, and G.P.A as proxies for cognitive ability.

3. The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings, 264.

4. Here’s something to help if you find yourself working with a brilliant jerk.