Bryan Caplan  

Predict the More Predictive Test

PRINT
Field Review... Anecdotes of an Informal Econo...
IQ scores predict a wide range of life outcomes and respond to incentives.  But almost all of the evidence that IQ predicts life outcomes comes from IQ tests that don't use incentives.  Which raises a big question: Would incentivized IQ tests be better or worse predictors of life outcomes than standard IQ tests?

I see arguments on both sides.  You could say that...

1. Incentivized IQ tests are cleaner measures of true intellectual ability, and will therefore better predict life outcomes.

2. Life outcomes are incentives, so incentivized IQ tests are more ecologically valid predictors of performance.

3. People respond somewhat myopically to incentives.  On this story, the response of IQ scores to incentives will exceed the respond of life outcomes to incentives.  Or from a slightly different perspective: Maybe an important cause of life success is the willingness to try when there aren't blatant incentives to exert yourself.  On either story, standard IQ tests will be more predictive of life outcomes.

4. Standard IQ tests measure intelligence and motivation, and are therefore better predictors of life outcomes than incentivized IQ tests, which are closer to pure measures of intelligence.

5. A key ingredient of life success is trying when effort pays and relaxing when effort doesn't pay.  On this story, incentivized IQ tests will be better predictors of life outcomes.

In the end, my intuition is that standard IQ tests will prove to be better predictors of life outcomes than incentivized IQ tests.  But I'm far from sure.  Your thoughts?  Further evidence?  Bets?


Comments and Sharing





COMMENTS (10 to date)
Philo writes:

It is impossibly difficult to measure the incentive that a particular person has on a particular occasion for doing well on the IQ test he is taking.

jsalvatier writes:

Perhaps incentivized IQ tests + a concienciousness test will predict better than both.

Blake R writes:

I'm inclined to say standard tests would do better for roughly reasons 3 and 4. The standard test partially captures the ability to focus in the absence of immediate or certain rewards. I wonder which of the two actually correlates the most with conscientiousness.

As Philo notes, "conditional monetary payments" are not equivalent to "incentives", although they evidently are motivating for the lower end of the scale. Are there other test variations that could help isolate intelligence from desire to score high on tests or responsiveness to monetary rewards?

Peter H writes:

I have a pretty good idea whether I will do well on an IQ test in advance. Doing well on an IQ test pays in prestige (I get a high score I can brag about). Getting a low score does not pay at all for a standard test. Thus if I think I can score well, I'll try hard to gain the prestige, but if I don't think I can, I'd rather not exert the effort. A conditional payment will change this for the low end, but the high end was already trying hard.

If this is the paradigm in effect, I would expect the tests with conditional payments to be slightly more predictive, since they capture the tendency of people to self-select into areas where they are enjoy comparative advantage. Thus someone who does poorly on the conditional payment test would be someone who either does not seek comparative advantage, or is just genuinely not very smart. By way of example, someone who is bad at math but extremely personable may not bother trying on an IQ test without a cash reward, but may make a buttload of money as a salesperson, and will have selected into sales because its where they enjoy a comparative advantage.

Doug writes:

A very easy way to tell. Does IQ + Conscientiousness on the Big 5 have better predictive power than IQ alone. Let's put it this way, say there's some true linear model:

Life Outcome ~ Intellect + Conscientiousness

But everyone thinks that IQ may be actually be a combination of intellect and conscientiousness. Well if you fit a linear model you're still going to see the same R^2, it's just that the coefficients will be different compensate for the conscientiousness loading in IQ.

E.g. if the true model was

Life Outcome ~ a * Intellect + 0 * Conscientiousness

And IQ's model is

IQ ~ 1 * Intellect + b * Conscientiousness

Then the linear fit will be

Life Outcome ~ a * IQ - b * Conscientiousness


Thus incentivized IQ tests should offer no predictive power above unincentivized IQ tests combined with a conscientiousness score from a personality test.

John writes:

If the incentives are compatible with what we see as successful life outcomes then the incentive tests should do better.

If internal motivation is more important to successful life outcomes then the non-incentive tests will predict better.

We're still probably stuck with the problem of what the person sees as successful life outcome versus what the generally applied external metrics are. Is becoming the CEO or owning the business the metric? Is a happy family life with modest finances the metic? Some level of wealth at retirement the level -- or ability to stop working at some age -- the metic? Was citizen Cain successful?

On a related aspect, I was talking with a friend the other day about these results and he pointed out some other studies found similar results -- but also found that when the incentive was removed the scores fell below what they were before the incentive was offered.

That raises a question, if performance on such tests does provide some indication of future performance does an enculturation of the new generation into an externally incentive mind-set bad for society?

ajb writes:

As Bryan and others suggest, Duckworth, et al are incorrect in assuming that monetary payments truly "incentive" tests. The more rigorous claim would be that adding short term incentives to the incentives already built in to doing well on tests seems to motivate low g test takers more than high g test takers. So what? It says that over responsiveness to short term incentives is itself consistent with low conscientiousness being correlated weakly with low IQ.

Steve Sailer writes:

The meta analysis by Duckworth that Bryan is citing is overly dependent upon three studies in the 1970s by a researcher who was later convicted of fraud. Take out those three studies and Duckworth's effects drop just out of the statistical significance range.

http://statsquatch.blogspot.com/2011/05/bad-apple-in-duckworths-iq-motivation.html

Steve Sailer writes:

Our society has a huge number of high stakes tests whose results strongly correlate with IQ tests: SAT, LSAT, GRE, ACT, MCAT, GMAT, military's AFQT, civil service exams, the NFL's Wonderlic, etc etc.

The main demographic diversion from IQ results visible are two:

- Predictably, Hispanics raised speaking Spanish tend to underperform on English language high stakes tests relative to their IQs measured on culture fair IQ tests such as Ravens Matrices, because of their deficiencies in English. Hispanic born in America tend to do about as well as expected on high stakes tests based on their IQs.

- Asians increasingly tend to outperform measured IQs on high stakes tests probably due to intensive test prep.

Otherwise, there isn't much difference demographically between what you see on low stakes tests and high stakes tests.

Steve Sailer writes:

Also, the best-known IQ test, the Wechsler, which emerged in the 1930s, is quite fun to take. (It's copyrighted and closely guarded, but you can read a description of what it's like to take the Wechsler in Dan Seligman's 1992 book "A Question of Intelligence.")

It's administered one-on-one in a conversational setting by a psychologist asking a wide variety of questions, some easy, challenging. The tester varies the difficulty of the questions depending upon how the testee is doing. They constantly move on to quite different types of questions before the novelty can drag. Testers are trained to play upon subjects' natural desires to impress a competent person, their curiosity to find out how they do, their competitiveness, and other positive urges.

Comments for this entry have been closed
Return to top