Bryan Caplan  

Hedengren's Dog

PRINT
DeLong on Obamacare... Driving the Cool Kids Out...
Two decades ago, economists started taking intelligence seriously.  Now economists are starting to take conscientiousness seriously.  Unfortunately, most existing data sets don't contain personality tests.  Even when they do, personality tests are only self-reports. Wouldn't it be great if we could retrofit every existing data set with a behavioral measure of conscientiousness?  Wouldn't it be even better if we could do this for free?

If you asked me these questions a few months ago, I would have replied, "Alas, that's impossible."  But GMU Ph.D. student David Hedengren has since changed my mind the old-fashioned way: By doing the supposedly "impossible." 

Hedengren's* new working paper "The Dog that Didn't Bark: What Item Non-Response Shows about Cognitive and Non-Cognitive Ability" hypothesizes that more conscientious people will try harder to completely answer surveys.  As a result, the failure to answer survey questions tells us something about your character.
[S]urveys contain a valuable but neglected piece of data: what respondents do not say. Respondents skip, refuse to answer, or claim ignorance on at least a few questions in virtually all surveys. When a respondent forgets to fill in answers to some questions on the survey form, or refuses to provide an answer to the interviewer, we gain important information about respondents. For example information about how careful respondents are, how much they value privacy, and how open they are to sharing personal information with a stranger.
The beauty of this hypothesis is that you can test it with virtually any pre-existing data set - even data sets that don't officially try to measure personality.  Hedengren then applies his insight to two major surveys: the German Socio-Economic Panel (SOEP) and the Survey of Income and Program Participation (SIPP). 

The SOEP already explicitly measures conscientiousness.  Hedengren shows that his non-response measure alone is a better predictor of income than the SOEP's self-reported conscientiousness:
We then test whether item response or conscientiousness and IQ, as measured in the SOEP, is a better predictor of economic outcomes. We find that item response is more strongly correlated with income than self-reported measures of conscientiousness. We also find that although item non-response is correlated with IQ it is not merely a proxy for IQ since it still is a significant predictor of earnings even after controlling for IQ.
The SIPP, in contrast, does not explicitly measure conscientiousness.  But Hedengren's measure is also a very good predictor of income in the SIPP as well as the SIPP Gold Standard (which uses administrative records instead of self-reports to measure income):
We demonstrate the potential usefulness of this metric by applying it to SIPP and the SIPP Gold Standard. In both self-reported personal income and annual earnings from administrative records, we show that item non-response is a consistent predictor of earnings with the same magnitude as graduating from high school relative to not graduating.
Predicting academic success is hard, but I've never been more confident that a GMU student's working paper will end up in a top journal.  Hedengren's paper is a godsend for every empirical labor economist.  They can immediately start using his novel metric, free of charge:
Our findings suggest that item-non response is a useful proxy for understanding human behavior. This proxy can be of great use to researchers since it already exists on every survey ever conducted without adding any additional respondent burden or data collection cost.
P.S. You should hire this man.

* "The Dog That Didn't Bark" is co-authored with Thomas Stratmann, but Stratmann assures me that Hedengren originated the key idea and did the lion's share of the work.



COMMENTS (22 to date)
AndrewC writes:

Given that this is dated for tomorrow (11/26/2012), I can think of four explanations:

1) A setting on my computer has generated a future date.
2) This update is forward-dated and anomalous.
3) I have never noticed that the timestamp on econlog runs ahead of EST.
4) Travelling has resulted in this post being made in an eastward time zone.
5) Dr. Caplan has given up on forecasting and has decided to literally blog from the future.

Leaning towards three; could be sold on five.

Fabio Rojas writes:

That's a great use of non-response. The nice aspect of this is that it's ironically complete data. There is a value for every respondent that answers at least one question.

moo writes:

[Comment removed for supplying false email address. Email the webmaster@econlib.org to request restoring your comment and comment privileges. We'd be happy to publish your comment. A valid email address is nevertheless required to post comments on EconLog and EconTalk.--Econlib Ed.]

Steve Sailer writes:

Paul Tough's current bestseller "Why Children Succeed" has a write-up on a study of conscientiousness on a low g-loaded part of the ASVAB using the NLSY that was fairly similar:

http://www.slate.com/articles/health_and_science/science/2012/09/how_children_succeed_book_excerpt_what_the_most_boring_test_in_the_world_tells_us_about_motivation_and_iq_.2.html

The people who finished this extremely boring subtest did better than their IQ would predict at various measures of being a good citizen and neighbor.

Steve Sailer writes:

As I wrote in my 2007 IQ FAQ:

Q. Wait a minute, does that mean that maybe some of the predictive power of IQ comes not from intelligence itself, but from virtues associated with it like conscientiousness?

A. Most likely. But perhaps smarter people are more conscientious because they are more likely to foresee the bad consequences of slacking off. It's an interesting philosophical question, but, in a practical sense, so what? We have a test that can predict behavior. That's useful.

Hamish Barney writes:

Does that mean that we will soon see calls from the public and politicians for people to be forced to complete these surveys to ensure that their income rises? ;)

Chris H writes:

Hamish Barney writes:

Does that mean that we will soon see calls from the public and politicians for people to be forced to complete these surveys to ensure that their income rises? ;)

If that happens then we'll have a perfect measure for the proportion of the population that doesn't understand correlation-causation fallacies! Science still wins!

Kevin writes:

I have an upcoming survey that will contain a (self-report) measure of conscientiousness and also various economic and health/risk behaviours. I'll have to see whether survey completion correlates with any relevant outcomes.

daubery writes:

I am moderately surprised. I consider myself to be quite consciencious, and I often decline to answer survey questions if I believe they are ambiguous or the allowed answers do not precisely fit my view. I think if I were less consciencious I would just put down something and move on.

Perhaps there are just a lot more people who refuse to answer survey questions because they aren't interested or don't want to think too hard, or simply give up half way through?

egd writes:

daubery writes:

I am moderately surprised. I consider myself to be quite consciencious, and I often decline to answer survey questions if I believe they are ambiguous or the allowed answers do not precisely fit my view. I think if I were less consciencious I would just put down something and move on

This could lead to an interesting "conscienciousness" survey. Give respondants a number of questions and answers that are ambiguous or do not precisely fit views (perhaps test students and provide a correct analysis but an incorrect conclusion) and see how, and whether, respondants answer the questions.

d writes:

"Two decades ago, economists started taking intelligence seriously."

Clearly, as we've seen with the heavyweights in the field like Acemoglu and Robinson or Paul Collier. They're taking it very seriously.

David R. Henderson writes:

@Bryan,
Excellent sight on Hedengren's part. Plus, anyone who leads off with my favorite passage from Sir Arthur Conan Doyle's work starts off with a plus in my book. I've often used the "dog that didn't bark" idea in my writing.

Like daubery, though, I'm a little surprised. I often "shirk" on questionnaires when I find the questions too intrusive and always, on principle, refuse to answer questions about my race.

Bob Knaus writes:

@David,
I too dislike questions about my race. I always pencil in "Human" and encourage others to do the same. My proselytizing has had little success.

Seth writes:

I had the same thought as Hamish Barney.

@Chris H. - I think we already have that with the percentage of people that believed home ownership caused conscientiousness.

Re: the paper

I'm not convinced that survey 'item non response' and income are indicators for conscientious.

I'm reminded of a recent EconTalk podcast with Paul Tough. He talked about an M&M experiment. Some kids, when given M&M's for correct answers on IQ tests test with higher IQ's. Maybe those kids respond to specific incentives better than the other kids where their IQ didn't fluctuate with potential rewards.

Was there any control for the incentives given for taking surveys? Maybe the folks with higher incomes just respond to such incentives better and what you're really picking up is their propensity to respond to financial rewards, rather than their conscientiousness.

Steve Sailer writes:

""Two decades ago, economists started taking intelligence seriously.""

I think the phrase "a tiny number of" was accidentally dropped from that sentence.

For instance, exactly one decade ago, I attempted to show an economist (one of the more well-known Nobel laureates, one whose expertise is in "human capital") the then-new book "IQ and the Wealth of Nations," and he looked at me like I had lice crawling in my eyebrows.

gwern writes:

This is cool work indeed.

The main downside I see is that it looks to me like you need to have pretty much the full survey response data-set, right? You can't get away with just a summary statistic like 'item response rate was avg 99%'. How many big interesting surveys have that raw data available?

----

Chris H: hah.

Daubery: almost by definition, if tons of respondents acted as you did, the survey givers wouldn't bother to ask those questions (since it would be a waste of time and they'd be asked why they were collecting data which is probably biased badly somehow).

http://www.gwern.net/Conscientiousness%20and%20online%20education

MingoV writes:
When a respondent forgets to fill in answers to some questions on the survey form, or refuses to provide an answer to the interviewer, we gain important information about respondents.

I believe the author drew too many conclusions from too little data. There are other explanations for failure to answer survey questions than either low conscientiousness or a desire for privacy. Here are a few:

1. Boredom with taking the survey. Every lengthy survey has more non-responses at the end than at the beginning.

2. Annoyance. Someone starts answering what he thinks is a survey on a particular topic and then gets annoyed when other topics appear.

3. Dislike of bias. A survey taker realizes that questions are being asked in a non-neutral way and refuses to answer them.

I have failed to respond to survey questions for all three of the above reasons, and I doubt I'm alone.

JVA writes:

From paper - "We find that individuals with higher response rates are less likely to die."

Nobody cares about the long run :(

John Ray writes:

Response sets tend not to generalize
This won't either

hanmeng writes:

I'm posting a comment to demonstrate conscientiousness. I can't wait for the money to roll in!

Floccina writes:

Hamish Barney wrote:

Does that mean that we will soon see calls from the public and politicians for people to be forced to complete these surveys to ensure that their income rises? ;)

[begin half sarcasm]Only if more conscientious people vote for one party than the other. If they do then when the that party that the more conscientious people vote for gets congress and the presidency they will push for forced completion of these surveys. Don't laugh too much the Republicans tried to increase home ownership when they got congress and the presidency because home owners are more likely to vote Republican Ha. [end half sarcasm]

gwern writes:

FWIW, I recently tried the non-response thing on a website community survey which included a Big Five questionnaire: http://lesswrong.com/lw/fp5/2012_survey_results/#7xl5

No correlation between number of questions omitted and the reported C.

Comments for this entry have been closed
Return to top