Bryan Caplan  

Studies Show

PRINT
Outsiders... Morning Links...
Warden: The part of the brain that remembers dance steps is also the anger center. So, juveniles who know how to fox trot are 10% less likely to commit a double homicide.

Prisoner: Who conducted this study?

Warden
: The Institute of Shut Your Fat Face!

               -The Simpsons, "The Wandering Juvie"
Whenever non-academics tell me that "studies show X," I cringe.  It's not because I'm skeptical of empirical research.  It's because the "chain of custody" from researchers to the man in the street is multiply broken.  Scholars publish results, some popularizers mis-summarize them, other popularizers pick and choose the mis-summaries they want to trumpet, and the public remembers the trumpeted mis-summaries that comfort and/or terrify them. 

And that's when the system is "working"!  I've also noticed that non-academics just let their imaginations run wild, then try to make their say-so stick by alluding to "studies."

If this depresses you, let me point out that matters are much better than they used to be.  It is now ridiculously easy to (a) call shenanigans on vague references to "studies" and (b) find out what studies actually show.  Using Google Scholar, for example, you can quickly locate the main serious research on any topic of interest.

To pick a topic at random, I decided to look into the health risks of oral contraceptives (OC).  I've often heard vague talk about blood clots and breast cancer being "linked" to OC, followed by heavy-handed insinuations that OCs are too dangerous to use.  If that is really the issue, though, the obvious question to ask is simply, "Do OCs raise overall mortality?"  A quick Google Scholar search of "oral contraceptives mortality" reveals multiple large high-quality studies, all of which find that OCs do not raise mortality.  Searches for the link between specific ailments and OCs produce less clear-cut results, but also reveal plenty of studies claiming health benefits of OCs.

Most complaints about "studies" are rooted in agnosticism.  Who can really know what's true and what's not?  My point is that agnosticism is misplaced.  Yes, blanket doubt is better than falling for fashionable nonsense.  But not only is the Truth is Out There; it's a lot closer than you think.


Comments and Sharing





COMMENTS (14 to date)
Stephen Z writes:

I laud your point but I cannot help but feel that you are preaching to the choir here.

steve writes:

"But not only is the Truth Out There; it's a lot closer than you think."

I think this implies that the evidence the public sees from the internet should lead them to be more trusting of experts. It is not clear to me that this is so. For example, there is clip after clip of Bernanke making claims about the solidness of the economy, the absence of a housing bubble etc.

When the nations arguably most important economist leaves a lengthy and easy to find trail of plainly incorrect predictions, why would the average person conclude that economists know what they are talking about.

steve writes:

Then faced with this "Bernanke" evidence. The average person is supposed to dive into the details of economic studies and divine which ones are correct. Much easier just to watch the contrasting Peter Schiff youtube clips and make your judgements that way. Course, this decision making process is susceptible to the broken watch.

steve writes:

[Comment removed pending confirmation of email address. Email the webmaster@econlib.org to request restoring this comment. A valid email address is required to post comments on EconLog.--Econlib Ed.]

Robert Arbon writes:

Good point Sir. The is a columnist (and full time medic) for the UK's Guardian who makes a regular feature of investigating the various stories of reporters claiming to be backed up by studies.

http://www.badscience.net/

The superficial subject matter is science and medicine however.

A particular egregious example may be found here:

http://www.badscience.net/2009/07/asking-for-it/

pk writes:

While I completely agree with the sentiment, I cringe at the notion the guilty can only be non-academics. Academics are not immune from faulty logic or from cherry-picking "facts" to make a point to their choosing.

Greg Ransom writes:

studies show that much of the pathology begins within the research itself -- the biases which determine what will be published, and what spin is put on data by researchers, etc.

I'm surprised you aren't familiar with this literature.

guthrie writes:

You could have easily included a second Simpson's sequence:

TROY MCCLURE
Just ask this Scientician!

QUICK SHOT OF MAN IN LAB COAT
'Uh...'

TROY MCCLURE
He'll tell you about the 'Food Chain'!

agnostic writes:

This broken chain of custody is even worse within academia itself. Popularizers and the man in the street have ideological biases, but it's those who have real skin in the academic game who are the most fearful of research that contradicts their worldviews.

Just about every academic field has its set of urban legends. Instead of "studies show," it's "As is well known, Hergerdorfer (1965) demonstrated..." Hey, if they said "as is well known," I won't bother looking up that citation!

There's a balance we have to strike between credulity and skepticism of so-called experts. Maybe some academic fields have better honesty policies, but in general I'd be more skeptical of academic citations that don't include the raw data than I would be of popular citations that don't include the data.

The popularizer may be trying to keep things simple presentation-wise for a mass audience. But if an academic article doesn't spell out the magnitude of the beta coefficients in "Hergerdorfer's (1965) regression demonstrating that...", that's even more suspect. Ditto when an academic article has regression coefficients, etc., *but no plot of the data*.

Yet few academics bat an eyelash when they read a data-free citation in a journal.

And this isn't even mentioning the problem of "As is well known" worldviews that are based on a single study! Let alone the trend toward burying the raw data deeper and deeper into a cyber-labyrinth where few people will bother twisting through. Articles in high-impact journals now read more like ScienceDaily news summaries -- no tables, plots, etc., and even the text uses adjectives like "large effect" rather than pinning it down with numbers.

I worry...

S. N. writes:

The internet also makes it easier to find unscientific or biased studies. It's important for people to learn how to distinguish between reliable and unreliable sources. If a website or article looks professional enough some people will be convinced that it is accurate. Online tools should help filter out untrustworthy information. For example, Google Scholar shows the source of an article and lists all of the articles that cite it. This helps one determine if the article is accepted by scientists. Additionally, the ease with which multiple studies can be found using the internet lets you compare a larger number of studies than you could before. This lets you draw a more balanced conclusion. -S.N. WCU 1257

Arnie writes:

LOL. LOL.

Just so you know, I WILL be stealing the "Institute of Shut your fat face" line for future use. Let Matt Groening know.

Brittany writes:

I will admit that I can be very gullible at times and convincing me that something is true based on some quick evidence is not too tough. However, I have become more aware of this weakness of mine and have learned to doubt others until I have some reliable substantiation. Ever since I took a statistics class my junior year in high school I am more suspicious of claims in commercials, advertisements, studies, etc. that we as an audience tend to accept as facts without proof. I know that I am easily convinced by commercials stating that their product will make you lose weight fast or claim other enticing effects. Even statistics are thrown around haphazardly and in most cases are very misleading and biased. For example, when Al Gore came out with An Inconvenient Truth I bought into the global warming theory whole-heartedly. I believed that because he had so much research and statistics behind it that it must be true. Not until I took an environmental class my senior year did I find information that showed how some of his graphs and research was not completely accurate and was skewed to make it fit his statements. I am not saying that global warming is still not an issue, but I am more cautious about believing extreme claims like Gore’s. In general I think that we all need to be more careful about believing everything we hear. And in relation to the government and the economy, we as a people should not be swayed by everything the government says, we need to form our own opinions and find credible information about things like the economic stimulus, the war in the Middle East, universal healthcare, etc.

Joe writes:

Studies, studies, studies. It seems nowadays that there is constantly a new study that explains a commonplace trend that "you" have abided by your entire life without even realizing it. There are studies for everything. Studies for why you brush your teeth with your right hand and not with your left and studies that determine just how successful your kids may or may not be. I grow weary of the studies that surround me, studies that beckon me in to whisper in my ear that "you are not normal...obey". Trends, they are great to know, general trends are beneficial to the human action for they guide what is expected of humane, not human, behavior.

Daublin writes:

I agree with the commenters who object to scholar.google.org being a reliable brand. All it takes to start a journal is a dozen or two people who say that a topic is important. Once it gets bootstrapped, its importance becomes self-fulfilling, because it's the people already involved who decide what papers matter and decide which people are doing the best at it. Outsiders are, by definition, not experts in the area.

However, at the same time, the journal format does sometimes work. The important thing is to check the source. Merely being on scholar.google.com is not that reliable, any more than being in a printed newspaper is all that reliable. However, you can check the people involved, and you can check the journal involved, and you can follow reverse citations to see what other people are saying about the paper.

Thus, Bryan's key point stands: the world has extraordinarily good access to the scholarly world's writings. Please, though, let's not endorse journals with too broad of a brush. There's an awful lot of complete rubish that is only published because there are an awful lot of poor researchers available to review it.

Comments for this entry have been closed
Return to top