Arnold Kling  

Data and Dogma

PRINT
Political Economy, Banking, an... Calomiris on the Crisis...

(i). Bob Sutton quotes a meta-analysis by Frank Schmidt and John Hunter published in 1998 on the factors found to affect job performance.


1. GMA tests ("General mental ability")

2. Work sample tests

3. Integrity tests: surveys design to assess honesty ... I don't like them but they do appear to work.

4. Conscientiousness tests...

12. Job experience (years)...

16. Years of education

17. Interests

18. Graphology (e.g., handwriting analysis)


I love it that "years of education" just barely beats out handwriting analysis.

(ii) Shannon Brownlee and Jeanne Lenzer write,


Jackson's findings showed that outside of flu season, the baseline risk of death among people who did not get vaccinated was approximately 60 percent higher than among those who did, lending support to the hypothesis that on average, healthy people chose to get the vaccine, while the "frail elderly" didn't or couldn't. In fact, the healthy-user effect explained the entire benefit that other researchers were attributing to flu vaccine, suggesting that the vaccine itself might not reduce mortality at all.

A basic lesson in AP statistics concerns the differences between an observational study and an experiment. There is a well-known finding that people who get flu vaccinations have lower death rates than people who do not. But this finding is not based on an experiment. It is instead based on observation of people who choose to get shots and people who do not. My hypothesis is that people who get flu shots are more conscientious than people who do not, and more conscientious people have lower death rates. Whatever the reason, the article cites research where what economists would call "natural experiments" show that flu shots do not affect death rates.

Consider the well-known observation that people with health insurance have lower death rates than people without health insurance. This, too, is based on an observational study, and once again it is likely that conscientiousness is a confounding factor. The Rand health study, which was a controlled experiment, found no differences in outcomes from different levels of insurance.

A final point is that in the public debate, I suspect that the demand is not for reliable studies but for studies that support one's point of view. Brownlee and Lenzer report on the way that vaccine skeptics are bullied by those with whom they disagree.

My advice to laymen would be to pay attention to how scientists argue. When the ad hominem attacks and bullying are flowing freely, assume that the attacking side has problems assembling convincing evidence.


Comments and Sharing





COMMENTS (13 to date)
SydB writes:

"When the ad hominem attacks and bullying are flowing freely, assume that the attacking side has problems assembling convincing evidence."

I'm not so sure: ad hominem often seems tangential to the issue of truth and evidence. Some brilliant people with strong arguments just can't help but bring ad hominem into their arguments. It's a personality trait/flaw. This doesn't make them less correct, just correct jerks.

Also consider: When the charges of ad hominem arise, it can also be considered a lack of evidence by those making the charge, because those making the charge focus on the ad hominem aspects that exist in ALL or MOST communications at some level.

Ad hominem changes the topic, and most arguments contain a mix of more or less relevant propositions. But responding to ad hominem with a charge of it is to accept the changed topic and focus on it--perhaps because one's counter argument to the main topic is weak.

Marcus writes:

"There is a well-known finding that people who get flu vaccinations have lower death rates than people who do not. ... My hypothesis is that people who get flu shots are more conscientious than people who do not, and more conscientious people have lower death rates."

Certainly such studies take into consideration the cause of death, do they not?

Jack writes:

Years of education may be collinear with mental ability: the latter captures benefits from education, formal and informal. Yrs ed is then redundant, no?

Ironman writes:

Arnold writes:

I love it that "years of education" just barely beats out handwriting analysis.

Academic credentials do not equal competence.

david writes:
Brownlee and Lenzer report on the way that vaccine skeptics are bullied by those with whom they disagree. [...] When the ad hominem attacks and bullying are flowing freely, assume that the attacking side has problems assembling convincing evidence.

Creationists and homeopaths are also roundly 'bullied' by the establishment, but that hardly suggests a lack of evidence. Rather a lack of patience, which is fully justified if you regard your opponent as being duplicitous or willfully ignorant in some manner.

In this case the current anti-vaccination alliance is dominated by people who sincerely believe that vaccines cause autism. There may be a recent surge of new members who suddenly want to criticize the notion of public health in general - always a libertarian bugbear - but I daresay that Kling doesn't quite appreciate who the poor 'attacked' victims are in this case.

For an 'establishment' view on the Atlantic article, see this example.

Remember that summoning well-reasoned and careful arguments takes time and work, and it is an actual documented strategy to waste scientist time in order to force them into either ignoring you (then declare victory), insulting you (then declare victory), or stopping their work in order to engage with you full-time (in which case you win anyway). There is an endless supply of pseudoscientific crankery and a limited supply of scientists who are also skilled in popular discourse. Would-be economists are welcome to try charting the equilibrium.

SydB writes:

"Remember that summoning well-reasoned and careful arguments takes time and work, and it is an actual documented strategy to waste scientist time"

Exactly. An argument I've often made. It's a "death by a thousand cuts" problem.

And going to my previous point, Isaac Newton and Leibniz, for example, often engaged in Ad Hominem. Ad hominem is old and has little or no bearing on the truth or falsity of an argument. It's added propositional noise, to be filtered out or ignored in most cases. And it is possible that those with weak arguments tend to look for ad hominem so they can disengage. The declaration of ad hominem is as much a declaration of weakness as it is of insight.

That said, ad hominem should be avoided and pointing out its SPECIFIC occurrences is appropriate. Or eventually ignoring people (or mocking them) if ad hominem is all they are capable of.

Regarding the "years of education" issue above: It's not clear to me how years of education can easily be incorporated into a study such as this. Most people have 12 (high school), 12+4 (bachelors), 12+4+1/2 (Masters), 12+4+~6 (PhD). How are these discrete values incorporated into a study such as this. And people study different topics (engineering, mathematics, psychology, etc). I think education is next graphology because they didn't have a way to incorporate it properly.

Jack writes:

@SydB:
A simple improvement over using Yrs education is (in a regression framework) to use dummy variables: No High School, BA/BS, MA/MS, PhD/MD (with High School as the omitted standard).

Steve Sailer writes:

I got hired in 1982 at a new, small marketing research firm that went on to revolutionize its industry. The HR department's hiring test was was simply the second semester final for one of the founders, a leading academic in marketing research, from his Advanced Quantitative Methods for Marketing Research 402 class. I spent 3 to 4 hours sweating over it, did well, and got hired. This demonstrated 3 of the top 4 factors: general mental ability, work sample, and conscientuousness.

Unfortunately, as our firm grew, we showed up on the EEOC's radar screen and they told us we couldn't use the test anymore without an expensive validation process. The quality of of our hires declined.

Dr. T writes:

"...the healthy-user effect explained the entire benefit that other researchers were attributing to flu vaccine, suggesting that the vaccine itself might not reduce mortality at all."

Hog swill.

It is certain that flu vaccinations resulted in fewer deaths among healthy persons and among the frail and elderly. Why? Because vaccinations reduce the prevalence of contagious diseases, and that means fewer future infections. You cannot judge the effectiveness of a vaccine by only looking at the death rates of those who got the flu. You also must look at the lowered death rates of those who never got the flu because many others got vaccinated. The well-known epidemiologic phenomenon of 'herd immunity' is based upon this, and I don't know why the authors ignored it.

Jim Ancona writes:

Dr. T,

While herd immunity is unquestionably important for many vaccines, a quick search seems to confirm my intuition that influenza vaccination rates and vaccine efficacy rates are too low for herd immunity to be much of a factor. Do you have any evidence that shows otherwise?

SydB writes:

Jack: Agreed. But I'm still thinking it's not going to tell us much because it's not the right way to address the issue. Particular jobs require a particular type of training (say computer programming) and, for a given job, most candidates will have four year degrees. There's very little to measure.

I say it's a largely meaningless piece of data. Except that there are not a few people with college degrees who end up with jobs that they could have entered (trained for) without those college degrees.

CJ Smith writes:

Regarding ad hominem attacks, I reminded of a half-truism prevalent in legal circles:

"When you have the facts on your side, pound the facts;
When you have the law on your side, pound the law;
When you have neither the facts nor the law on your side,
pound the table and opposing counsel."

Walt French writes:

Odd that SB & JL distinguish observation from experiment, without referring to the most important component of science: a falsifiable theory, without which observation is merely a jumble of disconnected anecdotes, and experiments are as useless as alchemists' efforts to turn lead into gold.

Not that the track record of vaccination is weak: we usually think of Jenner's 1770 work with cowpox (hence, "vaccine") as being the start of immunization, but China was inoculating people a millennium ago and there are suggestions of it 1000BC in India. While economic ideas ebb and flow with the fashion of the latest source of wealth, scientists and doctors have built an incredibly detailed understanding of HOW vaccination works. The theoretical model underneath viral diseases grows stronger by the day, as thousands of scientists pool their efforts for the public good.

Immunization generally is an unquestioned success, having eradicated smallpox, a major killer, during our lifetimes. People without robust immune systems are alive today simply because of it. The spread of disease has been vastly limited by it. To casually toss out a lazy "hypothesis" of vaccination being unhelpful is to ignore the mechanism and history of success of all public medicine. What motivates professionals to expose such sloppy thinking outside of their field of expertise?

I trust that the economic hypotheses developed at EconLog are more inspired by appreciation for all the facts that the fevered imagination that we see applied here to a spotty reading of a part of some small set of data.

Comments for this entry have been closed
Return to top