Bryan Caplan  

My Defense of Experts Against the Leading Expert

PRINT
Global Warming and CO2 Subprime Daily Briefing, Dec. ...

Forget "the Best of 2007"; Philip Tetlock's Expert Political Judgment may well be the best book ever written on political psychology. (See here for an earlier discussion). I say this even though I'm a big defender of experts, and Tetlock's book is usually interpreted as a grand debunking of experts' pretensions.

How can I live with this tension? Check out my review essay of Tetlock's book in the latest issue of Critical Review (sorry, no online access yet). Its thrust:

Tetlock underestimates experts in general, and does too little to discourage demagogues from misinterpreting his results.

How does Tetlock underestimate the experts? In a nutshell, his questions are too hard for experts, and too easy for chimps. Tetlock deliberately avoids asking experts what he calls "dumb questions." But it is on these so-called dumb questions that experts' predictions shine, relative to random guessing. Conversely, by partitioning possible responses into reasonable categories... Tetlock saved the chimps from severe embarrassment that experts would have avoided on their own.

Furthermore, even if the experts are no better than Tetlock finds, he does too little to discourage demagogues from misinterpreting his results as a vindication of populism. There is only one major instance in which Tetlock compares the accuracy of experts to the accuracy of laymen. The result: The laymen (undergraduate Berkeley psychology majors - quite elite in absolute terms) were far inferior not only to experts, but to chimps.

I add:
As the book is written, it is too easy for a casual reader to think that Tetlock's main point is that political experts are no better than astrologers. If I were Tetlock, I would have tried harder to immunize readers from this misinterpretation. Above all, I would have repeatedly emphasized that "The experts have much to learn, but they also have much to teach," or at least "However bad experts seem, laymen are far worse."
P.S. Don't miss Stephen Miller's "Conservatives and Liberals on Economics: Expected Differences, Surprising Similarities," in the same issue of CR. Here is wisdom:
Liberals and Democrats are usually more likely than conservatives and Republicans to favor government intervention in the economy, and are more suspicious of business. However, the differences between conservatives and liberals are often fairly small, and the data indicate that conservatives, too, are wary of free markets, bordering on being hostile to them - especially when it comes to particulars, rather than abstractions.
Update: Steve Miller points out that:
There is a gated, but free link:
http://www.tandf.co.uk/journals/titles/08913811.asp

Click on "free sample copy" and register -- that should open up the whole issue.


Comments and Sharing





TRACKBACKS (1 to date)
TrackBack URL: http://econlog.econlib.org/mt/mt-tb.cgi/778
The author at The Undercover Economist in a related article titled Too hard for experts, and too easy for chimps writes:
    Bryan Caplan is in his pomp, reviewing Philip Tetlock's Expert Political Judgement:Tetlock underestimates experts in general, and does too little to discourage demagogues from misinterpreting his results. How does Tetlock underestimate the experts? In ... [Tracked on December 13, 2007 1:57 AM]
COMMENTS (5 to date)
Punditus Maximus writes:

Turns out conservatives aren't able to generalize from their own experiences via empathy in order to create a coherent philosophy which applies both in their own life and to society as a whole. So they engage in a lot of "for thee, but not for me."

I didn't precisely need a study to tell me that.

Steve Miller writes:

Thanks for the nod!

There is a gated, but free link:
http://www.tandf.co.uk/journals/titles/08913811.asp

Click on "free sample copy" and register -- that should open up the whole issue.

Steve Miller writes:

Thanks for the nod!

There is a gated, but free link:
http://www.tandf.co.uk/journals/titles/08913811.asp

Click on "free sample copy" and register -- that should open up the whole issue.

Steve Miller writes:

Sorry for the dp!

eric writes:


I find it difficult to understand why you admire his writings so much. Not only because you seem to have a significantly different view of experts, but because I think he findings are bunk. Is the rubber chicken circuit involve that much back scratching? It's one thing to be civil and listen to people you disagree with, quite another to consider your intellectual adversaries the Greatest Thinkers in Their Area of Expertise, because this would suggest your opinions are wrong, or at least inconsistent.

When asked if he is a fox or a hedgehog, he said he didn't know, but perhaps in general a fox with a little hedgehog in him--isn't everyone? One of his examples, on the Iraq war, he found foxes tended to be correct that sectarian violence would occur, but they qualified their predictions so much it wasn't clear what their predictions were ex ante. And this is the general pattern: foxes qualify their statements a lot, but are more correct--when you ignore the qualifications.

He is describing everything and nothing. He can't even categorize himself. If as a fox I state the future will be bright, unless it's cloudy, in which case it won't be, I have 100% of being correct, but that's totally useless. If as a hedgehog I state the earth in 10 years will be colder and a contrary hedgehog takes the other side, one of us hedgehogs will be correct on the big issue. But that's true regardless.

For an expert on forecasts, prediction, and calibration. I wonder if he sees the cognitive dissonance here: he is being a useless expert on the uselessness of experts.

Comments for this entry have been closed
Return to top