Philip Tetlock, one of my favorite social scientists, is making waves with his new book, Expert Political Judgment. Tetlock spent two decades asking hundreds of political experts to make predictions about hundreds of issues. With all this data under his belt, he then asks and tries to answer a bunch of Big Questions, including "Do experts on average have a greater-than-chance ability to predict the future?," and "What kinds of experts have the greatest forecasting ability?" This book is literally awesome - to understand Tetlock's project and see how well he follows through fills me with awe.
And that's tough for me to admit, because it would be easy to interpret Tetlock's work as a great refutation of my own. Most of my research highlights the systematic belief differences between economists and the general public, and defends the simple "The experts are right, the public is wrong," interpretation of the facts. But Tetlock finds that the average expert is an embarassingly bad forecaster. In fact, experts barely beat what Tetlock calls the "chimp" stategy of random guessing.
Is my confidence in experts completely misplaced? I think not. Tetlock's sample suffers from severe selection bias. He deliberately asked relatively difficult and controversial questions. As his methodological appendix explains, questions had to "Pass the 'don't bother me too often with dumb questions' test." Dumb according to who? The implicit answer is "Dumb according to the typical expert in the field." What Tetlock really shows is that experts are overconfident if you exclude the questions where they have reached a solid consensus.
This is still an important finding. Experts really do make overconfident predictions about controversial questions. We have to stop doing that! However, this does not show that experts are overconfident about their core findings.
It's particularly important to make this distinction because Tetlock's work is so good that a lot of crackpots will want to highjack it: "Experts are scarcely better than chimps, so why not give intelligent design and protectionism equal time?" But what Tetlock really shows is that experts can raise their credibility if they stop overreaching.