The main finding in Philip Tetlock’s awe-inspiring Expert Political Knowledge is that open-minded “foxes” are better predictors than theory-driven “hedgehogs.” But toward the end of the book, he has a fascinating chapter about a fascinating exception.
Background: There’s a whole industry of “scenario consultants” who claim to mitigate overconfidence – and attendant financial embarassment – by making businesspeople imagine reasons why they might be wrong. As Tetlock explains:
They appeal to clients to stretch their conceptions of the possible, to imagine a wider range of futures than they normally would, and then to construct full-fledged stories that spell out the “drivers” that, under the right conditions, could propel our world into each alternative future.
Tetlock tried out this technique on his political experts. He made them imagine all sorts of possible scenarios about (a) Canadian disintegration, and (b) the Japanese economy. His findings:
1. Before the exercise, people’s probabilities at least added to about 100%. The exercise led participants to violate this elementary logical principle: after thinking about different scenarios, every outcome seemed more likely. For the Canadian problem, the average expert gave all the possibilities a 158% chance of happening. The error, in essence, was that merely imaginining possibilities, however outlandish, made people take them seriously. As Tetlock explains:
One takes a vague abstraction, all possible paths to Canada’s disintegration, and explores increasingly specific contingencies. Quebec secedes and the rest of Canada fragments: the Maritimes – geographically isolated – clings to Ontario, but Alberta flirts with the United States… [T]he images are vivid, the plotlines plausible, and it becomes increasingly taxing to keep all the other logical possibilities in focus.
2. The exercise led people to raise their probabilities for events that didn’t happen. In other words, more openness led to less accuracy.
3. The people most subject to this error were foxes who specialized in the topic at hand! Hedgehogs’ “dogmatic” scoffing at odd hypotheticals saved them from a serious intellectual embarassment.
One upshot is that the scenario consulting industry purports to help solve the problem of overconfidence, but in fact seems rather overconfident in the value of their method.
But for me, the more fundamental lesson is that – the whole literature on overconfidence notwithstanding – it’s not that hard to make people suffer from excessive open-mindedness. All of which backs up one of my favorite sayings: “Keep an open mind, but not so open that your brain falls out.” (Richard Feynman)
READER COMMENTS
spencer
Jan 16 2006 at 9:11am
You might apply this lesson to the issue of price controls on fuel in Hawaii about 6 months ago. You, essentially every other conservative blog — especially by James Mason profs. — , the WSJ editorial page, the Heritage Foundation, the AEI, and I could go on and on, published profound article about how it would cause shortages, lines and hurt poor people.
But all of you were completely wrong.
Why?
I have my own thesis, and it has nothing to do with left vs right wing biases. But I would like to see some of you explain why you were completely wrong before I give my thesis.
Randy
Jan 16 2006 at 9:22am
Spencer,
Actually, I was completely right. I posted over on Jane Galt that the prices would ease due to market forces, but that in Hawaii the politicians would take credit for it.
spencer
Jan 16 2006 at 11:19am
Congradulations Randy– even though I have not heard many politicans taking credit for the lower prices.
but that does not negate my question of doesn’t having so many people being wrong imply that there is something wrong with the methodology they all applied?
John P.
Jan 16 2006 at 2:01pm
Spencer — I realize that this is irrelevant to your point, but I assume you’re referring to George Mason profs., not James Mason profs.
“That wasn’t very sporting, using real bullets.”
Chris Bolts
Jan 16 2006 at 2:41pm
Well, spencer, you were right because of the price decreases. But I’d like to see you hold to that assessment as gasoline prices rise.
Barkley Rosser
Jan 16 2006 at 3:45pm
John P and spencer,
No, these guys are at George Madison University… :-).
John P.
Jan 16 2006 at 5:30pm
No, Barkley, you must be thinking of George Kennedy.
“My boy says he can eat fifty eggs, he can eat fifty eggs.”
James
Jan 16 2006 at 9:15pm
I have a theory as to why so many conservative bloggers were wrong about Hawaiian price controls. They were basing their ideas on this syllogism:
1. If a price ceiling is imposed below the price at which a market will clear, there will be shortages.
2. Hawaii’s law imposed a price ceiling below the price at which a market will clear.
3. Therefore there will be shortages.
It’s a valid argument with a false conclusion, so either premise 1 is false, or premise 2 is false, or both. I think the myriad conservative bloggers were wrong in their conclusion because 2 is a false premise. But then I’m probably being narrow minded.
Comments are closed.