Bryan Caplan  

Superforecasting on Epistemic Bait and Switch

PRINT
More really good news... Thanks Texas...
A great discussion of the mistaken Iraq forecasts of the intelligence community (IC) from Superforecasting:
[F]ew things are harder than mental time travel.  Even for historians, putting yourself in the position of someone at the time - and not being swayed by your knowledge of what happened later - is a struggle.  So the question "Was the IC's judgment reasonable" is challenging.  But it's a snap to answer "Was the IC's judgment correct?"  As I noted in chapter 2, a situation like that tempts us with a bait and switch: replace the tough question with the easy one, answer it, and then sincerely believe that we have answered the tough question.

This particular bait and switch - replacing "Was it a good decision" with "Did it have a good outcome?" - is both popular and pernicious.  Savvy poker players see this mistake as a beginner's blunder...  Good poker players, investors, and executives all understand this...

So it's not oxymoronic to conclude, as Robert Jervis did, that the intelligence community's conclusion was both reasonable and wrong.  But - and this is the key - Jervis did not let the intelligence community off the hook.  "There were not only errors but correctable ones," he wrote about the IC's analysis...  Would that have made a difference?  In a sense, no.  "The result would have been to make the intelligence assessments less certain rather than to reach a fundamentally different conclusion."... This may sound like a gentle criticism.  In fact, it's devastating, because a less-confident conclusion from the IC may have made a huge difference: If some in Congress had set the bar at "beyond a reasonable doubt" for supporting the invasion, then a 60% or 70% estimate that Saddam was churning out weapons of mass destruction would not have satisfied them...

But NIE 2002-16HC didn't say 60% or 70%.  It said, "It has..." "Baghdad has..."  Statements like these admit no possibility of surprise.  They are the equivalent of "The sun rises in the east and sets in the west."  At a White House briefing on December 12, 2002, the CIA director, George Tenet, used the phrase "slam dunk."  He later protested that the quote had been taken out of context, but that doesn't matter here because "slam dunk" did indeed sum up the IC's attitude.
As far as I know, Tetlock is not a pacifist, but this is still intriguingly consistent with my common-sense case for pacifism.  I'll know more Monday, when I finally get to meet him in person.  If you see me there, please say hi.




COMMENTS (9 to date)
Bedarz Iliaci writes:

I would query again the significance of the number like 60-70% for non-repeatable events?

How are these numbers calculated? Does there exist an algorithm to calculate the probabilities?

MG writes:

Since you are on a Tetlock state-of-mind, for the visual/auditory inclined, see : http://edge.org/

They sponsored and meeting with Tetlock and 20 members of their expert community and filmed the session.

Baconbacon writes:

@bedarz

The esimate is a level of confidence, which is repeatable. An intelligence officer will be asked to assess many situations over his career.

RPLong writes:

I'm with Bedarz. "70%" sounds like a big number, lots of confidence. An officer may analyze many situations, but unless that officer conducts a perfect Bayesian analysis of each of those situations and feeds them into a formal, all-encompassing model stated to address precisely the historical question we're talking about, it's no better than arbitrarily subjective verbiage like "I think I'm pretty sure, but maybe not."

Brad writes:

Well if you give odds you could do something like this and test how well you are doing.

http://slatestarcodex.com/2015/01/01/2014-predictions-calibration-results/

Nathan Smith writes:

I'd be interested in knowing what the argument would be that invading Iraq in 2003 would have been worth it if the odds that Saddam was pursuing WMDs were 100%, but not if they were 60-70%. I find the claim odd.

My support for the overthrow of Saddam had, and has, little or nothing to do with WMDs. My primary reason for supporting the war was to overthrow a horrible totalitarian regime. My secondary, more tentative reason (at least in retrospect) that it may have helped the war on terror by sucking al-Qaeda into a bloody quagmire where they had to ruin their reputation in the Muslim world by killing lots of Muslims. (That argument may be too morally repugnant to deserve support even if it's valid in a Machiavellian realpolitik sense, so I'm doubly ambivalent about it.) If there were no objection to Saddam except that he was probably-- or even certainly-- making WMDs, I'd have opposed the war. That's why I favor the recent Iran deal.

But I can see why someone might think it's intolerable for Saddam to have WMDs. I remember that it was sometimes argued at the time that Saddam would never use al-Qaeda to smuggle WMDs to attack the US, and the obvious silliness of this argument (can't people make tactical alliances with ideological foes? what about the Hitler-Stalin alliance of 1939? or for that matter the alliance of the US and UK with Stalin's USSR against Hitler?) serves as a helpful reminder of how the hawks' case got underplayed by a dovish press, while dovish arguments tended to get a lot more respect than their inherent logic merited. Anyway, suppose you're worried about Saddam getting nukes and passing them on to al-Qaeda and you think that absolutely has to be prevented. Obviously it wouldn't matter that much if the odds that Saddam is pursuing WMDs gets downgraded from 100% to 60-70%.

That doesn't absolve the IC for overstating the strength of the evidence. But I think it does leave the suggestion that uncertainty supports the case for pacifism rather high and dry. If it's not irrational to buy insurance, it's not irrational (though perhaps it's unjust) to launch pre-emptive wars against regimes that will, with non-negligible probability, become existential threats.

wd40 writes:

This thread as well as the arguments made at the time by the Bush administration displays a woeful ignorance of option theory. At some cost, the administration could have waited until they gathered more information that confirmed (or disconfirmed) their belief that Saddam had weapons of mass destruction. Hans Blix, the UN weapon's inspector, argued for more time, but the Bush administration, despite the inability of the weapons inspectors to find such weapons based on information the US provided, refused to wait.

Andy Hallman writes:

@Nathan Smith

My secondary, more tentative reason (at least in retrospect) that it may have helped the war on terror by sucking al-Qaeda into a bloody quagmire where they had to ruin their reputation in the Muslim world by killing lots of Muslims.

I'm having trouble following this part. You think that starting a war is good because all the killing will lead to a terrorist organization killing even more people and this will, in the long run, save so many people it will justify all the killing? Talk about a leap of faith, Nathan!

Why is it intolerable for Saddam to have WMDs? Far worse rulers had WMDs and did not use them such as Stalin and Mao.

Hussein and al-Qaeda are not just ideological foes with a different idea of the ideal society. They were very much rivals. Even among allies, do they just sell each other nuclear weapons? No. You brought up the US and UK, so it is worth remembering that even among those close allies, the US did not trust the UK with nuclear information after WWII. If even those two countries with a "special relationship" don't trust each other, why on earth would Saddam trust al-Qaeda with a nuclear weapon?

...serves as a helpful reminder of how the hawks' case got underplayed by a dovish press.

Nathan, I really like you and your usually insightful comments, but sometimes I wonder if we're on the same planet. The press was so dovish that MSNBC fired TV show host Phil Donahue in the lead up to the invasion because he was antiwar. A leaked memo from NBC indicated the network's executives considered him a "difficult public face for NBC in a time of war."

This is a good place to start: https://en.wikipedia.org/wiki/Media_coverage_of_the_Iraq_War

Anyway, suppose you're worried about Saddam getting nukes and passing them on to al-Qaeda and you think that absolutely has to be prevented. Obviously it wouldn't matter that much if the odds that Saddam is pursuing WMDs gets downgraded from 100% to 60-70%.

What is the chance that he uses them even if he gets them? Even if the chances of him using them are high, aren't they higher if he were invaded than left alone? That's what I don't understand about the pro-war argument. It's hard to imagine a scenario that justifies invasion.

But I think it does leave the suggestion that uncertainty supports the case for pacifism rather high and dry.

Uncertainty supports whatever side does NOT favor huge upfront costs. The side favoring huge upfront costs is the side advocating lots of killing to start with and then hope for the best.

James writes:

The reference to poker players is interesting. David Sklansky is a somewhat respected writer on poker. He has coined what he called the "fundamental theorem of poker" which basically says good playing is when you do what you would do if you knew exactly what cards everyone was holding. Of course this is wrong. Raising when all you have is a pair of threes is not good play, although it's what you would do if you knew your opponents had even weaker hands.

Comments for this entry have been closed
Return to top