Bryan Caplan  

Converting Ordinary Language to Probabilities: CIA Edition

China: The real problem is nom... The Huge Benefits of the Shari...
Interesting email from Todd Proebsting, reprinted with his permission.

Dear Prof. Caplan,

Your recent attempt at a wager on Iran's nuclear progress, and the subsequent exchange with the author about why you wanted 10:1 odds reminded me that the CIA tried to quantify what certain claims meant:

100% Certainty

The General Area of Possibility


give or take about 6%

Almost certain


give or take about 12%



give or take about 10%

Chances about even


give or take about 10%

Probably not


give or take about 5%

Almost certainly not

0% Impossibility


It's from a 50-year old CIA proposal.


It's interesting to me that "Almost certain" was only 93%.  I think the CIA would have argued that you should have been asking for even longer odds given the original claim.

Cheers, Todd Proebsting

Comments and Sharing

COMMENTS (11 to date)
Sam writes:

Bizarre that the language is asymmetric about 50%. Is there some prospect-theory justification for that?

Adam writes:

Or there's the IPCC (TS, box TS1) typology:

Virtually certain: 99–100% probability
Very likely: 90–100% probability
Likely: 66–100% probability
About as likely as not: 33–66% probability
Unlikely: 0–33% probability
Very unlikely: 0–10% probability
Exceptionally unlikely: 0–1% probability

Tom West writes:

I remember having a long debate with my wife about this early in our marriage.

It took a while to understand that we weren't using the terms the same way:

"Absolutely certain" me: 90% her: 100%
"Very certain" me: 80% her: 100%
"Pretty certain" me: 70% her: 98%
"Fairly certain" me: 60% her: 95%
"I think so" me: 25% her: 90%

Eventually, I just started using actually percentages. As for her, if she's willing to venture an opinion, then she's right (with exceptions countable on one hand in 25 years).

Al writes:

It's interesting to me that the CIA define "Probable" as approx. 75%, but "Probably Not" as approx. 30%.

The definitions are not balanced between events which are likely and other events which are equally unlikely.

There are also a few grey areas, most notably events with probability of approx. 12% to 20% have no suitable description under this proposal.

There appears to be a considerable amount of discretion involved.


1. Using these definitions an event with probability 39% would be described as "Probably Not", but an event with probability 61% wouldn't be described as "Probable" (it's in a grey area between "Chances about even" and "Probable").

2. An event with 85% probability would be described as "Probable", but one with a 15% probability would fall in the large grey area between "Almost Certainly Not" and "Probably Not".

Al writes:

Adam: At least the IPCC definitions are symmetrical about 50%. I don't much care for their incredibly wide "About as likely as not band" which ranges from 33% to 66% though. 66% is about twice as likely as not.

Richard writes:

I think that it depends on what you're talking about. If I say "I'm absolutely sure that she has a crush on you," it probably means about 90%, since I think that's about the maximum level of certainty you can have about something like that. If I say "I'm absolutely certain that the sun will come up tomorrow," it's more like 99.999999999%, since we can be that certain about physical phenomena.

Ken P writes:

Like Al, I have a problem with the lack of symmetry between probably and probably not. It's like nails on a chalkboard.

It sounds like Watson would be answering only when almost certain.

Dick White writes:

Can someone refer me to a pithy essay on how one concludes, for example, that a given outcome is, say, 63%-87% likely (Probable)?

D. F. Linton writes:

Pretending that you can estimate a numerical probability for future events that will either occur or not occur once is at best a rhetorical device and at worst gross self-delusionary scientism. You can estimate odds for a bet or weigh whether given odds are favorable or unfavorable given your present knowledge and preferences. But such polls will vary from person to person; otherwise you would take the other side of bets? Frequentist considerations aside, how can the truth or falsity of such statements even be proven?
If I say there is a 83% chance that some event will occur tomorrow and it happens I say see! If it doesn't happen then we just experienced a 17% probability event.

_NL writes:

Tax opinions have similar methods of converting confidence levels expressed as phrases into numerical percentages, and here is one common interpretation:

“Will” opinion: 90-95% probability of position being sustained.

“Should” opinion: 70% to 75% probability of position being sustained.

“More likely than not” opinion: More than 50% probability of position being sustained.

“Substantial authority” opinion: Less than “more likely than not” but greater than “reasonable basis” (34% to 40% likelihood)

Realistic possibility of being sustained on its merits: One-in-three possibility (33%)

“Reasonable basis” opinion: Higher than “not frivolous” but lower than “substantial authority” (20-30%)

Not Frivolous: Lowest level at which there is some modicum of comfort as to a position.

These aren't exact, and the percentage level assigned will vary across situations and has varied over time. Some practitioner interpretations treat "reasonable basis" as 1 in 3 while reducing "should" to 60% chance.
Joe Topp writes:

The CIA, where even the error bars have error bars.

(e.g. give or take about 10%)

Comments for this entry have been closed
Return to top