Interesting email from Todd Proebsting, reprinted with his permission.
Dear Prof. Caplan,
Your recent attempt at a wager on Iran's nuclear progress,
and the subsequent exchange with the author about why you wanted 10:1 odds
reminded me that the CIA tried to quantify what certain claims meant:
100% Certainty

The General Area of Possibility

93%

give or take about 6%

Almost certain

75%

give or take about 12%

Probable

50%

give or take about 10%

Chances about even

30%

give or take about 10%

Probably not

7%

give or take about 5%

Almost certainly not

0% Impossibility

It's from a 50year old CIA proposal.
It's interesting to me that "Almost certain" was
only 93%. I think the CIA would have argued that you should have been asking
for even longer odds given the original claim.
Cheers, Todd
Proebsting
Bizarre that the language is asymmetric about 50%. Is there some prospecttheory justification for that?
Or there's the IPCC (TS, box TS1) typology:
Virtually certain: 99–100% probability
Very likely: 90–100% probability
Likely: 66–100% probability
About as likely as not: 33–66% probability
Unlikely: 0–33% probability
Very unlikely: 0–10% probability
Exceptionally unlikely: 0–1% probability
I remember having a long debate with my wife about this early in our marriage.
It took a while to understand that we weren't using the terms the same way:
"Absolutely certain" me: 90% her: 100%
"Very certain" me: 80% her: 100%
"Pretty certain" me: 70% her: 98%
"Fairly certain" me: 60% her: 95%
"I think so" me: 25% her: 90%
Eventually, I just started using actually percentages. As for her, if she's willing to venture an opinion, then she's right (with exceptions countable on one hand in 25 years).
It's interesting to me that the CIA define "Probable" as approx. 75%, but "Probably Not" as approx. 30%.
The definitions are not balanced between events which are likely and other events which are equally unlikely.
There are also a few grey areas, most notably events with probability of approx. 12% to 20% have no suitable description under this proposal.
There appears to be a considerable amount of discretion involved.
Examples:
1. Using these definitions an event with probability 39% would be described as "Probably Not", but an event with probability 61% wouldn't be described as "Probable" (it's in a grey area between "Chances about even" and "Probable").
2. An event with 85% probability would be described as "Probable", but one with a 15% probability would fall in the large grey area between "Almost Certainly Not" and "Probably Not".
Adam: At least the IPCC definitions are symmetrical about 50%. I don't much care for their incredibly wide "About as likely as not band" which ranges from 33% to 66% though. 66% is about twice as likely as not.
I think that it depends on what you're talking about. If I say "I'm absolutely sure that she has a crush on you," it probably means about 90%, since I think that's about the maximum level of certainty you can have about something like that. If I say "I'm absolutely certain that the sun will come up tomorrow," it's more like 99.999999999%, since we can be that certain about physical phenomena.
Like Al, I have a problem with the lack of symmetry between probably and probably not. It's like nails on a chalkboard.
It sounds like Watson would be answering only when almost certain.
Can someone refer me to a pithy essay on how one concludes, for example, that a given outcome is, say, 63%87% likely (Probable)?
Pretending that you can estimate a numerical probability for future events that will either occur or not occur once is at best a rhetorical device and at worst gross selfdelusionary scientism. You can estimate odds for a bet or weigh whether given odds are favorable or unfavorable given your present knowledge and preferences. But such polls will vary from person to person; otherwise you would take the other side of bets? Frequentist considerations aside, how can the truth or falsity of such statements even be proven?
If I say there is a 83% chance that some event will occur tomorrow and it happens I say see! If it doesn't happen then we just experienced a 17% probability event.
Tax opinions have similar methods of converting confidence levels expressed as phrases into numerical percentages, and here is one common interpretation:
These aren't exact, and the percentage level assigned will vary across situations and has varied over time. Some practitioner interpretations treat "reasonable basis" as 1 in 3 while reducing "should" to 60% chance.The CIA, where even the error bars have error bars.
(e.g. give or take about 10%)