Bryan Caplan  

Tristan Caplan's Tetlockian Glossary

The root causes of natural dis... Paul Pillar on Iranian Sanctio...
This summer, I assigned Tetlock and Gardner's Superforecasting (see here, here, and here) to my homeschoolers.  They took to this masterpiece of political psychology like fish to water.  I'd already taught them about human beings' wacky mapping from language to quantitative probabilities, but these passages have really stuck with them:
In March 1951 National Intelligence Estimate (NIE) 29-51 was published.  "Although it is impossible to determine which course of action the Kremlin is likely to adopt," the report concluded, "we believe that the extent of [Eastern European] military and propaganda preparations indicate that an attack on Yugoslavia in 1951 should be considered a serious possibility." ...But a few days later, [Sherman] Kent was chatting with a senior State Department official who casually asked, "By the way, what did you people mean by the expression 'serious possibility'?  What kind of odds did you have in mind?"  Kent said he was pessimistic.  He felt the odds were about 65 to 35 in favor of an attack.  The official was started.  He and his colleagues had taken "serious possibility" to mean much lower odds.

Disturbed, Kent went back to his team.  They had all agreed to use "serious possibility" in the NIE so Kent asked each person, in turn, what he thought it meant.  One analyst said it meant odds of about 80 to 20, or four times more likely than not that there would be an invasion.  Another thought it meant odds of 20 to 80 - exactly the opposite.  Other answers were scattered between these extremes.  Kent was floored.
Ideally, analysts would adopt standard conventions on the correct way to translate from English to math.  But since that won't happen anytime soon, my son Tristan proposed a simple, effective half-measure.  Namely: every serious book (and perhaps every article) should include a Probability Glossary in the opening pages, alongside the List of Illustrations and List of Tables.  In these glossaries, each author would explicitly state what probabilities (or probability ranges) he assigns to probability-relevant English words.  For example, here are roughly the probabilities I have in mind when writing this blog:

English Term

My Probability

Absolutely certain




Almost Certain


Highly Likely












Highly Unlikely


Almost impossible




Absolutely impossible


Filling out this table makes me self-conscious of the inadequacies of my current usage.  But it's still a big step forward in the War for Clarity.  Indeed, reading this table might make you realize you and I suffer from illusory disagreement - or illusory agreement.  And even if publishers are loathe to add a Probability Glossary, there's no reason why every thinker who wants to signal seriousness couldn't publicly post one on his blog or webpage.

When you do, please credit my son for the idea...

Comments and Sharing

COMMENTS (19 to date)
Effem writes:

What about "beyond a reasonable doubt?"

Matthew Moore writes:

Interesting. I wonder how culture mediates. I would never use likely and unlikely in this way. For me, they mean >80% and <20%. Maybe this is British English. What you call 'likely'
I call 'more likely than not'.

Otherwise I agree, except 1 in 20 doesn't seem almost impossible to me. Maybe 1 in 1000, or greater. For example: 'it's almost impossible you will die during this surgery'

[Greater than and less than signs converted to HTML entities. You cannot use the keyboard greater than or less than signs.--Econlib Ed.]

Matthew Moore writes:

EDIT: * "For me they mean >80% and

@Effem: this was raised in a recent high-profile case in England. The judge said he was unable to give guidance. For me "reasonable doubt" is a far higher standard than that usually actually imposed de facto in the courtroom. I suspect most jurors use a "more likely than not" standard.

Q&A with judge:
Q4. Can you define what is reasonable doubt?
Answer: “The prosecution must make you feel sure beyond reasonable doubt. A reasonable doubt is a doubt that is reasonable. These are ordinary English words that the law does not allow me to help you with, beyond the written directions [he had already given them]”.


David R. Henderson writes:

Nicely done. Compliments to Tristan.
My one edit would be to make unlikely below 45%.

Floccina writes:

The rule should be always put your numbers in.

John Alcorn writes:

@ Bryan Caplan:

Compliments to Tristan!

C. Behan McCullagh included such a glossary in his excellent book, Justifying Historical Descriptions (Cambridge U. Press, 1984), at page 52:

extremely probable = 100-95%
very probable = 95-80%
quite or fairly probable = 80-65%
more probable than not = 65-50%
hardly or scarcely probable = 50-35%
fairly improbable = 35-20%
very improbable = 20-5%
extremely improbable = 5-0%

According to Prof. McCullagh, most people would use grades roughly like these. Over the years, I have found them a helpful discipline for thinking about probabilities, and about how to express them consistently.

Your table is somewhat more fine-grained, in useful ways.

BTW, in my comment at your blogpost, Four Decades of Middle Eastern Disaster: The Proximate Cause, I considered including mathematical probabilities in my hypothetical prediction, but decided that to do so would be overkill. Tristan reminds me that I should have included them. Here is what I had in mind:

• The Shah's erratic response probably will embolden revolutionaries to try and seize power: p = .7
• The attempt at revolution might or might not succeed: p = .5
• A successful revolution almost certainly will cause great disequilibrium in the Middle East: p = .95
• Therefore, given the Shah's behavior, the overall probability of great, protracted disequilibrium in the Middle East is: p = 0.33

John Alcorn writes:

@ Matthew Moore:

Thank you for sharing the quotation and item about "reasonable doubt."

I recall reading somewhere that "reasonable doubt" at trial is not a matter of probability. Rather, there is reasonable doubt if the prosecutor cannot provide a satisfactory answer to a relevant argument by the defense. The thought is that the prosecution must effectively neutralize every substantial point made by the defense.

Perhaps it was in L. Jonathan Cohen's stimulating book, The Probable and the Provable (Oxford U. Press, 1977).

gwern writes:

The intelligence agencies, the CIA specifically, already have a standardized set of subjective probability terms: the Kesselman List of Estimative Words

(Do Tetlock & Gardner not mention that at all? It's usually mentioned after recounting those anecdotes about people realizing how much they numerically disagree when using the same vague descriptions.)

E. Harding writes:

In U.S. political terminology, races are classified as toss-up, lean, likely, and safe. By my interpretation, toss-up is less than a 65% chance of a race going either way, Lean is between and 65% and 90%, Likely is between 90% and 98.5%, while Safe is 98.5% or above. Of course, others may use the terms differently from how I do.

David O'Brien writes:

Similar to the way the Chili scale is arbitrary between two different restaurants, with food on one menu with 2 chili's beside it is the same as another that indicates it as 3 or 5 chilis. How we discuss probability sometimes seems arbitrary & subjective.
You have devised the Scoville scale (measurement of the pungency (spicy heat) of chili peppers) of Probability.
I'd like to see this adopted by an international body such as the
Thank you

Bedarz Iliaci writes:

I suspect that the numerical values provide an illusory certainty and in fact, the common English terms are more precise.
What exactly one would mean by 65 percent chance of an event happening? And how is this degree of quantification made? Is there an algorithm that yields these numbers? I suppose Bayes'theorem.
But then you have the problem of justifying the quantities assigned to the priors.

Tracy W writes:
What exactly one would mean by 65 percent chance of an event happening?

That, if one were to give 1000 events each a 65% chance of happening, one would expect after the fact to find that about 650 of those events had gone that way and 35% not.

Eg: imagine meteorologists predict a 65% chance of rain over the course of a day. If I collected data on 1000 days where meteorologists predicted rain and found that on 648 days it rained and on 352 days it was fine all day, then it looks like meteorologists are fairly well calibrated.

But then you have the problem of justifying the quantities assigned to the priors.

Well if over 1000 cases of predictions of 65% chance of rain it turns out that it rained 648 days then we can tentatively conclude that your priors are quite good.

Steve Bacharach writes:

Your kid sounds great, but he's not exactly breaking new ground here:

David Shera writes:

Statistician Fred Mosteller did a paper/study on this topic. Some takeaways:

Probably is about 2/3 chance
possible is about 1/3 chance

modifier words like "very" tend to push higher.

So when President Ford talked about the Swine Flu, there was a "very real possibility", translates to about 50%, which just like saying "I don't know."

Same when you hear people say "cautiously optimistic". Translation: I don't know.

Matt C. writes:
Effem writes: What about "beyond a reasonable doubt?"

"Beyond a reasonable doubt" means "nearly 100% certainty when all doubts that are reasonable to a case such as this have been removed."

blink writes:

You set a great example here, Bryan.

An interesting point is the asymmetry: "highly likely" and "highly unlikely are not quite complements, with the "unlikely" side reflecting greater certainty, and a similar case with "possible" and "probable."

Do you see this as reflecting typical usage, a quirk of language, or your own idiosyncratic approach?

Bedarz Iliaci writes:

Tracy W,

My question was to the probability of an event and you answered with events.

Consider the example Caplan gives:

... an attack on Yugoslavia in 1951 should be considered a serious possibility.

This talks about a specific event that may come t happen--an attack on Yugoslavia in 1951. Now what could possibly be the events that are needed to make sense of the probability definition according to you?

Khanin writes:

Would you say that it is "almost impossible" to roll two sixes?

Patrick Haugey writes:

This is interesting. My brother is a detective with a fugitive squad. He was tracking a murderer. When he found him, he called his sergeant who asked him for how certain he was. He is not allowed to use 100% for liability reasons, in case of mistakes So he said 97. That wasn't good enough to send more men. So he said 98.9. I think reason and clarity for probability such as posted above s only useful in a vacuum. The law makes certainty more variable.

Comments for this entry have been closed
Return to top