Full Site Articles EconLog EconTalk Books Encyclopedia Guides

# Scott Alexander's Self-Evaluations

 Henderson and Hooper on Flawed... A little bit of knowledge is a...
I just learned that blogger Scott Alexander starts ends the year by scoring the accuracy of his numerous annual predictions: here are the results for 2014, 2015, and 2016.  Since he assigns probabilities to each prediction, Scott can graph his ex ante subjective probabilities against the ex post objective probabilities.  Here's his graph for 2016.  The red line shows perfect calibration; the blue line shows Scott's actual calibration.
Contrary to Tyler, this is not "just another kind of religion," but this intensity of scrupulous rationality does approach the divine.  Or as a wise man once said, "People who say it cannot be done shouldn't interrupt those who are doing it."

richard writes:

It looks like he's just learning to make fewer predictions, which might be a kind of wisdom

"Of 50% predictions, I got 8 right and 5 wrong, for a score of 62%
Of 60% predictions, I got 12 right and 9 wrong, for a score of 57%
Of 70% predictions, I got 13 right and 3 wrong, for a score of 81%
Of 80% predictions, I got 13 right and 3 wrong, for a score of 81%
Of 90% predictions, I got 16 right and 1 wrong, for a score of 94%
For 95% predictions, I got 9 right and 0 wrong, for a score of 100%
For 99% predictions, I got 3 right and 0 wrong, for a score of 100%"

brokilo writes:

@richard

2014: 59 predictions
2015: 35 predictions
2016: 99 predictions

You mean less >90% confident predictions?

robc writes:

It looks like he is underestimating the probabilities.

With the exception of the 60% category, he overshot his category mark each time.

Which means he is either bad at predicting probabilities or he was sandbagging. Or the sample size was just small.

brokilo writes:

@robc

He's not perfect at predicting probabilities. The whole point is that he calibrates himself as time goes on.

He reviews how much he over/undershot his probabilities and factors that into next years predictions. Looks like he needs to mark his confidence up by about 5-10 percent across the board this year.

I doubt I could do much better if I measured myself in this fashion.

writes:

I made a similar set of predictions for 2016. Turns out I was pretty overconfident.

Michael Keenan writes:

Richard, I think you might have misinterpreted it; he's increased the number of predictions he made over time, not decreased.

He made 59 predictions in 2014, 55 in 2015, 99 in 2016, and (though Bryan didn't link to it), 105 in 2017.

Ryan Murphy writes:

Ye whose purported subjective probabilities are poorly calibrated, confess to the community at the Towne Square!!

Richard writes:

My mistake, I was skimming, thanks for pointing it out.

writes:

Another way to look at it is that he is now only making predictions about things he is sure of, regardless of what "subjective probability" he happens to attach to the prediction.

That would be a special version of robc's "sandbagging" comment, but in this version of it, it's impossible to self-detect because it would certainly be a subconscious act.

This brings me to the whole problem with the "rationalist community." We presume that it's "rational" to build a predictive model in anticipation of a particular outcome. But decisions have to be made about what goes in the model and what doesn't. Ie., it's subjectivity dressed up in the language of objectivity. Relevant variables might be omitted from the model for erroneous, subjective reasons. Irrelevant variables might be included in the model for erroneous, subjective reasons.

To my knowledge, rationalists aren't doing single-variable analysis to determine which of the included variables really are statistically relevant. Or, when they are, their subconscious mind makes sure to pad the model with various other factors that are sure to make the important variables look better and the offensive variables look worse.

They call this rationalism, but it's just a mental game we play to make our subjective beliefs look more credible.

writes:

Do a controlled study with more than one individual and I'll pay attention.

Comments for this entry have been closed