Scott Sumner  

Extreme outliers: How meaningful are they?

Buyer Licencing of Illegal Dru... Take That, Madame Secretary...

This post is about the way I think about extreme outliers. It's very unscientific, but I hope the comment section will help me to better understand this issue.

Suppose you have two variables, X and Y, which are (supposedly) positively correlated. But the very highest value of X is associated with very the lowest value of Y. Or assume the two variables are supposed to be negatively correlated, but the highest value of X corresponds to the highest value of Y. Should that make us suspicious of the alleged relationship?

I was thinking about this a couple years ago when reading a paper on the impact of Smoot-Hawley on stock prices (unfortunately I've forgotten the author). The paper claimed that news making Smoot-Hawley more likely to pass tended to raise stock prices in both 1929 and 1930. My own research reached the same conclusion for 1929, but the opposite conclusion for 1930. Unfortunately, my own research was less systematic, mostly based on reading the NYT and observing that stocks seemed to fall on news that Smoot-Hawley was moving closer to passage in the spring of 1930. So that suggests the other study should be taken more seriously.

But there's just one data point I just can't get past---Hoover's decision to sign the bill, (which was made on a Sunday). Hoover had been under heavy pressure to veto the bill, from US exporters, from foreigners, and from a letter signed by over 1000 economists. He rejected that advice, and the next day the New York market saw the biggest percentage decline in all of 1930. (About 300 trading days--as Saturday was still a workday back then). Even worse, the news media seemed to attribute the crash to the decision by Hoover to sign the bill, quoting stock traders who believed the same.

That's just one data point, but it's hard for me to get past that one data point. How could something that was supposedly boosting stock prices, and was by far the major news story, be associated with the biggest stock market crash of the year, immediately after the decision was actually made? It's possible, but it seems really unlikely.

Suppose you believed that direct democracy led to bad political outcomes, because philosopher kings were much better than mob rule. In that case, how likely it is that the one country with by far the most direct democracy in the entire world, would also be arguably the best governed in Europe, and perhaps the world? Possible, but how likely?

Screen Shot 2016-04-02 at 2.50.53 PM.png
Suppose you noticed that America scores higher on happiness rankings than does Europe, on average. So you developed a hypothesis that social welfare states are less happy. How likely would be that a country which by some measures has the world's most generous social insurance system, is also the world's happiest country?

Screen Shot 2016-04-02 at 2.49.47 PM.png
And while we are on the subject, suppose you thought deregulation and privatization made people unhappy. How likely would it be for the world's most free market economy (excluding level of taxation and government spending) to also be the happiest?

Suppose you thought that the East Asia tiger economies were successful because they rejected the neoliberal agenda coming out of Washington, and instead had state directed development strategies. If that were true, how likely is it that the two very richest East Asian economies would also be number one and two in the world in the Heritage Ranking of Economic Freedom?

Suppose you believed that monetary policy was ineffective at boosting NGDP at the zero bound. In that case, how likely is it that the fastest 4 month stretch of NGDP growth in American history would occur during a period of near-zero interest rates, right after a easily identifiable monetary shock (March-July, 1933).

And while we are at it, suppose you believed that the credit channel explains why growth is slow during and after a banking crisis. How likely is it that the fastest stretch of NGDP growth would occur during a period right after America's worst banking crisis, and during a period when 1000s of banks were still closed down? Again, not just growth during the financial crisis, but perhaps the fastest NGDP growth ever, during arguably the worst banking crisis ever.

Suppose you thought that inflation was caused by bottlenecks in the economy, and deflation was caused by slack. How likely is it that the price level (WPI) would rise by 20% during a period of 25% unemployment (1933-34)?

And speaking of the zero bound, just how likely is it that the biggest two day stock rally in US history would (just randomly) occur immediately after Hoover announced a proposal to allow the Fed to print more money, for each ounce of gold backing.

Suppose you thought that Mexican-Americans had a propensity to rape and murder. (Hmm, where have we heard that theory?) How likely is it that America's most Mexican major city (of the top fifty) would also have the lowest murder rate, and perhaps the lowest violent crime rate?

Here's how I look at it, and I want you to tell me why I'm wrong. If you have only a few observations, then extreme outliers are no big deal---but your study is also not very reliable. If your study includes a large number of observations, the odds of the most extreme value of X and Y being correlated in the opposite direction from the actual relationship seems very low. Am I too suspicious of extreme outliers? What do you think?

PS. You should be suspicious of my "largest two day rally" which sounds like data mining, and a violation of the EMH. But there really was news on the second day, as Congressional leaders agreed to speed the bill through Congress.

Comments and Sharing

CATEGORIES: Economic Methods

COMMENTS (18 to date)
entirelyuseless writes:

I think you are too suspicious of outliers. Think of this situation: being subjectively more sure of things is correlated with being objectively right.

This is a real correlation. But take the few beliefs that a random person is most subjectively sure about: those might well be false beliefs. The reason is that although being right is a cause of subjective certainty, the cases of greatest subjective certainty are the cases where all the other causes of subjective certainty line up, regardless of whether the person is objectively right or not.

Matthew Moore writes:

Extreme outliers usually mean to me that there is an unmodelled third factor that mediates the relationship between the two. If you are lucky enough to get that third factor, then the usual trade-off vanishes, or is much reduced.

I notice that most of your examples are outliers that are bilaterally positive - we got two good outcomes where we usually only get one. If you get the third factor, you can move to the extreme ends of both variables.

There are of course many other cases where you get two bad outcomes together - usually in this case, the third factor is war.

Now, this is dangerous thinking. In some ways, it is tautological. Something must explain the pattern. But equally, it means that any unique characteristics of the outlier can be pointed at to explain the problem and retain the hypothesised trade-off or relationship. And we often see this fallacy. Company X managed to get both A and B - therefore let us all imitate Company X.

But generally, I think there is always some cultural solution to any incentive problem. When the outliers are nations, I look for a cultural explanation.

jc writes:

Yep, assuming your data is good, something is missing from your model.

Missing mediator (per Matthew). Or moderator. And it just happened to prominently show up in the place where X and Y were the most extreme.

Or maybe this extremeness is fundamental, e.g., some sort of extreme threshold effect that eventually renders the relationship sharply curvilinear.

So yeah, it should give you pause. But I think I'd trust the general trend a bit more. Something like: "X usually leads to Y. But when this Mystery Factor is present, it's a whole new ballgame. Fortunately, this Mystery Factor seems to rarely occur."

Lliam writes:

Wouldn't that be reasonably testable? Create a set of a few thousand pairs of numbers with a programmed correlation (across the set) and look for how often the largest values are correlated opposite to what's expected?

Sieben writes:

Suppose you thought that exercising gave you more energy. How likely is it that the times you feel MOST exhausted are after a vigorous workout? Pretty likely, since that's how workouts can be.

Suppose you thought that going to college was a smart financial move. How likely is it that the poorest you'll EVER BE is in college? Pretty likely, since you're in debt and don't have a job.

Suppose you believe you found your true love. How likely is it that you'll experience the WORST day of your life while you're married to them? Pretty likely, since you'll be with them a long time.

What is the probability that the countries that do the best along some metric will be outliers in all sorts of ways?

Snark aside, I deny we have a reasonable sample size for any of this. There are just barely over 100 "democracies" in the world, and most of those countries are backwards culturally and probably shouldn't count as democracy. The total number of countries that have tried serious social systems is extremely low.

Scott Sumner writes:

Entirely, I'm a Rortian, so I don't accept the distinction between subjective belief and objective truth as being meaningful.

Everyone, In some cases there's not enough data, or perhaps the natural experiments are not well defined. In those cases the outlier makes me very suspicious. But yes, if you have the ability to scale up the test with more and more data, and add more controlling variables, then by all means do so.

Sieben, I don't quite follow your examples. Does anyone believe that college makes people rich the moment they graduate? The examples you provide don't address the questions you are asking.

bill writes:

I once listened to a debate on evolution vs. intelligent design. The moderator asked the scientist what it would take for him to reject the theory of evolution. Answer: just one fact or observation that contradicted it.

AbsoluteZero writes:

As others mentioned it's often simply an indication the model or theory is incomplete. But we all know incomplete or even incorrect models can be useful.

The definition of "extreme" is important. Without a definition it's not clear why a particularly data point is regarded as an extreme outlier. Often the definition is simply that it doesn't fit the model. In that sense extreme outliers tell us about the boundaries of the model or theory. This is common in many fields, physics and electronics are obvious examples.

Noah writes:

There's an exception to every rule?

Benjamin Cole writes:

Great post. Covered the waterfront as the old-timers used to say.

El Paso? Who knew? Or NYC No. 2 safest?

East Asian tigers? Do city states count?

A question: in an ideal world, would there be nothing except open city states?

J Storrs Hall writes:

Remember that if you have assumed a linear relationship, and the actual one is nonlinear, points farthest from your central cluster will typically also be farthest from your regression line.

Scott Sumner writes:

Absolutezero, I meant an outlier where the most extreme value of one variable corresponded to the most extreme value of the other variable, but in the opposite direction from what the model predicts.

AbsoluteZero writes:

I see, only in this sense, and extreme is simply the largest or smallest within the range of data already seen. In that case, assuming the data is valid, it shows the model cannot be complete. Usually one starts looking for one or more variables not taken into account by the model. At least that's how it always happens in physics and EE, etc.

emerich writes:

Commenters seem to have missed what looks to me to be the main point of the post: The premises in the suppositions are deliberately dubious, and the "extreme" examples aren't necessarily outliers. For example, Scott, I assume you do not believe that East Asian economies are successful to the extent that they rejected the neoliberal agenda, though some Asian leaders (e.g., Lee Kuan Yew) justified authoritarian policies on that premise. And maybe someone believes that direct democracy is the main source of economic and political dysfunction, but I'm not sure who. (Institutions and culture play lead roles in many accounts I've read.) Etc.

Neil writes:

I used to read Jude Wanniski's blog before he died. I remember he had a post in which he showed news indicating passage of Smoot-Hawley was more likely to pass led to drops in the Dow and vicaversa. He was convinced that the tariff act directly lead to the stock market crash.

Scott, if Jude had the correlation backwards (at least for 1929), then your theory would be that a monetary shock is what lead to the stock market crash?

Scott Sumner writes:

Neil, Yes, I think he got things backward for 1929. I believe the causes of the crash were, in order of importance:

1. Unexplained
2. Tighter money
3. GOP incompetence in trying to pass Smoot-Hawley
4. Political conflict between Germany and France expected to increase.
5. Tighter anti-trust enforcement announced

Joe writes:

The problem is that a lot of your "suppose" statements don't fit the model of regression pointing one way, outlier pointing the other. I would think that direct democracy correlates to good political outcomes as a whole, so the best-governed country being the one with the most direct democracy doesn't surprise me. The supposition is simply false.

The same is true for state-led capitalism leading to higher wealth. If you graphed state intervention vs. wealth, you'd likely get less intervention = more wealth. Now, if the wealthiest country on Earth was North Korea, we'd have the sort of problem you describe.

Your best example is your initial one. Talk of passage of Hawley Smoot correlates to higher stock prices EXCEPT when the bill actually passes. Then it appears to cause a crash. Here the outlier calls the whole model into question. Again, it's hard because the relationship of news > markets is based on fallible human impressions of what might happen.

Maybe there are two models at hand. 1) news of government action vs. stock prices and 2) passage of actual tariffs vs. stock prices.

Government intervention in health care vs. cost as a % of GDP is probably another good example. I would bet that more market mechanisms = lower cost and higher quality health care. The United States would be the outlier on the graph because we combine profit-seeking companies with socialist payment mechanisms.

Jeffrey S. writes:

Generally it is a good idea with outliers, especially if they don't fit in with the rest of your data, to ask why that is...and there are usually good reasons for why they are outliers:

In other words, Hispanics are still more criminal than whites and less criminal than blacks, even if one or two upscale cities on the border, with lots of first generation immigrants (the real problems start in on the second generation forward) are well policed and safe:

Comments for this entry have been closed
Return to top