Deadly Chicago Econometrics

Go Arnold! Thanks for clearing up this issue.

BTW, what do you mean by "classical" statistics? Is there some other form of analysis for evaluating the significance of results?

Chris: The alternative to Classical Statistics is the (unfortunately initialed) Bayesian Statistics, which tries to attach probabilities to relationships given the data (e.g. "given the data, what are the odds that there's no relationship between healthcare and health?"), as opposed to the classical approach which focuses on the probablility of observing particular data given a certain relationship between the variables. (E.g. "what are the odds that we'd see this data if there were no relationship between healthcare and health?") At least to my mind the Bayesian approach makes a lot more sense, and is much easier to work into standard decision theory, but is often tougher to implement.

Arnold: Is your beef with estimating aggregate relationships per se, or just with those who estimate only aggregate relationships and don't dig any deeper? The fact (assuming hypothetically that that the claims were sound) that there's no aggregate medicine-health relationship, but that there are a number of positive relationships at the disaggregated level would suggest exactly the sort of process Bryan posits. That seems to me to be an interesting finding...

The fact (assuming hypothetically that that the claims were sound) that there's no aggregate medicine-health relationship, but that there are a number of positive relationships at the disaggregated level would suggest exactly the sort of process Bryan posits.

Arnold,

Sure. But again, that seems to be a criticism of poorly-implemented aggregation in econometrics rather than aggregation per se, so am I correct in assuming that your problem is with the former rather than the latter?

As a quick aside... I'd never heard of measurement error biasing estimates toward zero, though I can see how it is true. So is the existence of unrecorded measurement error part of the reason that shrinkage estimators (ie James-Stein) work so well in practice? I know there are theoretical results that show that overall risk is decreased when you shrink estimates toward zero even without the presence of measurement error; but I'm wondering if in practice, shrinkage estimators do well partly based on theory and partly based on the presence of measurement error. Also, can they estimate measurement error by the amount of shrinkage that is needed?

Right, but what I wanted to say was that, while I'm more of a Bayesian than a frequentist, it's not really fair to place the blame of "accepting the null" on the frequentist's shoulders: even in stat 101 people are taught that "failure to reject" does not mean "accept".

So, if you set up a classical test with the null hypothesis as "health care has no effect," you are giving yourself a low probability of rejecting that hypothesis.

I don't know that I agree with this statement. You automatically give yourself a low probability of rejection if the null is true. The overall probability of rejection still depends on whether or not H0 is true, and on how far from the truth it is. I'll certainly admit it's easier to think of this in Bayesian terms because then you can say things like "the probability H0 is true" as opposed to just saying "it's either true or not."

Then of course there's the other side of the coin with hypothesis tests: given a large enough sample size, you will ALWAYS reject the null hypothesis. Then you have statistically "significant" results, but they don't necessarily mean anything in context.

Then of course there's the other side of the coin with hypothesis tests: given a large enough sample size, you will ALWAYS reject the null hypothesis.

This is probably nitpicking, but presumably you don't mean "ALWAYS" to include the case where the null is actually, exactly true - but rather that, even if it's arbitrarily close to being true, there will be a sample size large enough to guarantee rejection, no?

Conchis: from a Bayesian perspective, since the null hypothesis is just a single value, it has probability 0 of actually being true. So, technically I should say "almost surely always" (ie, true except on a set of probability 0).

Of course if you're a frequentist, then any talk of "the probability that the null is true" doesn't really make any sense. In that case I still have to disagree with the notion that a hypothesis test automatically sets you up with a small probability of rejecting the null. It depends on whether the null is true or not. If you wanted to use Bayes rule, you can write out the overall probability of rejection as a function of the marginal probability that the null is true: P(reject H0) = P(reject HO| H0 false)P(H0 false) + P(reject H0| H0 true)P(H0 true). But in frequentist circles, you are kind of stuck with those conditional statements P(reject H0 | H0 true) (ie, the alpha level that is set beforehand) and the associated power P(reject H0 | H0 false), which depends on the sample size.

:)

Actually (to be even more pedantic) while it's perhaps unlikely that you'd do so, what's to stop you specifying a prior that places a strictly positive probabilty on the null?