Arnold Kling  

Stat Fight, Con't

Stat Fight... Yes Donald, Beliefs About Econ...

Bryan says,

a fair number of papers have a lot of data points. So rejecting the null hypothesis of no effect is pretty easy. Moreover, it is fairly common in both literatures to discuss the magnitude of the effect, not just statistical significance.

I agree with the technical point, but I would have to see specific papers. Sometimes, you do not have nearly as many data points as you appear to have. One of the reasons that I have given up hope of settling macroeconomic questions empirically is that with serial correlation and structural change, the effective number of data points is ridiculously small, regardless of what somebody might report as the "number of observations." Those specific issues don't affect the data that we are talking about, but other issues do.

He also says,

I agree with Arnold that we lose some valuable information when we look only at the aggregate effect of medicine or parenting. (Indeed, that was my whole point!) But that aggregate information is good to know, and one can imagine the public digesting it

All I can think of is the public digesting Paul Krugman's argument that we spend more money on health care than countries with socialized medicine, we have the same longevity as those countries, therefore socialized medicine is more cost-effective.

In fact, we spend $2000 more per capita than other countries. Taking David Cutler's value of a life-year as $100,000, the critical value for whether that $2000 is cost-effective is whether it increases our longevity by one week. (UPDATE: see the comments for a valid criticism of this arithmetic) If you want to try to find that week by controlling for all the other factors that affect longevity (genetics, homicide rates, traffic fatality rates, etc.), go ahead. But I think it's like trying to call balls and strikes from a helicopter.

As to studies of parenting, I would be curious to know whether you as a pro-natalist regard your parenting style as being perfectly consistent across children. For each child, have you used the same method for getting the child to sleep through the night? For toilet training? Will each child have to wait until the same age to get a particular toy or a particular privilege?

If you are like most parents, your answer would be "no." Yet aggregate studies of the effects of parenting almost always assume that the answer is "yes."

And just when you want to say that "Even though I do some things differently with my children, I have the same basic personality with all of them," someone like Judith Rich Harris will come along and say that the effect of your basic personality as a parent can't be separated from your genetic effect.

I think that in both health care and parenting, there is a sleight-of-hand going on in claiming no effect. Intuitively, we think of the effect as big, based on a disaggregated view. But then the statistical "expert" comes along and does a macro study that introduces a lot of variation in the dependent variable due to extraneous factors, then picks a weak proxy for the factor under consideration, and pronounces the effect of that factor to be zero.

Comments and Sharing

CATEGORIES: Economic Methods

TRACKBACKS (2 to date)
TrackBack URL:
The author at Exploit the Worker in a related article titled Econometrics Blogging writes:
    Bryan Caplan and Arnold Kling are going back and forth and back and forth on topics related to aggregation and rejecting null hypotheses. Sure, that sounds boring, but these issues have important implications for the conclusions we draw from empirical... [Tracked on June 3, 2005 4:09 AM]
The author at The Liberal Order in a related article titled Cripple Fight!!!!! writes:
    Bryan Caplan and Arnold Kling are at it. See here; then here; and then here. Here's the Download cripplefight.wav. In Arnold's first post he claims,Similarly, with parenting, my guess is [Tracked on June 3, 2005 8:53 AM]
COMMENTS (10 to date)
Nicholas Weininger writes:

Uh, wait a minute. We spend $2K more per capita *per year* than other countries, so something like $150K more per person per lifetime. So the cost-effectiveness threshold would be an increase of 1.5 years or so in life expectancy. Still probably too small to pick out of the noise, but not as obviously ridiculous. I agree completely with your general thesis that life expectancy is a very bad proxy for health care efficacy, but your math is off here.

Arnold Kling writes:

Good point that its wrong to try to match an annual number to a cumulative number. But we have not been spending $2000 more per year for 75 years. The difference was much smaller if you go back further, especially if you go back more than 25 years.

And I am not sure that cumulative changes are well picked up by longevity as conventionally calculated. See my essay on longevity.

So I agree that my math is not correct. But the right math, which may not be do-able, would show a much smaller expected increase in longevity than 75 weeks.

Matt Festa writes:


Another bit of evidence to back up your point is that the empirical tests have not supported some of the most important models in economics. I am thinking, of course, about the long battle over the random walk of consumption and the fight over ricardian equivalence (which is based, in part, off the permanent income hypothesis).

I would also like to point out the paper by Krueger that argues that a rise in the minimum wage did not cause a decrease in employment. Empirically, have we been able to come up with a conclusion that budget deficits increase int. rates (ever week I get a different paper arguing a different conclusion).

I guess this is why microeconomics is on a more solid foundation than macroeconomics. You have larger and more accurate data sets on which you test. But the macro testa are hard to set up, have lots of proxy variables, and are riddled with time series problems.



conchis writes:


Sorry for being slow, but how exactly does failure to confirm "important" models, support Arnold's point? On the one hand, random walk consumption is based on a raft of pretty implausible assumptions (lack of credit restrictions, quadratic utility/certainty equivalence, no adjustment costs or durable goods, infintely lived consumers etc.) so even from a purely theoretical point of view, you wouldn't expect it to hold. On the other, when you're testing for a random walk, aren't you testing for a zero effect of things like lagged expectations of changes in income? Finding such a zero effect is exactly what Arnold claims is easy, but that's fairly consistently rejected.

Matt Festa writes:


You bring up a lot of good points. As a newbie to the field of economics (I take my MS comp in economics tomm morning) let me make the following observations

1) The Random Walk of Consumption

According my lecture notes and papers, the random walk of consumption is not that out there. It is based on standard microeconomic theory about indifference curves, budget contraints, ordinal utility, etc. There are also tests that take into consideration durable goods, borrowing costs, and other empirical objections. The tests still fail (although the ones that take into account these points tend to have a "better" fit.) This suggests myopia, which is a bit troubling. But I don't think the random walk theory is theoretically wrong, it is based on standard microeconomic theory.

2) null hypothesis

I have to stand corrected on this point. My point was that the tests in macroeconomics seem to be rejecting some of the more commonly used (and debated) models. I offered ricardian equivalence and the random walk of consumption as two examples. This has led some economics, ie Ed Prescott, to reject econometrics as "play-nomics" precisely because they are rejecting so many good models.

From re-reading the posts, Arnold seems to be making a more specific point about null hypotheses, to which I would add "never accept the null," always say "I cannot reject the null hypothesis." "I can reject the null hypothesis to the 95% confidence level."

Thanks for the friendly conversation. What say you?

Conchis writes:

Hi Matt,

Perhaps we mean different things by random walk consumption. I would say that once you're building into your models things like credit constraints, precautionary saving, habit formation etc. etc. to explain excess sensitivity and excess smoothness, then you're not really talking about a random walk any more. You're explicitly trying explain why consumption isn't a random walk. (I was under the impression that these bits and pieces can go a fair way towards explaining the failure of the random walk hypothesis, but perhaps I'm wrong about that.)

As a general methodological point, I would be incredibly suspicious of arguments that econometrics must be flawed because it rejects "good" models. It seems a little rich - after being drilled for so long with the mantra that "the test of a good model is its ability to fit the data, rather than it's assumptions" - to suddenly turn that on its head and start claiming that the evidence must be wrong because my models are really good.

Now, perhaps you/Prescott are making the weaker claim that all this rejection is merely suggestive of problems with econometric methods, and so we ought to think about such potential problems a little more. I don't have much problem with that, but the fact that the data rejects particular models is still, of itself, no evidence that any particular criticism of econometrics is true. Rather, I think you need to point, as Arnold has done, to specific ways in which econometrics might go wrong.

conchis writes:

... It just occurred to me that you might mean that econometrics often can't reject the possibility that the good models are wrong , rather than that it rejects good models - in which case I have a little more sympathy. Part of this problem seems to be tied up with the somewhat arbitrary imposition of confidence intervals, but I guess it's also kind of inherent in the classical approach, and could be improved on by Bayesian methods (where tractable) or simulation-type stuff. I'll readily admit to not knowing anywhere near enough about either of those to be able to say much useful on the topic.

Chris R writes:

You're exactly right (and I'm saying this as a macro guy in training). You and Arnold basically agree on the ridiculous lack of good data in macro and the need to be parsimonious at the expense of misspecifying a model. That leaves two credible alternatives that I can think of.

1. Find the most disaggregated data that you can possibly find in order to do your analysis. If you're talking about inventories, take firm-level data seriously. If you're talking about labor markets, take individual-level data seriously. If you can't find good canned data, consider going out to get your own. This is something that development economists have learned after running 1,000,001 growth regressions and coming up with no answers.

2. Calibrate, simulate, recalibrate. Maybe go Bayesian when the need strikes. Regular hypothesis testing requires a well-defined alternative hypothesis in order for Neyman-Pearson to work. Finding clear alternative/null combinations that perfectly fit economic theory seems to be damn near impossible to me. That's the whole 0.99 vs. 1.00 argument above. This, in my opinion, is the main contribution of the RBC literature to thinking about how we do macro as a science--models as metaphors, thought-experiments.

3. Take the econometrics in perspective. When testing asset pricing theories, do you really believe that you've found an arbitrage opportunity, or are the econometrics wrong? Do you have measurement error? Time-varying parameters? Omitted variables? Endogeneity? That might get me slapped down here in unit-root land, but let's put the econ back into econometrics.

People don't really accept that empirical economics is hard because running a regression is so easy. There aren't any $100 bills lying out there on the sidewalk that could be picked up by applying some significance test. Others have already picked out correlations using arbitrary downloaded datasets from the BEA. On top of that, they don't understand what a t-stat can and cannot say. We don't deal in certainties; we're not the math or theology department.

Matt Festa writes:


This is a fascinating conversation.

Let me clarify my position here because from your two posts, I don't think we are so far apart.

1) Random Walk of Consumption

My only point about the random walk of consumption is that it is based off of sound, basic economics: ie rational consumers maximizing utility. The rejection of the random walk troubles many macroeconomics because it seems to be provide evidence agains this assumption
In order to get around this, some add budget constraints, account for durable goods, etc. to explain why the tests may fail but the intuition correct.

Then there are those who argue for myopia.

2) E"con"ometrics?
If we take the pure popperian/friedman method seriousely, "we look at how well the model predicts, not at how logical it is" Ie, if the model is based off of crazy assumptions BUT explains 95% of the facts, we accept that model and reject the one that "seems" more correct.

But the pure form the this hypothesis has to be wrong. I remember the example of sun spots and recessions. Sun spots correlated extremely well with recessions, but does that mean we accept sun spots as THE reason for recessions? No, obviousely we want some sort of logical explanation as to why the phenomena is happening.

I also remember the big hub-ub over the phillips curve (the tradeoff between inflation and unemployment.) Sure, the empirics tested well and seemed to work, but there was no economic reason for it to work. (ie it wasn't utility maximizing for the consumer to behave the way the curve was suggesting). And when policy makers exploited the tradeoff, the relationship broke down.

3) The assumptions behind economics
Economics assumes rational optimizers (consumers max utility, producers max profits). We assume this not because consumers are always rational, but because we believe that economic forces will, overtime, make people behave more rationally (economically).

4) Econometrics
Personally, I am with you that you absolutely need econometrics, if nothing else than for it to be your link to your real world. You need it to test your theoretical models, see which ones are working and which one's aren't. But econometrics, at the end of the day, can't tell you WHY those models aren't working. Going back to the phillips curve, it became clear that the reason why the phillips curve worked was that it hadn't been exploited yet.

With the random walk of consumption, maybe it's there are cultural forces pushing people to "buy now, think later." Or perhaps it is the case that it isn't really rational to sit down at the kitchen table and figure out your mean income over your actuarial forecasted life span (I think it's a combination of the two.)

So....I think at the end of the day you need to pay very close attention to econometrics, but you have to be extremely careful. Econometrics comes with lots and lots of warning flags (esp with time-series data) that often leave it open to misinterpretation (and sometimes, partisan abuse)

conchis writes:


I don't think we're too far apart either. (I probably could have made that clearer in my last post.)

I guess I still don't think that the random walk consumption is a good example of your point, because I tend to think there's more wriggle room in it's assumptions than there is in the econometrics. But we can reasonably disagree about that - and in any case, the failure of one example doesn't necessarily invalidate your point.

Obviously there's a need for a bit of a reflective equilibrium type process in moving between models and data. Whenever models don't fit the econometrics, you've always got two options: reject the model, or reject the econometrics. In some cases, there are obviously good grounds for being sceptical of the econometrics: but even with the something like the philips curve case, you need to say what that is - why it is that you think the data are wrong, beyond "my model doesn't predict them" (and if you're relying on the data just being a statistical anomaly, you need to be open about that too). And on the other hand, there's almost always good reason to be sceptical about assumptions underlying economic models too.

Economics assumes rational optimizers (consumers max utility, producers max profits). We assume this not because consumers are always rational, but because we believe that economic forces will, overtime, make people behave more rationally (economically).

Actually, I think by and large we assume it because it gives us tractable models that we can work with rather than ones with really messy maths, or which just don't give us any predictive ability. Selection pressure means that rationality is far more reasonable as an assumption about firm behaviour than consumer behaviour (and even then only as a long run outcome, which may not be all that helpful if you're looking at one shot games). It would be silly build models aroudn automatons that didn't respond to incentives at all, but there's a lot of room between that and rationality...

Right, I think I'm going to stop ranting now. :)

Comments for this entry have been closed
Return to top