Arnold Kling  

Should Macroeconometric Models be Outlawed?

Russ Roberts is Wrong... Bravo for Scott Sumner...

Sometimes, people want something so badly that there is money to be made selling them a defective product. Werner Troesken's paper (full version here) on the history of quack medicines offers an example. No matter how many of these medicines were demonstrable failures, the demand for new quack medicines was high. People who are really sick tend to cling to unreasonable hopes.

I want to suggest that decision-makers with a lot of power are desperately needy for scientific solutions in the same way that individuals with serious afflictions are desperately needy for remedies. As a result, decision-makers fall for models the way sick individuals fall for quack medicine.

In macroeconomics, the scientific basis for the sort of macroeconometric models that I learned to work with early in my career has been in doubt for thirty years. In that time, I cannot recall seeing a single peer-reviewed journal article that uses such models. And yet, those are the models that the Fed, the CBO, and private consulting firms use to make forecasts and assess the impact of stimulus measures. This paradox has been noted by Greg Mankiw in The Macroeconomist as Scientist and Engineer.

The way I see it, a question like "How many jobs will policy X create?" is not really answerable. Yet telling that to a policymaker is like telling a patient that his or her illness is incurable.

The only way to get patients to stop falling for quack medicine was to outlaw snake oil. This suggests a precedent for what to do about macroeconometric modeling.

Comments and Sharing

COMMENTS (10 to date)
John Thacker writes:

OTOH, note that a significant number of economists have questions about whether outlawing snake oil really solves the problem, or whether it's not worth the cost.

Andrew writes:

Dear Arnold,

I read your blog regularly and respect your opinions, but your criticisms of macroeconometrics are really misguided. I work at the Fed, and yes the use of the VAR/DSGE models to answer questions like "How many jobs will policy X create?" is widespread. And, yes, from reading your "Lost History of Macroeconometrics," there is a valid point in showing how flawed--terribly flawed--these tools are. Or, for that matter, how trivial economic forecasting (if we'll settle on one use of such models) can be.

But your description of the average policymaker/central banker is a caricature. It's ridiculous. The average Fed economist--however deeply entrenched in DSGE-type work--has the EXACT SAME view of the flaws in their tools as you do! The whole enterprise--more art than science, for sure--reminds one of FDR's quotes: "It is common sense to take a method and try it. If it fails, admit it frankly and try another. But above all, try something." With all due respect, you offer nothing in return; however your positive (in the formative sense) policy prescriptions masquerade as "something," they're "nothing"; you present only negative prescriptions. My main point is that your median Fed macroeconometrician is not an idiot, he/she is just using the best tools available to actually "do something."

And, no, the average "consumer" (FOMC member, other policymaker, politican, etc.) of this average macroeconometrician's work is not grasping onto the results as "science" or "fact" or "a good forecast"--it just informs their view of the world along with the other data and models before them. It's a tool, not some sacred cow. You present the macroeconometrics modeling enterprise as if it was a religion when it's really much more humble. The whole enterprise--again, more art than science--has a pragmatist core to it that I don't think you respect. I wish you did--I wish you picked up the models and tried to shape them to be something better than the pathetic things they are now. I wish you, with your great intelligence and wisdom, "tried something."

Lastly, you may retort and critize my view of The Policymaker and their importance (or, rather, their impotence, in your view), but while I am every bit of a libertarian as you are (at least if I could remake our institutions from a blank slate), when I come to work, I'm pragmatist.

With all respect and sincerity,

aretae writes:

Perhaps you could do the same with climate modeling?

Dave Troy writes:

This is an interesting debate. I think Andrew brings up a good point. Just because something is flawed, does that mean that it offers no utility as a mechanism for inquiry?

In the end policy decisions must be made by people based on a synthesized rationale about what makes sense. Are you saying that models offer no utility at all as inputs into this decision making?

Certainly the use of a model to defend a fundamentalist position (like supply-side economics) represents the worst kind of abuse. But is there no middle ground?

Additionally, just because our models now may be deeply flawed, does it follow that useful models cannot ever be built?

While climate modeling is still quite imperfect, it is arguable that climate models today are more complete and more predictive than those from 30 years ago. Should development of those models have stopped then because a perfect model cannot be built?

Where is the meta-research on modeling that shows that it is impossible to ever build a good model? I'm prepared to believe that the math inherently includes too much compounded chaos to ever be predictive, but I have not seen anyone try to prove it. (I'm a novice to this field, as well, so that might be why.)

Until such a proof is offered, it seems to me the pragmatic thing to do is to try to continue to improve models, because the rewards of getting a good-enough model outweigh the costs. The trick, in the meantime, is to remain humble about their conclusions, as Andrew suggests.

Respectfully submitted,
Dave Troy

JPIrving writes:

I agree completely, I know first hand that the USCIS (immigration feds) use bogus "input output models" from firms like as criteria for participation in the "immigrant investor" program where they shake down foreign entreprenuers who want a green card.

Essentially: The Korean investor who wants his kids to go to American schools pays a consultant to show job growth using an off the shelf macro model, and then the Feds give him and his family a green card.

Complete waste of resources, if they have money and skills I think just about everyone can agree they should be let right in.

So in some limited way these things are actually affecting investment and migration decisions, but only because of foolish immigration laws!

eric writes:

r.e. Andrew

Sometimes negative prescriptions are what is required. As counterintuitive as it may seem, perhaps doing nothing is preferable to "trying something". Of course, this does make it harder to justify one's position at the Fed.

Colin K writes:


I think the question may be whether you and your colleagues are in a good position to evaluate whether your use of models is sufficiently humble, to use your phrasing.

I'm willing to assume that everyone's intentions are honest and sincere. But self-knowledge, particularly on an institutional level, can be difficult.

My definition of Macroeconomics is in part:

The study of overall statistics about income, debt, production, and employment. You would think that statistics are dull, but macro economics is a center of intellectual ferment.

Motto: "We just don't know, but we are willing to guess". Critics say that it is history confused by mathematics, or mathematics splattered by history.

Macroeconomics is Astrology, Not Science

Frank J. Tipler is Professor of Mathematical Physics at Tulane University.

He says: "Our leaders are being advised by macroeconomists who haven’t got a clue where they are leading us. Their actions may lead us out of the current recession, or they may lead us into a depression as bad as the Great Depression."

Science is about prediction and precise explanation. It is not enough to construct a different explanation about each past event. Science must produce a consistent, precise explanation for all of the relevant past events.

Then, real science predicts the future and is testable according to those predictions. If a "science" cannot predict what will happen, then it is clear that it does not understand enough about what is going on, and of course it is of no practical use in arranging for a better life.

- -
It is an awesome responsibility to manipulate people's income, assets, and choices, because you have a feel for how things might work out better if you do this or that.

Don't take that responsibility. No one has a grant of Celestial Authority to manipulate people's lives because they want to do good.

George Washington probably died from the application of insight, anecdote, and the medical arts. His physicians bled him to death. They just had to do something.

Loof writes:

The quack medicine analogy is good: applied to small (micro) economics when modeling selfish decisions to allocate limited resources; just as well as to large (macro) economics, as the sum total of selfish interest activity engineered for the selfishness of each national state.

The is in contrast to the genuine medicine of self-interest (ego and empathy) of each personal individual freely going about their business with free and fair markets to create the true wealth of individuals, nations and the world.

Norman writes:

While this works as a rant, you offer absolutely nothing to convince the unconverted. From this and other posts, it sounds like you would attempting to give policy advice based on feeble attempts to use data be sacked... and replaced with political pundits and journalists, so long as they happen to repeat vague incantations about Recalculation. And you may be right, but you've not yet offered a reason for someone who currently likes data to think so. "If I'm right the recession will be really bad" is a rather feeble attempt.

Those responding to Andrew above that nothing may be better than "doing something" are quite right from a policy perspective. From the perspective of science, though, you can't just reject a (deeply) flawed system and instead say we should draw our knowledge from nowhere. This is error of the Intelligent Design folk. If you're going to say "this is a pathetic attempt to understand macroeconomic issues," you have to throw us a bone, preferably one a bit of meat (actual data, rather than story and opinion) on it.

Comments for this entry have been closed
Return to top