Scott Sumner  

Shun the paper, not the person

PRINT
Bryson on U.S. Standard of Liv... Bio of Elinor Ostrom...

Paul Romer has recently been posting on the subject of intellectual honesty in economic research. I'm supportive of his argument that economists should behave like honest scientists, rather than lawyers who withhold information that is detrimental to their argument. I'm agnostic about his specific criticism of papers by people like Robert Lucas and Edward Prescott, but my hunch is that he's at least partly right about the specific defects he finds. (I base this on nothing more than one of Lucas's fans admitting that one of the papers did have one of the flaws claimed by Romer. I'm in way over my head, and no one should pay any attention to my views on the question.)

But I'd like to focus on where I disagree with Romer.

1. I believe intellectual dishonesty is very widespread in economics, and indeed in other sciences as well. Not complete dishonesty, as when people completely make up results. That's pretty rare (I hope.) But Romer is trying to hold economists to higher standards than that, not just avoidance of outright lies, but also refraining from misleading claims.

Here's just one example of why I think Romer is wrong in claiming that intellectual dishonesty is mostly confined to a small group. My experience is that empirical studies in economics are frequently tweaked until the researcher finds a "statistically significant" t-statistic of 2.0 or more. I can't prove this, but I strongly believe it to be true. And yet I rarely see published papers that say "When I first did the study I found a t-statistic of 1.2, and then I tweaked the model over and over, and/or adjusted the data set until I got a t-stat above 2." There are entire fields in economics, such as finding "anomalies" in asset prices, that seem fundamentally dishonest to me.

2. Romer seems to think that the intellectual dishonesty is mostly confined to a small group of Chicago School economists. [As an aside, although I went to the University of Chicago (as did Romer), I'm not the sort of Chicago School economist that he is referring to. My views on macro are closer to new Keynesian than real business cycle. I believe in sticky wages/prices.]

Romer might be right about the Chicago School. But can we trust his judgment on this issue? I'm not sure about that. In recent years I've found Keynesian claims in the blogosphere to be increasingly far-fetched. At times there seems to be willful denial of the obvious failure of the Keynesian model, both in terms of fiscal stimulus, and unemployment compensation. "Tests" are suggested, and when the results come in unfavorable to the Keynesian model, the test suddenly never happened. But I'm an anti-Keynesian, so I can't really trust my own judgment on this issue. It's much easier to notice flaws in those you disagree with, than those you agree with. Maybe that's also true of Romer.

3. Here's one piece of evidence that Romer may be somewhat lacking in the ability to judge intellectual honesty in others. In this post he provides a twitter exchange with a supporter of Lucas. At one point the supporter claims that Lucas was more honest than 99% of the profession. Romer interprets that claim as defending dishonesty; he suggests that the Lucas supporter is offering an "everyone does it" defense. But of course that's not at all what the "better than 99%" claim means. To use a Christian analogy, it means something more like "we're all sinners, but he sins less than almost anyone else." Someone saying that is clearly not defending sin.

Elsewhere Romer portrays Stigler as a sort of Svengali who corrupted the Chicago school of economics. But in this post he clearly misunderstands what Stigler is saying. Stigler is making a positive claim (which seems accurate to me) not a normative claim.

And here Romer insists that there is such a thing as "objective truth":

"My fundamental premise is that there is an objective notion of truth and that science can help us make progress toward truth."
But a few paragraphs earlier sounds positively Rortian:
"In our discussions, claims that are recognized by a clear plurality of members of the community by as being better supported by logic and evidence are the ones that are provisionally accepted as being true."
Yes, he can use "provisionally" as an out, but when are our beliefs no longer "provisional"? How big must the "plurality" be to make it "objective truth?" And is objective truth never wrong? Remember that we're talking about macroeconomics here, how often do we have metaphysical certitude?

4. So far I'm just quibbling over minor issues of interpretation. And again, I think Romer is on the right side when fighting for intellectual honesty. But in this post I believe Romer goes far off course:

This description of the reputational equilibrium in science shows that it is inherently unstable in the face of entry by people with the norms of politics. If a small group of people with the norms of politics can establish themselves inside a science, they can destroy the equilibrium that sustains trust and communication. If the entrants guided by the norms of politics start publishing and citing Willie Horton papers, the scientists on the other side would be foolish to keep entering into a discussion with a presumption that "reasonable people can differ" and a willingness to be persuaded that some other position might be true. Politics begets politics.

The only way I can see to protect scientific discourse is to limit entry into the discussions of science. But this MUST NOT BE DONE on the basis of beliefs about what is true. It must instead be based on a demonstrated commitment to the norms of science. As part of this process of defending science, exclusion by shunning, plays an essential role. People who show, by publishing even one Willie Horton paper, that they are not committed to those norms, have to be excluded. So too must the people who promote and encourage Willie Horton papers. In science, "It was my PAC, not me" should not be an acceptable defense.

. . .

I think that the best course is to start by focusing on narrow specifics. In a paper:

1. Is a specific mathematical argument logically correct?
2. Do the words that the author uses to describe a mathematical result accurately convey what it shows?
3. Is the author giving a word or phrase a meaning that is diametrically opposed to the one that most readers will assume?
4. Does the author invoke the "Humpty-Dumpty" defense that a statement is true because "I use this word to mean something different from what everyone else thinks it means"?

These are simple questions of fact that we should be able to resolve. We can start on these first, but in this post, I am trying to be transparent about where I think that this process will should ultimately lead-to shunning the work of several prominent economists and their followers.


Here Romer is clearly talking about shunning people like Lucas, Prescott, and their followers.

I've meet many, many conservatives who tell me they've stopped reading Paul Krugman. I am often frustrated by Krugman posts, and occasionally feel he makes misleading arguments. But he also continues to do posts that are very insightful, and thus I feel an obligation to keep reading his blog. If Paul Romer stops reading brilliant economists because he disagrees with their methods in a single paper, then the problem of "epistemic closure" in the profession will only get worse. Shun the paper, not the person.

I've always tried to live up to the standards discussed by Romer, but I'm sure that I've fallen short here or there. On the other hand I also know for a fact that there have been times when commenters thought I was being dishonest, or purposely misleading, when that was not the case. [That's the price one pays for being a contrarian.] And because of these false accusations, I know for a fact that if others followed Romer's advice, they would have found at least one "Willie Horton" in my 1000s of blog posts. And Romer says that one is enough.

Actually, his strict standard can't do much harm to people like Lucas and Prescott; they have their Nobel Prizes. It's people like me that are much more at risk. I can see it now: "Sumner thinks the Great Recession was caused by tight money, not financial crisis. Only a crackpot or liar could make that claim." I'm the sort of person that would be shunned, not Lucas. Maybe that's why I feel so strongly about this issue.


Comments and Sharing






COMMENTS (24 to date)
Adam writes:

Professor Sumner,

Are you at all familiar with Deirdre McCloskey's body of work on the subject of rhetoric and ethics in economics? I only ask because I've noticed you sound a lot like her and her colleagues, and not just in this post.

RSF writes:

Seems like a very accurate post.

It seems like Romer's stated goal is admirable. However, when you call for greater intellectual honesty and largely accuse your opponents of misbehavior, you aren't going to influence behavior. Instead it is most likely to turn into a mudslinging contest.

Additionally, there is a slippery slope between calling out intellectual dishonesty and shunning people based on unpopular views.

Scott Sumner writes:

Adam, Yes, my views have been influenced by McCloskey.

RSF, I wish I understood mathematical econ better; I don't feel qualified to judge the extent to which Romer's complaints are accurate. But I agree that he should have used a more diplomatic approach.

Khodge writes:

Scott: well put.
I have worked in finance for many years and it is not just the effort to prove a point (confirmation bias) but, in spending hours in front of a computer, trying to make sense of the numbers, trying to interpret the charts, trying to re-create last week's brilliant insight/presentation, often clouds the vision and masks the inadequacy of this week's data.

Nick Rowe writes:

Good post Scott.

"The other side does X too" is not a defence of doing X, but it is a good defence against someone who is saying only one side does X.

Most academics would love to get papers published where the results merely failed to support (as opposed to rejected) some pet theory.

I am personally annoyed at Paul Romer because he accused me (me!) of being a hardline believer in perfect competition trying to obfuscate the existence of non-rival goods. Hey, I've been doing macro with monopolistic comp for 30 years now!

And his post on David Andolfatto was just plain nasty (as well as wrong, and a bit mathy itself). David was very polite in response.

I reckon he has a point about some papers having too much math and not enough intuition, so you can't see what's going on, and where the bodies are buried. Been saying the same myself, at times (but then I'm biased, because I'm bad at math). But you can't use that to go off on a biased crusade against the feshwater side, because there's a lot of bodies buried secretly in the New Keynesian math too. Like the Calvo fairy, whose random flight rigs the model in favour of inflation targeting. And the transversality conditions, which amount to simply assuming, for no reason internal to the model, that the economy eventually returns to "full employment".

Almost makes me want to join Minnesota.

(Really rather mild expletives deleted, to conform to Econlib policies)

Econymous writes:
I believe intellectual dishonesty is very widespread in economics, and indeed in other sciences as well.

This is almost certainly true.

In surveys, a surprising number of scientists (including economists) admit to dishonest research practices.

And there was the famous case where 2/3 of economic authors to a specific journal withheld their data in spite of hypothetically being required to give it up. If you look at Google Scholar and check works that have cited this paper, you'll see that, over two decades later, it's hardly better.

This paper is perhaps the best overview of problems with Economic research, including dishonest practices.

It's certainly not widely discussed within economics, but the literature (in economics and elsewhere) on researcher dishonesty is unambiguous (that last link really gives a great overview). If some government position had analagous lacks of safeguards for dishonesty as those faced by researchers, economists would write op-eds about how obviously problematic it is. The (relative) lack of introspection in the field is amazing.

Tom Brown writes:

Scott you write:

"Yes, he can use "provisionally" as an out, but when are our beliefs no longer "provisional"?"

I thought the answer to that has been clear now in the sciences for greater than a century: NEVER!!!

Given the right evidence, our most trusted theories: of quantum mechanics, evolution, relativity, thermodynamics, etc, can be bird cage liner tomorrow. It just takes a single case of something they cannot explain.

I see nothing at all to criticize in those parts of what Romer wrote.

All knowledge of objective reality is provisionally true. The bit about peer review and consensus, etc, just describes our imperfect procedure for assigning confidence levels... all of which should fall below 100%. So I read the "provisional" not as an out, but as a common and key work that appears in such descriptions of science.

JLV writes:

In re: #1, this is why an increasing number of journals (the AEA journals, RESTAT, etc) require replication materials.

But I hear this claim from a lot of researchers whose PhD training pre-dated the "credibility revolution". The criticism seems to stem from an understanding of how empirical work was done circa 1985, not 2015.

Tom Brown writes:

Scott, regarding your final quote from Romer, you have a LOT more skin in the game than do I, so I'm sure I'm missing some of what concerns you, but again, I don't see Romer's comments there as applying to your belief that

"...the Great Recession was caused by tight money, not financial crisis."

I see it applying to a paper submitted which purposely attempts to be rhetorical about the subject rather than scientific & honest (regardless of whether the paper presents evidence for or against the claim).

But I see your point about paper vs person. Although in other sciences people have had their careers ruined by a single egregious incident of falsifying data or being deceptive (amongst other reasons: what ever happened to the cold fusion guys?). Still Linus Pauling was a great physicist despite going off the rails on vitamin C. Newton's work in alchemy doesn't mean his work on mechanics should be shunned. Lol.

Regarding the comments above: I didn't know that Romer had attacked Nick Rowe. Just the fact that Nick is clearly perturbed by it makes me open to the idea that Romer was off base about that... Nick has a pretty thick skin in my opinion. I'll ask him for a link.

Also, regarding David Andolfatto, I did read that exchange between Romer and David, and I agree the tone was nasty on Romer's end. I too have followed Andofatto, and I think Romer was off the mark there (at least in terms of tone). David has taken the time (as have you all) to answer my dumbass questions in detail, even going so far as to ask his other colleagues there at the St. Louis Fed. So perhaps I'm biased. ha.

Scott Sumner writes:

Everyone, Nick's comment makes me even more sure that Paul Romer is off base here. I had forgotten about the charge that Nick was obsessed with the perfect competition assumption---Nick's more attached to the monopolistic comp. assumption than almost anyone I know. The fact that Romer could have been so clearly off base regarding Nick makes me much more doubtful about his interpretation of the others.

Scott Sumner writes:

JLV, Providing those materials in no way addresses my first point. I'm not accusing people of faking the data or results.

JLV writes:

@Scott:

I think you're being a bit too harsh on the importance of replication materials. For pretty much any empirical paper in the AER, you can download the data and code used to produce the results, and tweak the specifications to see if your belief of p-hacking is correct.

JLV writes:

To double down: in order to mark your beliefs to market, you would do just that. Or get an RA to do it.

Scott Sumner writes:

JLV, My claim is that people don't do that very often, except perhaps for major publications. I'd guess 90% of stuff getting published (especially in the minor journals) is never replicated. But I admit I'm a bit out of date on this, so please correct me if I'm wrong.

Econymous writes:

Daniel Hamermesh has done good empirical work on this topic. Scott is exactly right that most studies with available data never have their data requested. Economists simply lack incentives to undertake replication attempts.

And there are plenty of journals where provision of data+code is not required, so somebody doing indefensible p-hacking can elect to submit there. Also, "Most journals with archives do not bother to make sure that anything is archived." (PDF)

Finally, a lot of p-hacking can be done in such a way that data provided to the AER (or elsewhere) will not show it. Data can be edited prior to submission. Data collection could easily be done in a way that the resultant data is biased. The file drawer problem looms large even with required data+code. One could easily think of several ways that dishonest practices could exist with data+code submission requirements.

Requiring data+code is an excellent first step, but it does not solve the issue of questionable research practices or motivated reasoning, especially in its sparsely required/enforced current form.

Benjamin Cole writes:

Macroeconomics is politics in drag.

The Market Monetarists are (mostly) an exception to this, partly because one can be an MM and still have liberal or conservative beliefs on related issues.

What is odd is that "tight money" or the putative benefits of zero inflation have become right-wing political beliefs. This is a recent development; see Milton Friedman bashing the Fed for being too tight in 1992 when this CPI was at 3 percent.

A cautionary observation: many new drugs go through the entire FDA phase 1 2 & 3 clinical testing processes, are approved and then enter the general population and then prove ineffective.

And to have faith in an economic model?

Andrew_FL writes:

I have no dog in this fight, except insofar as Romer is trying to paint with a broad brush about the correlation between practices he criticizes and ideology.

If it becomes a popular meme that "right wing" economists are intellectually dishonest as a matter of course, the political fallout will be disastrous.

So the question I wonder about is, what's the appropriate division of resources devoted to "policing your own side" and "policing the other side"? If you don't police your own side at all, you'll end up with to many of your ideological fellow travelers engaged in intellectual dishonesty, thereby discrediting you by association. On the other hand, if you don't expend any of the effort set aside for policing intellectual dishonesty on "the other side," they can't be counted on to do it themselves, the result will be that it looks like only one side is being intellectually dishonest: again, you discredit yourself by association.

Actually, this is a question all intellectually honest economists should try to grapple with, at least if they believe their ideas are correct and care about other people, especially in politics, recognizing that.

Brian writes:

"The only way I can see to protect scientific discourse is to limit entry into the discussions of science. But this MUST NOT BE DONE on the basis of beliefs about what is true. It must instead be based on a demonstrated commitment to the norms of science. As part of this process of defending science, exclusion by shunning, plays an essential role."

Any talk of shunning, whether of the paper or the person, means the speaker has already left honest debate behind and has fully descended into mere political posturing. Even with a paper, one can disagree with its claims, even show that it is wrong, and still not "shun" it. It is worth talking about and reading if only to understand (and teach) where people can go wrong. Romer's whole approach is lacking in honesty.

That his point is merely political (in favor of his own side, of course) is amply shown by his claim that dishonesty is limited and found mostly in the Chicago School. Yeah right, whatever....

Nick Rowe writes:

From Paul Romer's latest:

" Their group-think allowed a coordinated move away from the use of data to evaluate or test a theory, and toward the use of calibration instead."

http://paulromer.net/what-went-wrong-in-macro-overview/

Totally false dichotomy. "Calibration" is like back-of-the-envelope empirics, only with a bigger envelope and sharper pencil, where you try to see if a micro-plausible number for e.g. labour supply elasticity gives you a macro-plausible number for e.g. the magnitude of business cycles. And if it doesn't (if you only get tiny business cycles, or need a stupidly large labour supply elasticity) you figure the model isn't empirically plausible.

Sure, calibration has its flaws, but so does econometrics (just look at the pathetic attempts to "identify" "monetary policy shocks").

And I really don't have a dog in that race. Though I strongly dislike RBC theory, I like how those guys used calibration to test (and reject) their theories. Is it enough? No. But it did add one more tool to our empirical methods (though it was really just an old tool used a bit more thoroughly).

Rich Berger writes:

"Willie Horton" papers? This reference gives Romer away. The famous Willie Horton ad was used against Dukakis in the 1988 presidential campaign. The ad was truthful, but considered to be unfair by liberals (you can view the ad yourself). Essentially what Romer is saying is that if there is a paper which puts our cause in a bad light, it should be shunned.

Some science, eh?

ThomasH writes:

In “What Went Wrong in Macro – Historical Details”

http://paulromer.net/what-went-wrong-in-macro-historical-details

Romer goes a ways toward the "all are sinners" point a view rather than single out Lucas and his group, although not quite as far as "hate the sin; love the sinner" that Sumner advocates, correctly, I think.

I wonder if greater acceptance of Stigler's views about the many macroeconomic mistakes made by Depression and Early Post War era policy makers would have led to a much faster convergence to the Neo-Liberal consensus with "Chicago" on the inside.

[Note: I do not know how to test the link.]

GU writes:
economists should behave like honest scientists, rather than lawyers who withhold information that is detrimental to their argument.

Actually, lawyers have an ethical duty to disclose evidence or information detrimental to their clients' cases. This is taken seriously by lawyers, as it could lead to sanctions, disbarment, or even a perjury prosecution. The only lawyers who routinely withhold evidence are prosecutors (usually with the cooperation of the police). For a non-government lawyer that isn't protected by governmental immunity, withholding evidence, or even legal precedents that cut against your case, is a dangerous game to play--it really happens a lot less than you apparently believe.

Economists have a lot less to lose, and are less likely to get caught, from withholding information. I would wager such withholding is more common in economics than it is in the legal profession (again, excepting prosecutors).

Koz writes:

I'm inclined to be sympathetic to Romer's bigger point but I strongly suspect that Romer is not the one to be instigating any sort of shunning.

Unfortunately in today's world there's a strong tendency where any kind of association with the Left, no matter how weak, leads to the sort of intellectual corruptions Romer is complaining about. Romer is not exactly a Maoist but even being (especially being?) on moderate Left is enough motivation to put ones' thumb on the scale in favor of his political or ideological interests at the expense of the truth.

Like Scott, I'm especially suspicious of Romer on this score but on slightly different grounds. Specifically, Romer's invocation of Willie Horton should be leading Romer in the exact opposite direction than in apparently does.

In my recollection, the accusations against Democratic candidate Dukakis by the George HW Bush campaign and the third party actors supporting him were all substantively true.

Willie Horton was an inmate in the Massachusetts penal system. He was furloughed by the Dukakis Administration. On his furlough he did rape and murder another victim.

Therefore those who wrote and financed the negative campaign ads against Dukakis were acting in the service of the truth whereas those who opposed them were acting in the interest of politically protecting their ideological interests. Therefore given Romer's stated agenda toward the advancement of scientific truth he ought to be in favor of the Horton ads.

Reading Romer's post, it's kind of ironic that he really doesn't describe in any kind of detail the nuts and bolts of the Horton analogy. If the thinks the whole thing is obvious to being beneath mention, it seems to me that is its own problem.

echan writes:

[Comment removed pending confirmation of email address and for making ad hominem remarks. Email the webmaster@econlib.org to request restoring your comment privileges. A valid email address is required to post comments on EconLog and EconTalk.--Econlib Ed.]

Comments for this entry have been closed
Return to top