Arnold Kling  

Trusting Scholarship

Are Economists As Smart as The... Compensation for Immigration...

Felix Salmon writes,

one of the problems with such research is the way that it's presented to the public, through newspaper and magazine articles, or TED lectures, or the online dissemination of pre-pubication versions of papers. The stuff which gets the most attention has very rarely gone through the peer-review process, and as a result it's often impossible for us, the final consumers, to get a good grip on what's worth taking seriously and what's just junk science.

As an aside, Salmon mentions the book by Raymond Fisman and Edward Miguel, called Economic Gangsters. I read the book a few weeks ago, and I have been meaning to blog about it. It has elements of Freakonomics in its description of the detective work of empirical analysis, and it has elements of The Undercover Economist in its focus on underdevelopment and its use of the basic economic model of incentives to explain behavior. I liked it a lot while I was reading it, but the longer it sat next to my computer waiting to be blogged about, the more it sort of faded for some reason. It's way better than most books you are likely to be encouraged to read, but I'm not ready to give it a "must-read" recommendation.

As for Salmon's larger point of when you should trust an economics paper, for me the key issue is how the authors treat potential objections. Basically, if the author ignores important potential objections, you cannot trust the paper. If the author meets the more important potential objections head-on, then that is better. As an amateur, your main disadvantage in evaluating a paper is perhaps not being as aware as a professional would be of potential objections.

I put no special value in peer review. You can have a peer who discards a paper for purely idiosyncratic reasons, or because he just does not understand key aspects of the paper. More often, you have a peer who approves a paper because it cites his own work favorably, which makes it immune to criticism in the eyes of the reviewer.

Comments and Sharing

COMMENTS (5 to date)
William writes:

Perhaps (but hopefully not), I see things a bit differently per my training.

As I am involved in law, I see dissent as a powerful and important aspect to preparing any document, or vetting any source.

Obviously if people in a similar position cannot critically involve their brains in a dialectic, I would rely doubt the strength of their authorship or research.

It seems to me that the lack of competition among ideas (externally or internally) is what Salmon wants, and I agree. However the lack of dialogue between you and your peers (if I was in your position) is very disconcerting. It is also a trend in scientific research, and though disheartening leads to the most amusing assertions via statistics, flawed basic logic, and the supplanting of logic with biases.

There is no better way to find some sort of truth than to consult a person who objects, dissents, and rallies devilishly to undercut your points. The value and usefulness of a man's work at times can be measured by the cunning and strength of his enemies, that is IF it addresses and stands up to them. That is what competition is all about.

Andy McKenzie writes:

Arnold, do you have a "must-read" list anywhere? I'm curious to see which books you consider must reads for the general educated public with perhaps a slight preference for economics. Thanks.

Ed Hanson writes:


You are close to using perfect as the only measure of a process. The proper measure is whether peer review, in general, improves scientific research in general. Whether such a process should be trusted always is, of course, plain stupid.

The answer you are looking for might be a better mechanism than peer review, if so develop it and put it into action. But if such mechanism is not readily available, and peer review process has become too often gamed, than create a journal or such, which its editors recognize these flaws in the system, and find more knowledgeable, less self-interested reviewers. If the problem you see is great enough, such a journal should flourish.

phineas writes:

Speaking as someone who frequently reads computer science journal articles, I fully concur with Arnold's attitude that "the key issue is how the authors treat potential objections.... As an amateur, your main disadvantage in evaluating a paper is perhaps not being as aware as a professional would be of potential objections.... I put no special value in peer review." In computer science, peer review evaluates for the presence of novelty, and absence of blatant error, but essentially does not evaluate quality. The end-reader's key task is to separate wheat from chaff, both within a single article and between the many articles published. There's an awful lot of chaff in computer science journals, and no doubt in economics and every other field (although ameteurs may be educated to some extent by stuff that for more experienced readers is just chaff).

Linda writes:

Peer review also has problems when a relatively small circle of people controls publication (or worse, funding) and is not disposed to entertain ideas contrary to its world view.

Sometimes well funded groups even start peer reviewed journals with the express purpose of publishing work heavily slanted towards one policy prescription or another in order to manufacture a consensus view--the "consensus view based on published work disagrees with your position" gambit that is heard so often in the global warming debate.

For empirical work, I would put the quality of the data set used at the top of the list of how well they handle objections, and also consider the level of statistical torture required to get the given result.

Comments for this entry have been closed
Return to top