Edward Leamer’s indictment of modern econometrics, “Let’s take the ‘con’ out of econometrics” is the best known critique of our habits as empirical economists but it has not been taken to heart by the profession.
There are a number of generic criticisms of regression methodology. As I recall Leamer's book Specification Searches, he argues that while theoretical tests of statistical significance are based on a single confrontation between model and data, the practice of econometricians is to try a number of empirical specifications before reporting the one they like the best. This means that reported standard errors are too low, t-statistics are too high, and reports of statistical significance greatly exaggerated.
The rest of Russ' post talks a lot about what I think of as confirmation bias. Basically, when we like a result, we give the methodology much less scrutiny than when we dislike it.
I think that good economists rarely change their minds about a topic based on a regression result. That observation, if true, suggests that something is wrong. Part of what is wrong is that we have confirmation bias, so even when we should change our minds we come up with excuses not to believe contrary results. But an even bigger part of what is wrong is that regression results are indeed quite fragile, so that one should not give them much weight.
For example, on the question of whether the death penalty deters murder, I would never trust a regression result. There are too many choices for specifying the causal variable. Is it a dummy variable set equal to 1 if the crime falls under the death penalty statute in the state where it is committed? It is a probability of being executed, based on some measure of past performance of the justice system? Is it some measure of what potential murderers think is the probability of being executed? Moreover, there are way too many important potential control variables. Etc.
My personal opinion, which is unlikely to be swayed much either direction by econometric results, is that the death penalty is unlikely to have much effect on murder. My guess is that most murders are committed under circumstance in which the killer is neither in the mood or in possession of the data to incorporate potential death penalty punishment.
On the other hand, society would benefit from making other crimes capital offenses. When I was in graduate school in Cambridge (Ma.), double-parking was rampant. I wanted to get around on my bike, and urban streets are hazardous enough as it is. Double-parked cars are a nightmare. I was a teaching assistant for Ec 10 at Harvard, and when it came time to teach the economics of crime, I advocated the death penalty for double-parking.
I am pretty sure that the death penalty would work in that case, because double-parking is not a crime undertaken by someone whose thought processes have shut down. Double-parkers are making rational calculations, so that the death penalty should have huge deterrent effects.
So from a deterrence standpoint, we should make a capital crime out of double-parking, not murder.
As far as crimes go, econometrics falls somewhere in between double-parking and murder. I'm not sure about the right level to set the deterrent, but it needs to be higher than what we have today.