David R. Henderson  

CBO's Minimum Wage Study: Dealing with Publication Bias

PRINT
Erik Prince on Collective Puni... The 2008 transcripts: The real...

I've finally got around to reading more details about the Congressional Budget Office's report on the number of jobs lost from raising the minimum wage. The CBO estimated a range of outcomes from raising the minimum wage from its current $7.25 an hour (1) to $9.00 an hour and (2) $10.10 an hour.

For $9.00 an hour, the CBO's estimated range was "a very slight increase" to a loss of 200,000 jobs, with the central tendency being a loss of 100,000 jobs.

For $10.10 an hour, the CBO's estimated range was "a very slight decrease" to a loss of 1,000,000 jobs, with the central tendency being a loss of 500,000 jobs.

Given all the studies on the minimum wage, how did the CBO reach these conclusions? The report explains it.

Several factors influenced CBO's conclusion about the range of elasticities for teenagers. First, CBO put more weight on studies using certain methodologies than on other studies. Several studies compare employment rates among states that have different minimum wages but otherwise similar labor markets; such analyses plausibly isolate the effects of minimum wages from the effects of national economic changes, such as fluctuations in the business cycle. Other studies try to isolate the employment effects of minimum-wage increases by comparing the national employment rate in years when the minimum wage was high to the rate in years when the minimum wage was low. CBO put the most weight on the studies of state-by-state differences, judging those studies to have estimated more accurately the effects of minimum wages on employment. Changes in state minimum wages are sometimes related to local economic conditions in ways that could lead elasticity estimates based on those changes to be higher or lower than the elasticity that would apply to similar changes in law in the future; CBO considered studies that took a variety of approaches to addressing that issue. (p. 22)

The CBO also, though, took account of "publication bias." The CBO explains:
Second, CBO considered the role of publication bias in its analysis. Academic journals tend to publish studies whose reported effects can be distinguished from no effect with a sufficient degree of statistical precision. According to some analyses of the minimum-wage literature, an unexpectedly large number of studies report a negative effect on employment with a degree of precision just above conventional thresholds for publication. That would suggest that journals' failure to publish studies finding weak effects of minimum-wage changes on employment may have led to a published literature skewed toward stronger effects. CBO therefore located its range of plausible elasticities slightly closer to zero--that is, indicating a weaker effect on employment--than it would have otherwise. (p.22)

Clever.

For more on the minimum wage, see Robert P. Murphy, "Economists Debate the Minimum Wage," February 3, 2014.

HT to Robert Eger.


Comments and Sharing





COMMENTS (7 to date)
gwern writes:

It's reasonable, yes, but I'm a little surprised they didn't go further but just left it at an intuitive downward tweak.

Meta-analysts have been studying publication bias for decades and have many ways of detecting and trying to correct for it.

For example, just a month ago a new method was proposed: "P-Curve Fixes Publication Bias: Obtaining Unbiased Effect Size Estimates from Published Studies Alone" (http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2377290), Nelson et al (fun trivia: the biased minimum wage literature is one of Nelson's examples, where their method corrects the estimate down to zero).

CC writes:

IIRC, Tyler Cowen pointed out that the effect of publication bias could be "backwards" or at least quite muted for minimum wage studies though. In other words, if a researcher finds no effect on employment (and esp. if he finds a positive effect on employment!), he is actually quite likely to publish it. After all, this is a "surprising" result and worthy of publication.

So I'm not sure the usual techniques for correcting for publication bias apply to minimum wage studies.

Separate issue: Someone who is more familiar with the methodologies might be able to help me out here. If one state increases its minimum wage while a nearby state doesn't, then researchers like to look at the change in employment *after* the law is passed.

But don't employers anticipate the legislation to a large degree? In other words, if the state assembly if pretty likely to hike the min wage next week, the guy running McDonald's is not exactly going to go on a hiring binge now. So the effect after the law is passed could be quite small. Does any study take this into account?

David R. Henderson writes:

@CC,
Separate issue: Someone who is more familiar with the methodologies might be able to help me out here. If one state increases its minimum wage while a nearby state doesn't, then researchers like to look at the change in employment *after* the law is passed.

But don't employers anticipate the legislation to a large degree? In other words, if the state assembly if pretty likely to hike the min wage next week, the guy running McDonald's is not exactly going to go on a hiring binge now. So the effect after the law is passed could be quite small. Does any study take this into account?
I’m not sure. I was about to say that the Meer/West article that Murphy discusses in the piece I linked to handles this. But they deal with a separate issue.

gwern writes:

> So I'm not sure the usual techniques for correcting for publication bias apply to minimum wage studies.

I think that suggestion violates the usual setup of these analyses: with NHST, you only ever reject the null hypothesis or conclude nothing. What's the null hypothesis? Almost always, h_0 = 0 ie no effect from minimum wage (either positive or negative). So you couldn't 'find no effect on employment', because that's the null hypothesis and you never ever 'find' it; you have merely failed to reject it. You don't get to put the magic little * indicating <0.05 in your tables of findings. I don't think I've ever seen a demonstration of publication bias in favor of the null!

But anyway, even if there was (and if there was, why this talk of a 'cliff' at p=0.05 with a net negative effect size? which the p-curve corrects to a zero effect size), Cowen then suggests publication bias in favor of positive effect sizes - which should be observable with normal techniques.

gwern writes:

@CC: actually, I think the most helpful way I can answer your question is to just show you the effect sizes weighted by standard error (which p-values are just a function of):

http://graphics8.nytimes.com/images/2014/02/18/business/economy/18economix-minimum/18economix-minimum-blog480.png

(backup: http://andrewgelman.com/wp-content/uploads/2014/02/Screen-Shot-2014-02-21-at-9.35.43-PM.png )

This is similar to a funnel plot. Anyway, if you look at it, you see studies clustering around 0, you see a strange asymmetry around 0 where there seems to be a lot of negative results close to 0 but not a lot of positive results (even though sampling error should be symmetrical around the true point-effect), you see no study finds a positive effect higher than +5 but you see that there are a few studies finding negative effects all the way out to -20 (!), and in fact, there is a visually unmistakable and huge skew to impossibly huge negative effects.

Gelman comments (http://andrewgelman.com/2014/02/22/quickies/):

Interesting but perhaps misleading, as of course any estimate of elasticity of -20 or +5 or whatever is just crap, and so the real question is what is happening in the more reasonable range.

The simplest story here is the opposite of the Cowen suggestion: people really want to find negative effects to minimum wage, and the -20 studies are a demonstration of what you can torture the data into confessing, and there are no +20 studies because people would dismiss it out of hand (unlike the -20s, since those favor their position).

CC writes:

Very cool, gwern! Thanks for posting that link.

And interesting point about +20 studies being dismissed out of hand.

The Meer and West paper referenced by David above, is quite clever. It points out that the effect of minimum wage being raised will likely be found in a reduced growth in hiring, rather than in employment levels. Here's a non-gated version from December 2013; Effects of the Minimum Wage on Employment Dynamics. From the abstract;

For both theoretical and econometric reasons, we argue that the effect of the minimum wage should be more apparent in new employment growth than in employment levels. In addition, we conduct a simulation showing that the common practice of including state-specific time trends will attenuate the measured effects of the minimum wage on employment if the true effect is in fact on
the rate of job growth. Using three separate state panels of administrative employment data, we find that the minimum wage reduces net job growth, primarily through its effect on job creation by expanding establishments. These effects are most pronounced for younger workers and in industries with a higher proportion of low-wage workers.

Comments for this entry have been closed
Return to top