Bryan Caplan  

Value-Added and Social Desirability Bias

PRINT
Can U.S. Presidents Much Affec... Globalization is not the probl...
Chetty, Friedman, and Rockoff's research on teacher value-added is widely interpreted as an argument for giving good teachers a fat raise.  Obama's 2012 State of the Union speech pulls a dollar figure right out of their abstract:

At a time when other countries are doubling down on education, tight budgets have forced states to lay off thousands of teachers.  We know a good teacher can increase the lifetime income of a classroom by over $250,000.  A great teacher can offer an escape from poverty to the child who dreams beyond his circumstance.  Every person in this chamber can point to a teacher who changed the trajectory of their lives.  Most teachers work tirelessly, with modest pay, sometimes digging into their own pocket for school supplies -- just to make a difference.

Teachers matter.  So instead of bashing them, or defending the status quo, let's offer schools a deal.  Give them the resources to keep good teachers on the job, and reward the best ones.
When I finally read the original paper, I was bemused to discover that Chetty et al.'s support for merit raises is muted at best.
Using Monte Carlo simulations analogous to those above, we estimate that retaining a teacher at the ninety-fifth percentile of the estimated VA [value-added] distribution (using three years of data) for an extra year would yield present value... [of] $212,000. In our data, roughly 9 percent of teachers in their third year do not return to the school district for a fourth year. The attrition rate is unrelated to teacher VA, consistent with the findings of Boyd et al. (2008). Clotfelter et al. (2008) estimate that a $1,800 bonus payment in North Carolina reduces attrition rates by 17 percent. Based on these estimates, a one-time bonus payment of $1,800 to high-VA teachers who return for a fourth year would increase retention rates in the next year by 1.5 percentage points and generate an average benefit of $3,180. The expected benefit of offering a bonus to even an excellent (ninety-fifth percentile) teacher is only modestly larger than the cost because one must pay bonuses to (100 − 9)/1.5 ≈ 60 additional teachers for every extra teacher retained.
The policy that dramatically passes the cost-benefit test is "deselection," better known as firing bad teachers.
To quantify the value of improving teacher quality, we evaluate Hanushek's (2009, 2011) proposal to replace teachers whose VA ratings are in the bottom 5 percent of the distribution with teachers of average quality.

[...]

Rothstein estimates that a policy which fires teachers if their estimated VA after three years falls below the fifth percentile would require a mean salary increase of 1.4 percent to equilibrate the teacher labor market.43 In our sample, mean teacher salaries were approximately $50,000, implying that annual salaries would have to be raised by approximately $700 for all teachers to compensate them for the additional risk. Based on our calculations above, the deselection policy would generate NPV gains of $184,000 per teacher deselected, or $9,250 for all teachers on average (because only 1 out of 20 teachers would actually be deselected). Hence, the estimated gains from this policy are more than ten times larger than the costs. [emphasis mine]
What's up?  I once again point my accusatory finger at Social Desirability Bias.  Rewarding good teachers sounds a lot nicer than firing bad teachers.  So when research comes along that potentially recommends both, pundits and politicians don't coolly crunch the numbers.  They leap to the recommendation that's pleasing to the ear.  So what if the original researchers find that firing bad teachers wins with flying colors?  Move along folks, nothing to see here...




COMMENTS (12 to date)
Anthony writes:

As a libertarian, do you trust government agency managers to select the correct 5% to fire?

Even if you test student improvement year-over-year, your looking at a sample of maybe 90 kids from a heterogeneous pool. Will that successfully identify the actually poor teachers, or just the unlucky ones? Or the ones who teach at schools with lots of poor kids?

Peter Gerdes writes:

Or politicians and pundits are well aware that firing bad teachers is both too politically difficult (teachers are risk averse and will fear they could be fired) and legally/administratively complicated.

Tom West writes:

How lucky are university professors whose VA is essentially incalculable (at least once they have tenure).

And heaven help the college lecturer, for if Bryan is correct, the VA on college students is entirely from the piece of paper at the end, in which case, fire the ones that pass the fewest percentage of students, especially in trades like medical school where the students can look forward to high incomes!

We all like accountability, but I've yet to see a large scale evaluation function of job performance that wasn't roughly the equivalent of Soviet's measuring nail production by the weight of the goods.

The only difference is we laughed at the Soviet's efforts.

Procrustes writes:

To be fair, it appears that they are only quantifying the impact of those bonuses on retention; presumably this incentive pay would also influence teacher behavior on the margin (and potentially attract better talent to the field in the first place).

On the other hand, while increasing the retention of star teachers may be a good local strategy, presumably most of those third-year teachers leaving one school district are simply moving to another district (mostly within the US), rather than dropping out of the profession entirely. If that's the case, the retention benefits from a national plan would be much lower than calculated.

coupon_clipper writes:

Procustes: Good point about local vs. global strategy. Firing a teacher may well just move that teacher to another district.

In a similar vein, the 250k figure is a "micro value added", not a "macro value added". To illustrate my point, imagine that really good teachers slightly increase the likelihood that students become NBA stars. In this case, a good teacher hardly raises national income at all. Those new NBA stars just displace other would-be stars.

In contrast, if good teachers increase the likelihood that students stay out of jail and instead work in "regular jobs", then that's a big big boost to national income.

I suspect that most of the 250k is attributable to the latter case, though I'm always surprised that people assume that it's 100%.

Floccina writes:

The worst teacher that I ever had, he could not control the students at all, my mother had had 30 years earlier. Had he been fired back then I think he would have had a happier life and the students would have been better off. Talk about aversion to firing!

Floccina writes:

BTW in many cases you don't even need to fire them, you can move them to non teaching positions.

Melanie Heisey writes:

Moving them to non-teaching positions sounds promising. Are there no programs that look for people in a field doing particularly badly in predictable ways, then offering them support and resources for switching to a career they're better suited for?

Anthony writes:

Procrustes:
presumably most of those third-year teachers leaving one school district are simply moving to another district (mostly within the US)

Not necessarily. Many people realize they're not cut out for teaching about two or three years into their career, and quit to try something else. Many of those fired after three years for being lousy teachers will leave the field entirely. However, the gains from such a policy may be exaggerated, as many of the worst teachers fire themselves after two to four years trying.

Procrustes writes:

Anthony: A quick google search led me to this NCES data (see table 3 in particular), and I stand corrected.

There's some movement between school districts in there, but around half is within district, and overall more teachers leave teaching entirely in their first five years (17%, of which 80% departed voluntarily) than end up in a different district (5%).

Of course, those statistics are for teachers in aggregate, and the best and worst 5% of teachers are unlikely to be representative. However, with only 5% of teachers moving between districts, it's probably safe to assume the effect I suggested is not material (although this leaves plenty of room for your "firing themselves" story).

Mr. Econotarian writes:

I wonder if the high level of teacher benefits lead to an artificial reduction in teacher salaries that reduces attracting enough quality teachers into the field.

Also I suspect what schools need more is quality training & support of teachers in the field to help them improve. Every NFL or NBA player still works with trainers and coaches every week no matter how good they are.

Cole D. writes:

"As a libertarian, do you trust government agency managers to select the correct 5% to fire?"

The better option would be to trust the consumer to make the decision by use of a voucher system. Give parents a voucher to be used at any school they wish, and perhaps even extend that to allow them to select specific teachers within the school. Word of mouth travels fast between parents. Quality teachers will quickly be identified by an increase in demand while poor teachers will be located through a shortage of students in their classroom. Teacher compensation can vary based on the level of demand for their services and teachers must have a minimum threshold to reach before they are in danger of losing their jobs.

Comments for this entry have been closed
Return to top