Arnold Kling  

The Chetty Teacher Study and the Hill Criteria

What Is the Male Marriage Prem... Solar Power Update...

This article discusses a study by Raj Chetty, John Friedman, and Jonah Rockoff on the effect of teacher quality on subsequent earnings. This is a study that has been cited a lot recently, by Alex, among others, as evidence that teacher quality matters. Below, I examine the study from the perspective of the Hill criteria for establishing causation. You may think that my grading is strict, but my reading of the study is that it clearly satisfies only 1 of these 9 criteria, while it clearly fails to satisfy several.

The Hill criteria were developed to argue that the evidence showed that smoking causes cancer.

1. Temporal relationship. Did the teacher quality difference take place before the earnings were measured? Yes.

2. Strength. Did teacher quality predict earnings as strongly as smoking predicts lung cancer?. I would say No.

3. Dose-response. Is there evidence that earnings are related to the amount of exposure to good teachers? No.

4. Consistency. Do other studies using different methods show a similar response? The results are consistent with the kindergarten teacher study, also by Chetty and others (but see Russ Roberts on that study). However, I would think that if teacher quality made such a strong difference, then we would see clearer evidence that attending a school with stronger teacher hiring policies affects subsequent earnings. As far as I know, there are no such studies No.

5. Plausibility. Is there a theory of a causal mechanism? Yes, we think that it is plausible that teachers make a difference. However, we do not have a specific causal mechanism. As the article points out,

Moreover, while the new research may identify HVA[high value added] teachers, it's still not clear what constitutes good teaching. Despite volumes of research, there are no criteria that enable schools to identify good prospects, nor are there a set of best practices to guide teachers in the classroom.


6. Consideration of Alternative Explanations. Can we rule out alternative explanations?

"Effectively, we identified experiments in the data when students come into contact with HVA teachers, such as when they change grades, leave or enter the school system," Chetty says.

The "natural experiment" methodology is intended to address the possibility of alternative explanations. However, in my mind, there still might be parental variables involved in the decisions that created the "natural experiments." Maybe

7. Experiment. Can the condition be prevented or ameliorated, the way that lung cancer can be prevented or ameliorated by cessation of smoking? If the "condition" here is low earnings, then it cannot be prevented by even a host of good teachers. This is a rare study that even suggests a small amount of amelioration is possible. No

8. Specificity. Can this be regarded as the causal mechanism? This criterion does not even hold for smoking and lung cancer, so some methodologists question whether it should be applied. But in this case, because there are so many other possible mechanisms affecting earnings, the answer would be No.

9 .Coherence. Are the results compatible with existing theory and knowledge? The results are consistent with Hanushek's work which suggests that there are differences in "value added" among teachers. However, the results are inconsistent with much work that shows that the effects of education interventions of all sorts dissipate within a few years. No.

Comments and Sharing

COMMENTS (7 to date)
Ken B writes:

An outstanding post. But the study does pass the Capitol Hill test: it serves the needs of a powerful body of rent-seekers. So it will be widely accepted. "Your superior intellect is no match for our puny weapons."

Also people are more persuaded by anecdote and all remember a great teacher.

Seth writes:

Did quality of students cause the quality teachers to look better?

mark writes:

One of the problems I have with this study, not the study so much as the extraploation from, it is the law of diminishing returns. If you have a shortage of high-earning jobs, conceivably exposure to a HVA teacher enhances your ability to compete for one. But if everyone had equal exposure to a HVA teacher, I don't think everyone gets a high-earning job, and I don't think everyone gets a higher-earning job, although that seems to be the argument Chetty and his followers take away and sponsor. First, there are only so many high-paying jobs. Whether my garbage man reads Proust or not, there is only so much taking my garbage away is worth. Second, at some point, if you equalize one variable, then the other variables kick in.

ghost of Christmas Past writes:

I agree with mark, if all students get HVA teachers (and I think they should, if possible) then the differences between graduates will relate to something other than teaching. This is the classic problem for people who hate to think about IQ because it is partly genetic: if you do equalize environmental factors, then the differences you have left will all be genetic-- and, I may add, will still be large enough that occupational sorting by ability will occur. HVA teachers will never turn lifetime earnings into a lottery uninfluenced by natural ability.

Komori writes:

Judging from my experiences in public school, I think they're approaching it from the wrong end.

Rather than try to find the rockstar HVA teachers, they should be trying to find the NVA (no/negative-value-add) teachers and get rid of them. This seems like it would be both simpler and more productive-per-cost. Because there were plenty of teachers that were either completely ineffective at teaching (students probably don't know more after the class than before it) or who seemed to be actively suppressing learning (students actually know less after the class than before it. And yes, I did have teachers who were teaching things that were flat out incorrect, like the science teacher who claimed there was no such thing as anti-matter, no matter how many published science papers you showed him).

Ken B writes:

I agree with you about the NVA approach, but think you are missing the point. The goal of puffing the study is not to enable firing bad teachers, it is to garner support for paying teachers more. This is why the study is being hyped and not subjected to the kind of careful, skeptical critique Arnold Kling gives it here.

Bryan Willman writes:

One problem is that for some very basic subjects such as reading and math, most students will be taught by somebody. So the test isn't the difference between "no education" and "really good educators".

But that will be the dominating effect - people who don't go to school, don't pay any attention, etc., will always pull down the Human Capital result.

The burned out/incompetent/simply bizarre teachers, who apparently arise in some number in *every* district to the great exasperation of administrators, are probably the 2nd large effect.
But in many (most?) districts even they will be swamped by the "did the pupils even show up" issue.

Comments for this entry have been closed
Return to top