Bryan Caplan  

Fade-out, Teacher Quality, and Summer Learning Loss

PRINT
More on Fox News... TSA's Phony Choice...
Getting students to learn is hard.  One of teachers' favorite consolations, though, is that "If I can just get through to these kids, it will all be worth it."  The hope, apparently, is that successful teaching will permanently changes students' lives.  But does it?  A new paper by Jacob, Lefgren, and Sims (ungated version here) rains on educators' parade.  High-quality teachers do change their students more, but they can't stop fade-out:
The primary claim of the recent teacher value-added literature is that teacher quality matters a great deal for student achievement... While this claim may well be true relative to other policy alternatives, our results indicate that contemporaneous value-added measures are a poor indicator of long-term value-added. Indeed, test score variation due to teacher value added is only about one-fifth as persistent as true long-run knowledge and perhaps one-third as persistent as the overall variation in test scores. Thus when measured against intuitive benchmarks, contemporary teacher value-added measures almost certainly overstate the ability of teachers, even exceptional ones, to influence the ultimate level of student knowledge.
They continue:
...Taken at face value, our results for two-year persistence imply that a policy intervention to raise teacher value-added by a standard deviation would produce along-run effect on student math achievement closer to 0.02 standard deviations than the 0.10 standard deviation increase found in the literature.
Which reminds me: Advocates of year-round school (and even worse, Saturday school) often argue that summer learning loss is large.  But notice: Summer learning loss is just a specific kind of fadeout!  Unless a high school diploma magically locks in knowledge, anyone who believes in severe summer learning loss should also expect kids in year-round or Saturday school to quickly lose their extra knowledge after graduation.  Strange as it may seem, then, summer learning loss is an argument for less education.  Why make the poor kids suffer if they won't retain what they learn anyway?

HT: Tyler


Comments and Sharing





COMMENTS (22 to date)
P.S.H. writes:

What do you make of the argument that, ceteris paribus, it is easier to relearn something you've "forgotten" than it is to learn that thing as a genuine novice?

Jonathan writes:

A contrary piece of evidence is the Chetty, Saez, et al working paper on Project STAR, which suggests that kindergarten differences pay off in adult earnings. http://obs.rc.fas.harvard.edu/chetty/STAR_slides.pdf

Hyena writes:

My experience is that fade out is rapid and large but balanced by familiarity.

When I sat down to learn Python, I realized that I'd lost most of my programming knowledge. But the moment I started looking at it, I was able to rapidly reconstruct much of it, enough that sitting down with "pure beginner" books is annoying and just reading over the documentation was far more productive.

I emphasized this effect when I did workplace training, especially on technical areas people might not encounter immediately: the goal was to ensure future familiarity, not complete knowledge. I said as much in every training round I conducted and was rated the most effective trainer in our department, both by trained staff and their managers.

From that experience I've concluded that the primary value of learning something in school is the ability to rapidly relearn that subject at a later date, to reactivate and reconstitute attenuated but largely extant pathways.

Philo writes:

"[A]nyone who believes in severe summer learning loss should also expect kids in year-round or Saturday school to quickly lose their extra knowledge after graduation." And also *before* graduation, unless what they learned long before the time of graduation were continually repeated or somehow reinforced up to that time (much of it is not).

Philo writes:

{Prompted by P.S.H.'s comment, above:}
Not only is it easier to relearn certain material than to learn it initially--especially in the case of skills as opposed to mere information--but even for information the awareness that it exists, even if you've forgotten the details, will probably not fade away, and may be of value in forming your overall outlook and in allowing you to find the forgotten details when you need them.

Troy Camplin writes:

Repetition is the soul of education. Nonlinear approacehs to education have also shown that 1) presenting everything right away, then going over it all slowly allows for more rapid learning, as well as better retention, and 2) that short breaks (such as Spring Break or Christmas) allow for what was learned to self-organize. The summer may be too long for retention. Perhaps students need more short breaks throughout the year. Either way, the linear view of learning is wrong and needs to be thrown out. Our brains are complex adaptive network systems, and we need to be educated accordingly.

Les writes:

I wonder if the concern about fade-out is overdone. Knowledge changes as progress is made, and what was important yesterday may no longer matter tomorrow.

Bryan perceptively asks: "Why make the poor kids suffer if they won't retain what they learn anyway?"

Perhaps one answer is that retention is not the most important factor. The most important factor may be learning how to learn.

Daublin writes:

A very good subject!

How much of the fade-out is explained by motivation, though? When I think about classes I was forced to take, I crammed for the tests but then forgot everything as soon as I could get away from the class. I almost eagerly shed all the crud that had been forced on me.

However, when I think about subjects I like, a better teacher makes all the difference. I retain that material much better, and I learn an order or magnitude more when the teacher is better. Saying fade out is inevitable as well as saying that good teachers don't prevent fade out, is hard to swallow.

Bryan, what about your own best teachers? Did you forget everything they said 6 months later?

Doug writes:

Holy moly! .02 regression coefficient is so incredibly low. This means (assuming normal distributions) that if we could replace every teacher with a "one in a thousand" teacher (99.9 percentile - 3.09 SD), then the average student would only be boosted 2.5 percentiles* (i.e. a student in the 50th percentile would now be at the 52.5th percentile of the old ranking).


* Explanation of math:
3.09 SD * .02 = .0618 SD.
Average student has a 0 Z-score (i.e. is at the mean of the distribution).
NORM(0 + 0.618) = 52.46%.
In addition students below or above average would be effected less since the derivative of the CDF Normal with respect to the Z-score peaks at 0.

Tracy W writes:

In Dan Willingham's book "Why do students hate school?" he presents some fascinating evidence that multiple years of practice of something are needed to engrave it in the brain for a life time. The studies he presented were of algebra. People who had taken one algebra class and then nothing else 6 months ago, when tested on algebra, had forgotten about half of what they knew. The people who had gotten an A on their end-of-course exam still knew more than the people who had gotten a C, but both groups had forgotten about half. If people had gone on to take another algebra class, they forgot more slowly, if they had taken a calculus class, they forgot more slowly again, and if they had taken maths beyond calculus, they still remembered as much algebra 50 years after their last maths class as 6 months after their last maths class.

Now of course, as anyone who has done them can tell you, calculus and post-calculus maths classes involve a lot of practising of algebra. That repetition seems to emblazon algebra into people's long-term memory.

Steve Sailer writes:

In the long run we're all dead.

But, there is a lot to be said for the short run, especially when there are windows of time that close. If a kid doesn't learn to read by 10 or 11, will he ever learn to read? If he doesn't join a gang by 17 because he kind of likes school, will he ever join a gang?

FC writes:

Does anyone "kind of" like school? I don't recall anyone who actually preferred school to work, drugs or TV.

Tracy W writes:

FC - I'm guessing you never encountered the perpetual student at university. (Although admittedly all the cases I can think of where I visited their living place had big TVs).

Mr. Econotarian writes:

What this data implies is that short-term test results may not correlate with long-term knowledge acquisition.

The problem is how to choose metrics that allow for appropriate educational process control. In manufacturing, it is the quality of the finished product that matters the most. A part that meets tolerances early in the assembly line could be damaged by the time product assembly is completed.

Perhaps this is how we should view education. Unfortunately, the "back propagation" (neural network learning term) of educational error from final student to earlier classes and teachers is a difficult equation.

Roger Sweeny writes:

I teach high school and I think that deep, deep, deep in our hearts, we all know that students forget a lot over the summer and also forget a lot over the course of a school year.

But like the elephant in the room, we refuse to allow ourselves to think about this. We pretend to ourselves that someone who got a 90 on the chapter test in October would also get a 90 if the test were given in April. After all, he or she has "learned" it.

But, of course, we do know that the score won't be 90 unless we "review" quite a lot before the test in April. However, to admit that the students really haven't learned much, that most of what they did was fairly short-term memorization, is to raise the question of what we are actually being paid for. So most of us don't go there.

(And, yes, lots of people in the ed business think that summer fade-out is a significant and real problem but fade-out within the school year is a very different and much smaller thing: qualitatively different, shouldn't even be called the same thing. That way the problem is they're not with us, rather than they're not learning.)

Tracy W writes:

However, to admit that the students really haven't learned much, that most of what they did was fairly short-term memorization,

How interesting. I'd say that short-term memorisation is the essential first step towards learning anything.

Roger Sweeny writes:

I'd say that short-term memorisation is the essential first step towards learning anything.

Some sort of memorization is certainly necessary.

If you memorize who the first five presidents are and then use that as a framework to organize your knowledge of the first three decades of the American republic, definitely a good first step.

If you memorize the times tables and use them to solve math problems more easily, definitely.

If you memorize definitions without understanding the definition or the words in the definition, no, not at all. It is depressing to have a student answer a test question or write an essay and make no sense because they are using words that sound like what was in the definition but which have totally inapplicable meanings.

If you memorize something and then use it, you are more likely to learn it. Alas, most units in a school course are pretty self-contained. You do it, get assessed on it, and don't return to it again until the mid-term or final. If a student is really interested and tries to fit things together, this may actually result in a fair amount of learning. However, since most high school students aren't inherently interested in the vast majority of what they are supposed to learn in high school, the result is usually several cycles of memorize and forget.

Tracy W writes:

If you memorize definitions without understanding the definition or the words in the definition, no, not at all.

When it comes to memorising the definition without understanding it, that's not my experience. When I started learning calculus, it took me about a year to understand why, after I'd differentiated something, setting it to zero produced either the maximum or the minimum. The teacher had carefully taught us why, it's just that I didn't understand the why. Until I did, I just memorised the explanation and reproduced it on tests and applied it in problems. A similar thing happened to me with deadweight loss in economics. In both cases, I eventually had a sudden breakthrough of meaning.

Of course it's nice if understanding can come simultaneously with first hearing/reading the definition. But brains don't always work that way. And if you don't remember the definition, you're not going to be able to understand it afterwards.

As for not understanding the words in the definition, that's a sign that the student hasn't properly memorised the right meaning of the words, or perhaps hasn't accurately memorised the definition.

I agree with you that schools could do with constructing their curriculae so as to encourage repeated returning to material, to support longer-term retention.

Roger Sweeny writes:

Tracy W,

You are absolutely right. If you memorize something without understanding it but use it frequently enough and stick with the subject you learned it in, yes, you may well get "a sudden breakthrough of meaning" somewhere down the road.

Alas, if you don't encounter the concept a number of times in the future, either because it is used a lot in that course or it comes up in other courses, that moment will never come.

And yes, a depressingly large amount of definition memorization consists of students mindlessly memorizing the words in a definition without really knowing what the definition means (or even what some of the words in the definition mean).

"Mr Sweeny, your definition is different from the one in the book." This bothers a lot of them, because to them a definition isn't an idea, it's a sequence of words. Like a magic spell, it has to be the exact words in the exact order or it just isn't the definition.

Tracy W writes:

Alas, if you don't encounter the concept a number of times in the future, either because it is used a lot in that course or it comes up in other courses, that moment will never come.

Quite. My point is that if I do understand the concept, but don't encounter it often enough to drive it into long-term memory, I will forget it, despite the understanding. A part of my job is working out why odd things are happening, and I've learnt that once I have figured out why it's happening, I have to write down the why, or I forget it.

Which brings me back to my first point - memorisation is an essential first step in the learning process.

This bothers a lot of them, because to them a definition isn't an idea, it's a sequence of words. Like a magic spell, it has to be the exact words in the exact order or it just isn't the definition.

This is inflexible knowledge, which if I understand the cognitive psychologists correctly, typically is something most people have to have before they can have flexible knowledge. There are some people who always seem to go immediately to the flexible knowledge on encountering a new idea (perhaps though they can think of exceptions), and probably everyone can sometimes go immediately from a new idea to flexible knowledge, but for most people, most of the time, inflexible knowledge seems to be an unavoidable first step in learning something. (See http://www.aft.org/newspubs/periodicals/ae/winter2002/willingham.cfm)

So, if the cognitive psychologists are right, finding this depressing is like a builder finding it depressing that so many buildings start with the foundations. You can be depressed about it, or you can be optimistic that at least the kid has memorised the definition and now you can build on that to a real understanding. Or, perhaps, you can come up with some brilliant breakthrough that removes the need to use foundations without losing the value of the building on top of them, but the last option is not entirely under your control.

Noah Yetter writes:

Getting students to learn is hard.

No. Children are born with a natural passion for learning. It's hard to stop a child from doing so.

What's hard is forcing students to learn things that don't interest them, which is the M.O. of our school system.

Tracy W writes:

No. Children are born with a natural passion for learning. It's hard to stop a child from doing so.

Ah, the idea behind whole language for reading instruction. What this ignores that evolution equipped us to learn certain things, like oral language, and folk physics, but what we want children to learn has gone beyond that, into things like writing, and mathematical physics, which are not the things that most people are born with a natural passion to learn.

Comments for this entry have been closed
Return to top