Bryan Caplan  

Seth Roberts on Education and Evaluation

PRINT
Mankiw: We Need Fiscal Hawks,... Comments on Eurozone Solutions...
Seth Roberts has an interesting response to "The Magic of Education" - that I don't go far enough:

Professors are terrible at evaluation. Their method of judging student work is very simple: How close is it to what I would have done? The better you can imitate the professor, no matter what the class, the higher your grade. This is one size fits all with a vengeance because there is no opting out. Sure, you can choose your major. But every class is taught by a professor. What if your strengths lie elsewhere -- in something that your professors aren't good at? Tough luck. Your strengths will never be noticed or encouraged or developed.

At Berkeley (where Bryan went and I taught) and universities generally, the highest praise is brilliant. Professor X is brilliant. Or: Brilliant piece of work. People can do great things in dozens of ways, but somehow student work is almost never judged by how beautiful, courageous, practical, good-tasting, astonishing, vivid, funny, moving, comfortable, and so on it is. Because that's not what professors are good at. (Except in the less-academic departments, such as art and engineering.) To fail to grasp that students can excel in dozens of ways is to seriously shortchange them. To value them at much less than they are worth -- and, above all, to fail to help them grow and find their place in the world after college.

I certainly agree that professors have major blind spots.  But if we're really "terrible" at evaluation, why do employers take our judgments so seriously?  Furthermore, if the traits professors fail to measure have high market value, why hasn't another evaluative industry emerged to pick up the slack?  You could say that beauty, courage, practicality, tastiness, astonishingness, vividness, funniness, and the like are all better measured on-the-job, but why would that be so?


Comments and Sharing





COMMENTS (21 to date)
Noah Yetter writes:

Furthermore, if the traits professors fail to measure have high market value, why hasn't another evaluative industry emerged to pick up the slack?

Because those traits can't be objectively measured.

Also: Goodhart's Law

joeftansey writes:

Maybe beauty and insight aren't important. Maybe conformity and following instructions is more important. Maybe not everyone needs to be Socrates.

Rohan writes:

But if we're really "terrible" at evaluation, why do employers take our judgments so seriously?

Are you really sure they do?

In my experience, prior work experience (in the same field) means much more to employers than your school. The only time school really matters is when you're evaluating someone who has no experience. You have nothing else to fall back on, other than school, really.

Seth writes:
The better you can imitate the professor, no matter what the class, the higher your grade.

This describes how managers evaluate their employees. They tend to evaluate the inputs (how they do their job) relative to how the manager thinks it should be done, rather than the outputs or what actually resulted.

This is especially true in places that have relative safe incomes that are not highly dependent on outputs.

Foobarista writes:

Disparate impact: Griggs versus Duke Power.

Employers have effectively recruited colleges to do first-round intelligence testing for them. They don't particularly care about the deep details as long as they have a way to figure out who not to hire that is cheap and legally defensible.

Most employers aren't so much worried about getting optimal hires as much as they're desperate to avoid total losers, particularly if they're from a "protected group", who can tangle them up in expensive lawsuits when they get fired.

Employers who _are_ interested in optimal hires still use this method to as a "necessary but not sufficient" test, figuring that most optimal hires will make it through a decent college.

Christopher Finocchio writes:

Employers listen to you because they want people who can produce work that looks like what their bosses produce.

Abe writes:
But if we're really "terrible" at evaluation, why do employers take our judgments so seriously?

When was the last time a group of employers asked to randomly audit your teaching, grading, and evaluation system? How is it possible that employers (or affiliate organizations) care so much about teaching/grades and yet absolutely never check to make sure your process works. There's no Deloitte and Touche of Universities.

That's a sure sign that education (from the employer perspective) is about signaling... not about actual content. You can talk about accreditation if you want - that's a rubber stamp these days.

PrometheeFeu writes:

"But if we're really "terrible" at evaluation, why do employers take our judgments so seriously?"

I don't think they do. They only care that you graduated. The main effect might be that as someone who went to school, I have used up 4 years of my life and $160,000. (not counting forgone income) This however only allows me to get a shot at a higher-paying career. If after getting an entry-level job I do not perform well, I will be fired and if that happens more than once or twice, I will simply be unable to get any more of those entry-level jobs for higher-paying careers. So this expenditure of time and money only makes sense if I am confident that I will be successful in a high-paying career. I think that's one big part of the signal employers get.

So I think employers trust you to have a very low standard. Dropping out of college is very difficult. Many of my classmates spent inordinate amounts of time procrastinating, not doing homework and missing many classes and graduated. (I was in many ways one of those people) I think professors basically can recognize people in the bottom X% where X is less than 30 with a low rate of false negatives. The false positives then find other ways to get into higher-paying careers.

RPLong writes:
But if we're really "terrible" at evaluation, why do employers take our judgments so seriously?

If Santa Claus and his Elves don't really exist, then why do children take their evaluation of their niceness or naughtiness so seriously?

Just because professors have managed to convince people that their opinions about students matter doesn't mean that their opinions are correct or accurate.

It's possible for millions of people to be wrong about something.

TL writes:

Response to RPLong: Certainly truth is not a function of the number of believers, but isn't the point that here we're discussing something with a market payoff, so presumably competitive pressure to get it right, whereas with elves and children, the social pressure is, if anything, to maintain the illusion?

Don Kenner writes:

Employers assume (rightly or wrongly) that having completed college you 1. are literate 2. can take orders. 3. can work under deadline. 4. might be capable of bringing creativity to your work. In other words, the college degree is what the High School diploma used to be.

Many Universities (and degree plans) are publicly funded rackets, with government intervention distorting market signals. Absent competition (because of high barriers to entry thanks to government), it is unlikely that businesses will have much reason, or opportunity, to consider professorial evaluation for its strengths and weaknesses.

RPLong writes:

TL, I guess what I'm suggesting is that when a country sinks enough public money into a given subsidy program [read: public education and publicly funded higher education], the pressure reverts back to maintaining the illusion rather than to preserve any sort of market function.

We are way beyond competitive markets when it comes to higher education. Subsidies, public school systems, and generations of feel-good post-modernist nonsense has ensured that nothing about the education market serves any real purpose other than to perpetuate the bloated socialist machine.

Is it any surprise that the students who excel to the utmost in universities go on to take either university positions or federal/state government positions? That is the real "market" function of the higher education system as it exists today, certain exceptions notwithstanding.

But if we're really "terrible" at evaluation, why do employers take our judgments so seriously?

I suspect they don't have much else to go on.

I do disagree with the "how close is it to what I would have done," because many times we use things like rubrics. But I do agree with the lack of judgement on other aspects of one's work or self.

John T. Kennedy writes:

"But if we're really "terrible" at evaluation, why do employers take our judgments so seriously"

I don't know, but I turn on the financial channels and see analysts with terrible track records taken seriously every day. The fact that your (academics in general) evaluations are taken seriously doesn't they're not terrible.


Perhaps in this case it doesn't matter much but they just need a reasonable sounding rationale for their decisions.

Tracy W writes:

Alternative view: professors should focus on their comparative advantage. Most professors' comparative advantage is that they have mastered a body of facts, theories and techniques that most people have not. Thus said professors have an advantage at assessing mastery of said academic work in their field. There's no reason to assume, however, that a professor of say, mathematics, is any better at assessing courage or beauty (of the non-mathematical kind) than any other person, so why should the professor waste their time assessing it?

Note that professors in fields like engineering, arts, architecture, graphic design, etc do assess aspects like practicality or creativity.

Basically, if you want to signal courage, take up mountaineering, or front-line duty in Afghanistan or political activism in Russia (or etc), there's no need to pay a professor to prove you're brave to employers.

Tim of Angle writes:

Basically, Roberts is criticizing colleges for not doing something that they aren't really trying to do. He is ignoring both focus and context.

Our educational model is built around hiring teachers who are (supposedly) good at thing X and paying them to train other people to do thing X. Nobody claims that the way the teacher does thing X is the only way to do thing X, nor even the best way to do thing X; what colleges do claim is that the way the teacher does thing X is a successful way to do thing X, and it hopes that the teacher can train students to do thing X competently at least the way the teacher does thing X.

All schooling (I'm not getting into the 'education' swamp, thank you) involves a combination of OBJECTIVE and METHOD. It's like building: carpenters build stuff with wood, masons build stuff with masonry. Roberts would be the guy saying, 'Jeez, this carpenter build this using studs, and it would have been better as a stack of bricks. This is a tragedy!' People would look at him as if he were an idiot, and rightly so, because he would be an idiot. Carpentry doesn't train you on every way to build stuff, but it does train you (presumably effectively) on at least one way to build stuff, and that gives you a skill you can use to make a living.

And this can be narrowed, too -- there's more than one way to toenail a stud, but the guy who's teaching you is going to teach you the way he does it. It may not be the best way, and it's certainly not the only way, but it will equally surely be an effective way, and will allow you to truthfully tell a prospective employer, 'I know how to toenail a stud.' And that's what the employer is looking for.

(If you want to study and speculate on other ways to build stuff, you can do that in your spare time; if you come up with a better way, start your own business. It's happened.)

Roberts is a classic example of someone insisting on making the best the enemy of the good, and such people are responsible for a lot of what is wrong with the world today.

Mark V Anderson writes:

Tim -- you make a lot of sense.

One sometimes hears the proverb that a hammer will define all the problems in the world as being a nail that needs to be pounded in. That is true, but it does not detract from our need to train hammers. Sometimes the problem really is that a nail needs to be pounded in.

I was not impressed with Roberts' posting. He defined the problem as being only able to evaluate if the student was good at being good at things that Roberts himself does well. He wasn't evaluating the things that the students do well. His solution is to let the students decide what they do well and then evaluated those things.

But I suspect that Roberts is even worse at evaluating what the students do well at. Presumably Roberts was good at evaluating what he does well at least, so he was going backwards. This is an example of how aiming for the best can be an enemy of achieving the good.

I very much agree with Bryan that most college education is an exercise in signaling. As such much education is a great waste of resources, as so many students spend so much time and money not to improve their skills and knowledge, but to simply show potential employers that they are the best choice. As many indicate in the thread above, such signaling does have some value to society as an effective winnowing device, but the cost seems way out of line with the benefits, IMO. But Roberts' post shows the danger of throwing out the baby with the bathwater.

JoeFromSidney writes:

In the mathematics classes I teach, there is only one right answer. Evaluating students on the basis of whether they get that answer is the only way to go. As to how they got the answer, that's another matter. There may be many methods that get the right answer to that specific problem. The question is whether a method will always get the right answer to every problem. If so, it's acceptable. But using a method that happens to get the right answer on a specific problem, but will fail on others, is not acceptable. If that constitutes grading the students on whether they did it as I would do it, so be it.

Tracy W writes:
You could say that beauty, courage, practicality, tastiness, astonishingness, vividness, funniness, and the like are all better measured on-the-job, but why would that be so?

For most of that list, because they can be demonstrated very quickly. Does a story make you laugh? Then it's funny. Does the meal taste good?
Courage and practicality are situation - specific, someone can be courageous about some things and not others (eg physical courage is not the same as political courage). So demonstrating them at uni isn't that useful, though of course some courses do cover aspects, eg engineering degrees cover some practical topics.

When I see recommendations like this from an professor, I interpret it as an attempt to gain a monopoly on credentialing everything instead of just academic subjects.

Maybe schools should stick to teaching.

Steve Sailer writes:

It sure sounds like Seth Roberts is tired of the huge number of East Asians at Berkeley. Private colleges tend to discriminate against Asian applicants, but there are barriers to Berkeley doing as blatantly that due to Prop. 209. That's why the average GPAs of incoming freshmen at Berkeley are so astonishingly high, way over 4.0.

Comments for this entry have been closed
Return to top