Bryan Caplan  

The SAT Puzzle

PRINT
The Mystery of Libertarian Pat... What Nudge Really Says...
In the comments, Joel cleverly turns a puzzling fact around on me:
Colleges care about applicants' SAT scores. Employers don't. What's going on? I'm tempted to just repeat my adage that, "For-profits are crazy," but even for-profit colleges care somewhat about SAT scores.
I'd say that this is an isolated puzzle in the for-profit world, not a systematic pattern of craziness.  But Joel still poses a good question.  Is it just fear of "SATs=IQ tests" lawsuits?  Or what?


Comments and Sharing





COMMENTS (36 to date)
Steve Z writes:

I think your gut reaction is right. Under Title VII, can't employers get sued if they administer tests that aren't "job-related", if there's a disparate impact? Even verbal tests of the subject matter relating to the job are insufficiently "job-related," so I imagine SAT tests would be a bridge too far. . .

Braden writes:

As a young guy with little experience, I've included my SAT score on my resume, and multiple managers have told me it got them to take me seriously. Programming is unusually meritocratic, in my opinion.

Jody writes:

I asked a guy his SAT score in an interview last week. He billed himself as really bright but not trained in the field. An SAT score seemed like a good way to verify the claim to me.

Mommsen writes:

Because SAT scores are poor predictors of future performance.

Alex J. writes:

You can get a floor of an applicant's SAT scores by finding his alma mater's threshold scores. Graduation and GPA implies IQ + Conscientiousness. It seems that knowing SAT in addition might not add much.

jc writes:

Off the top of my head...

1) Yes, some IO/HR scholars explicitly suggest never using GMA tests like SATs when other valid predictors are available. Iirc, it is directly related to potential lawsuits. Other scholars, though, say that's hogwash - they're fine to use and, also, by far the best predictive tool we have (their words, paraphrased by me; don't have cites handy, but can find them if anyone cares).

2) Maybe the private sector doesn't have to ask about SATs because colleges have already done that for them. Future workers go to school to signal characteristics employers find valuable, and schools in effect sort the wheat from the chaff for firms.

3) Maybe GPA is a better predictor because underlying constructs such as work ethic and social skills (w/ so many classes basing grades on team or group work these days) are built in, in addition to simple GMA?

4) Sociology's brand of institutional theory suggests that isomorphic practices based on custom or tradition are more sometimes more important than whether a tool is actually useful. HR scholars have, for years, tried to dissuade the private sector from it's overuse of unstructured interviews, which have very little predictive validity. Maybe it's just customary to not ask about SAT scores after higher levels of education than high school have been obtained.

sean writes:

in many/most finance jobs (wall st, hedge funds, etc) they will ask for it, and it matters.

Joshua Macy writes:

SAT scores are a reasonably good predictor of performance at college (or at least they used to be...don't have any recent data). But why would employers care about that when they can look at your actual performance at college? If they performed poorly at college despite stellar SATs they're probably not a good prospect; if they were outstanding at college despite blah SATs they are probably a good prospect; if they're among those whose SATs fairly closely matched college performance, the SATs add no useful information.

Jim writes:

Bryan,

Some employers do use SAT scores to screen hires, at least in the financial sector.

http://www.csmonitor.com/2004/0518/p13s01-legn.html

http://www.wallstreetoasis.com/forums/sat-scores-in-interview

http://news.google.com/newspapers?nid=1955&dat=20031117&id=pdIvAAAAIBAJ&sjid=H6MFAAAAIBAJ&pg=4406,3431348

thrill writes:

Since SAT/GPA combination (taken in high school) has a 0.6 correlation to college performance (according to the College Board), it's quite a bit better than "poor". If SAT is weak compared to college grades, then exploration of why is in order. If SAT is strong compared to college grades, then exploration of why is in order. If SAT tracks college grades then both can probably be taken on face value.

Fabio Rojas writes:

A few quick points:

1. Many organizations still use standardized tests for hiring and placement: the military, various gov't agencies (the USPS for example requires basic math skills), etc. Many individuals still signal labor market quality with tests (e.g., "Dr. X is board certified.")

2. Even though standardized tests are a good predictor of performance, they aren't the *best* predictor of performance. Usually the best predictor is prior performance. In other words, if you are hiring for a specific job, it's best to see how the person has done in the past. Tests are second best.

As people progress through their careers, you have better information than tests, especially in jobs that require different kinds of skills.

3. Finally, jobs have specific skill profiles and one size fit all tests (SAT, IQ tests) may not be what you want. So it may be better to resort to other signals.

Bottom line: tests are useful, but they aren't the bottom line.

Micael Edwards writes:

I've been required to submit a transcript to verify my degree and GPA. My SAT & ACT scores are shown on it, and I remember interviewers commenting on them.

JH writes:

By the time you are entering the work world, the SAT was too long ago. Similarly, as you move farther away from graduating college, many employers stop caring about your GPA.

As others have said, over time, there is better information to use.

blighter writes:

Back when I was job hunting, after some time sending out resumes and hearing nothing back I went ahead and revised my resume to put my GRE score on the front page, near the top & got an interview with the very next company that saw it.

I think companies are always looking for proxies to intelligence but have to be as circumspect as possible in order to not run afoul of discrimination regulations.

So, the answer to Joel's clever turnabout is that for-profits *do* care about the information contained in SAT scores, very much, but for a variety of reasons -- prob. not least the maintenance of the higher-education oligopoly's certification power -- we have forbid them from showing they care while allowing universities to openly care. Companies must, therefore, rely on schooling as a proxy for what they actually care about unless the job-seeker gives them the information of his own accord.

Charles Murray has written some interesting stuff on the bizarre way our system is set up to reward credentialism while de-facto outlawing efficient accumulation of credentials.

Jeremy, Alabama writes:

JC's first point covers it. Each of my levels of academic achievement have been completely eclipsed and orphaned by its successor. Once you have achieved a new level, nobody ever, ever asks about the previous level again.

Robert Johnson writes:

Joshua Macy nailed it.

Not only have universities already parsed the SAT data for employers, but SATs do a better job predicting academic performance than on the job performance.

As far as using SATs as a substitute for an IQ test, I'm lost as to what the point is. Is it really that difficult to judge the IQ of an applicant who you are interviewing? If you've gotten burned on that one before, there are many books for interviewers that include questions that are designed to reveal a candidates mental abilities, creativity, etc.

Rolf Andreassen writes:

It seems clear enough: Employers care about job performance, and universities care about graduation rates and the rate at which their graduates get good jobs - both are selling points. So universities want students who are bright and hard-working, and the best available proxies are SATs and extracurriculars. Employers, on the other hand, have the GPA, a more recent filter of harder tasks which also has higher prestige. Done.

Debating the value of SAT scores

So for us, SAT is useful as a predictor in conjunction with high school GPA for predicting first year grade averages, but not much more than that. In fact, for our student population (half are first-generation college students) there's a good chance that the SAT underestimates their potential to graduate, leading them to be 'underpriced' in the admissions market compared to schools that put more emphasis on SAT.

Dan writes:

My pet theory:

Conscientiousness is more important than intelligence for most entry level jobs. GPA is an indicator of the former, SATs are an indicator of the latter. Almost everyone looks at GPA.

When the entry-level job does require a high degree of intelligence, you'll see companies ask for it, i.e. investment banks, management consultants, etc.

SydB writes:

Having spent perhaps ten years as an engineer and manager for a fast growing high tech company, and having interviewed at least a thousand junior and senior engineers during that time, I'd summarize the following from my experience.

1. I developed a fairly precise set of questions related to the topics of systems engineering, programming, computer technology, communications technology, etc. As well questions to dig into interpersonal skills.

2. If someone answered them well, but had a bad SAT/GPA, I'd probably hire them.

3. If someone answered them poorly, but had a great SAT/GPA, I'd probably pass.

So at least from my perspective, the SAT was meaningless.

John Thacker writes:

You can't ignore Griggs v. Duke Power Company, which is still controlling. I think that companies are right to be afraid, particularly when school is such a proxy.

jc writes:

@Fabio is right. Past performance is the best predictor. Assuming accurate information in that regard is not available, employers often look to other predictors.

Schmidt & Hunter's 1998 meta-analysis sums up 85 years of employee selection research. Here are their predictive validities of a few personnel measures (from Table 1):

GMA tests alone .51
GMA + integrity tests .65
GMA + work sample tests .63
GMA + structured interviews .63
GMA + conscientousness tests .60
GMA + peer ratings .58
GMA + reference checks .57
GMA + unstructured interviews .55
GMA + graphology .51

Iirc, that last one, handwriting analysis, was studied in France and Israel; it didn't add any more power. :)

http://www.moityca.com.br/pdfs/SchmidteHunter1998.pdf

Maximum Liberty writes:

I work for a background screening firm. For many of our customers, we verify educational credentials. We can almost always get the type of degree, the date it was conferred, the major and minor fields of study, and whether honors accompanied the degree. We can also usually get a GPA. This normally takes 1-3 days. I won't say how much it costs, since our pricing is confidential, but it's not a lot. My point is that it is easy, cheap, and consistent to get this information.

If we wanted to get SAT scores, it would be a pain. We would probably request a transcript. Since educational records are confidential by law, most educational institutions have stricter procedures for getting a transcript than they do for verifying the information above. (In a verification, we tell them the information that we have and they confirm it.) To get a transcript, we would have to pay a fee. If they can't accept a credit card by phone or internet, we will have to mail them a check. Some schools only allow students to request transcripts. So, figure it takes 2 to 3 times longer to get this information, on average.

Once we get the transcript, it might or might not have SAT scores on it. The SAT went from a 1600 scale to a 2400 scale (and some schools ignore the last third). So, the data you get will be inconsistent.

Each time we touch the case, our costs go up. Many education verifications only get touched once -- we call the school and get the verification. Getting a transcript will almost always require us to touch the case twice -- once to order and once to process the record. On top of that will be the school's fee. So, figure that the cost has tripled.

Now, given the longer time, lower consistency, and higher cost, is the SAT score worth getting as a data point? Others have pointed out that the college degree provides much of the information that an SAT score does; still others have pointed out that other tests might be a better fit for the job. And then there's the litigation exposure to tip the folks who are undecided.

That's why I think employers generally don't go looking for SAT scores. That does not mean that they don't find it useful information when the applicant presents it immediately and for free -- just that they don't make it part of their process.

Max

Boonton writes:

I think its due to one simple fact, SAT scores spoil after a few years. You take the SAT in High School, you then go to college. College tends to be a very formative time and you emerge a different person. What does your SAT have to say about you now? Very little I suspect.

Employers could ask you to retake the SAT's but is there any evidence that SAT score correspond well to employee performance? They were created to mirror future *academic* performance!

I suspect SAT scores might be tapped by employers who hire from the pool of high school seniors and grads who are going to college or are in college, say interns or research assistants etc.

Kevin Bob Riste writes:

Colleges, more so than employers, are frequently compared and ranked according to metrics which take student SAT scores directly into account. Failing to pay attention to this will lead to a steady erosion of applicant quality, regardless of whether SAT scores are even a good metric for performance. It's a real response to a status-generated phenomenon, in short.

Erich Schwarz writes:

The reasons private employers don't care about SATs is that they don't have to care -- the colleges have done the caring for them, so the employers can avoid getting sued under "disparate impact" for using the SAT scores, yet can use admission and passage through college as a proxy for those scores!

It's the equivalent of money-laundering. Call it "IQ-test-laundering".

Really, isn't that obvious?

Steve Sailer writes:

The U.S. military annually hires tens of thousands of new enlisted recruits who typically haven't gone to college, so the military makes an excellent comparison to corporations that mostly hire college graduates.

The U.S. military is obsessed with IQ.

It makes every applicant for enlistment take a huge ten-part ASVAB test, the core of which is the four-part AFQT test. The AFQT is one of the mostly highly g-loaded IQ tests ever developed. (It's the test featured in the middle part of The Bell Curve.)

Since 1992, when the military downsized after the Cold War, the U.S. military has almost completely avoided allowing applicants who score at the 30th percentile or lower on the AFQT to enlist. Only about 1% of enlistees have IQs below 92. During the worst years of Iraq, when recruiting was hardest the military bent its standards to meet its recruitment goals, but it bent more on things like taking high school dropouts and criminals than in taking people who scored lower on its AFQT!

Steve Sailer writes:

Two major parts of American society, academia and the military, are largely held exempt from disparate impact law.

Not surprisingly, both are obsessed with using standardized testing in selection.

The Ricci case involving the New Haven fire department promotion test was particularly ironic because the Yale Law School in New Haven prides itself on having the highest LSAT scores in the country.

Peter writes:

Really does anybody outside academia even care what one's GPA is, much less what alma mater or SAT score. Any place I have worked over the last twenty years only cares about performance and social intelligence, nothing else matters. When I used to be a manager and I would get reports from HR that the person I interviewed had lied every which way from Sunday on their resume it didn't even bother me, I would simply offer them less as they obviously impressed me enough with the interview for me to bite. And if they didn't work out, no big deal, I have relocated and fired people the same day; that's part of life and you are under no obligation to keep dead weight on the payroll. Not sure what folk's obsession on here is with academic or professional credentials, they simply don't matter in most industries outside academia.

Mark writes:

Interesting perspective Peter and the poor sap you didn't hire because you were hiring that liar was probably too dumb to succeed anyway. Someone told you what you want to hear and they impressed you. Save for extreme examples, I wouldn't have the slightest idea on how to judge social intelligence. The fellow I work with now who has the highest social intelligence is a good worker, well liked and has a drug problem. Amen Steve Sailor.

Boonton writes:

Steve,

Two major parts of American society, academia and the military, are largely held exempt from disparate impact law.

So what? What evidence is there that fire fighters who score high on standardized tests actually do better fighting fires and saving people? The New Heaven argument got people ruffled because it sounded fair, give a test and whoever gets the best scores gets the promotion....but no one has ever, to my knowledge, produced any evidence that such testing culls out superior firefighters, or cops etc.

The US military, likewise, is like a mini-planned socialist economy sitting inside our largely market economy. It makes sense that testing would be used because its an efficient way for PLANNERS to ALLOCATE resources.

But as we all should know for all the Hayek talk around here, markets are not centrally planned and decentralized planning is more efficient with most markets. I say the reason employers mostly don't use SATs isn't because of 'disparate impact law' but simply because the test doesn't tell employers much.

Boonton writes:

My theory in a nutshell: Only for a handful of organizations does mass use of standardized testing make much sense. The primary types of organizations where it does make sense are ones that need to bring in a huge number of people and put them into different buckets.

Academia is an example that fits. So is the military and some government hiring. Beyond that I think there are few other areas where it makes sense. Most private companies are relatively small or have relatively decentralized hiring systems where individual managers are able to evaluate prosepctive underlings. There may also be an agency problem. Even if, say, IBM or Microsoft would be better off allocating its labor by using standardized testing, its management would resist. Do you want to pick your team or have the computer assign it to you? Both the military and academia lack the manpower to have generals and professors going to various high schools and hand picking their regimends and classes!

For those trying to pin discrimination law in the US, why not an easy emperical test: What about China and India that have massive growth in employment in the last decade and lots of new company formation. Any evidence these companies are using standardized tests of any type to do their hiring? If not why not?

Steve Sailer writes:

First, there's an enormous amount of data on the positive correlations between scores on general mental ability and job performance across a huge range of jobs. It goes back over several generations, is consistent, and is thus pretty boring because nothing much new ever shows up. It's also extremely politically incorrect because it always shows disparate impact.

The military is particularly interested in it because they don't like dying, and it's easy to die in the military when somebody is incompetent. Air Force pilots, for example, really don't like it when their engines flame out due to faulty maintenance. So, standardized test scores are used in deciding which enlistees will study aircraft maintenance school.

Boonton writes:

Steve,

Most military jobs don't involve dying or even a high risk of dying. Most military jobs are in logistics, support, and administration. The fighter pilot, the special forces team, the stuff that they want to make action movies about requires a lot more than IQ tests or general SAT scores to screen applicants.

And while I agree not dying is a strong incentive military history is sadly not one of brillant applications of 'not dying'. In fact quite often it is a history of finding stupid ways to get people needlessly killed! Your faith in socialistic planning is deeply misplaced, even when death is used as an incentive!

Standardized testing is used because its probably the most efficient way to get stuff done in a centralized planning environment. A huge organization has to apply some objective way to staff jobs like aircraft mechanic otherwise decisions will start to be made by cronyism, favoritism, nepotism and so on.

Compare to the decentralized market economy. How does the owner of a small jet get his engines maintained? Assuming he isn't a mechanic himself, he will likely hire a firm or small company that does engine maintance. That firm wins his trust by signalling that they know their field well and provide good service for the price. He probably wouldn't ask to see SAT scores for the mechanics. What would that tell him? It's not the individuals so much as it is the team that the company assembles. Is the mechanic with the high SAT score being mentored by a master mechanic who is teaching him the nuances of the field that can only be learned by spending years in it? Is the manager trying to cut corners by making the high SAT mechanic do too many jobs in too short a time? In the private sector the owner of the plane would probably find little value in test scores.

Likewise the small company providing mechanic services would find limited value. They are building a team. Two low scoring rookies might be a better fit when matched with a mature seasoned mechanic who will guide them than two high scoring types who will think they know more than the old man and clash against his wisdom.

Now a lot of military functions are outsourced to the market but the military still remains a centrally planned environment. It may be that way out of necessity. It doesn't follow then that the non-socialistic aspects of our economy should mimic the methods of the socialistic ones.

So I'll ask again any evidence that China and India use standarized testing more extensively in their private sector employment? With China especially you can't claim a cultural bias against testing since the Chinese were using standarized tests in Confucian philosophy to staff their civil service nearly a thousand years before any Western student filled in a cirle with a #2 pencil.

rickywcu writes:

I do not believe the SAT is a valid indicator of how someone will perfororm at their job. It does not take into account someones social skills or how well people work in group settings. It is taken before college and is out of date by the time we enter the work force. Many skills may have been aquired by someone in college,whic is also not measured by the test. Many people take classes on preparing for the SAT and other standardized tests. College GPA, recomondations from professors and superiors should be weighted the most, some people just don't take standardized tests as well as they interact with people, which in most jobs today(besides programming jobs) is the most important skill. SAT should not be used as a decision maker in the hiring process.

Jameson Burt writes:

What SAT Really Tests (see b).

a. Several comments amount to employers updating college entrance SAT information with actual college graduation and field of graduation.
At that time, the "prior" SAT information gets set to represent perhaps 1/30 the college graduation information.
This merely formalizes the problem with Bayesian probabilities.

b. In thinking about how colleges use the SAT for student entrance, while family wealth might predict college graduation as well, we idealize that the SAT more justly selects students.

In unintentional practice, the SAT tests a student's ability to accomplish, particularly being prepared and preparing for the SAT.
Those paying $600 for Princeton SAT prep courses gain advantage and contribute to their earnestness at doing well (beating) on the SAT.
Competition in the SAT sometimes seems like an old Greek wrestling match with few rules, where some competitors break fingers.
The best college prep schools reside in Korea,
where two schools outdo all other schools worldwide, getting average SAT scores of 800 on math and about 760 on both English reading and writing.
The SAT characterizes itself not by imaginations in our heads, but by actual student endeavors and college admissions seeming machinations.
Said simply, the SAT is what it is.

For all that, colleges attend less to student's SAT scores than
to the AP courses students take.
A survey of college admissions showed the percentage of colleges giving emphasis to
SAT: 55%
AP courses taken: 80% (as I recall).

Comments for this entry have been closed
Return to top