Bryan Caplan  

The Ethics and Etiquette of Statistical Discrimination

PRINT
Generally Educationally Defici... Michael Pettis on Financial Fr...
No matter what they say, everyone engages in statistical discrimination. (See also here).  Judging everyone as an individual is expensive, and relying on statistical generalizations is a cheap and effective alternative.  You don't clutch your purse when you see a bunch of little old ladies approaching on a deserted street.  You don't offer a policeman a joint.  You don't hire a guy with a mohawk as a receptionist at a law firm - even if he promises to get a hair cut.  Why not?  Because on average, little old ladies don't commit violent crimes, policemen arrest people for possession of marijuana, and guys with mohawks have trouble with authority.

Of course, the inevitable existence of some statistical discrimination doesn't make the practice immune to criticism.  You can grant that it's OK to some degree, but - even if the law is silent - still limited by ethics and/or etiquette.  But precisely what limitations do you think are justified, and why? 

P.S. If your behavior is inconsistent with your principles, please consider the possibility that your principles are unreasonable. :-)


Comments and Sharing





TRACKBACKS (1 to date)
TrackBack URL: http://econlog.econlib.org/mt/mt-tb.cgi/3692
The author at The Liberal Order in a related article titled A Post From My Past writes:
    Given this recent post by Bryan Caplan, I post something I wrote in September of 2005 on my previous blog in response to this past blog post of Bryan's. I'm going to go out on a limb here and bet... [Tracked on July 7, 2010 8:16 AM]
COMMENTS (27 to date)
Phil writes:

Perhaps if the variation within the group is much higher than the group difference from the mean, the benefit (the amount of error reduced by statistical discrimination multiplied by the cost of error) is much, much less than the cost (the discriminated person's inconvenience or displeasure at being discriminated against, plus the discriminator's cost of determining the actual characteristics of the individual).

If the benefits are higher than the costs, discrimination is rude, even if you have a taste for it.

So it's not rude to not invite your cop friend to your heroin party, it's mildly rude to not invite your secular Muslim friend to your bacon party, but it's very rude to not invite your friend of race X to your book club because statistics show that X's read 10% fewer books than non-X's.

Haven't really thought this through, it's off the top of my head.


HispanicPundit writes:

What about when it harms someone else? That seems like a fair limitation.

If statistical discrimination leads me to give someone lower preference in an interview, for example, maybe I should give that person a deeper look to compensate for what could be my statistical error?

Or how about, if it runs parallel to racial stereotypes. For obvious historical reasons.

Not saying I necessarily agree with these reasons, just these are the most often cited objections.

Henry writes:
What about when it harms someone else? That seems like a fair limitation.

If statistical discrimination leads me to give someone lower preference in an interview, for example, maybe I should give that person a deeper look to compensate for what could be my statistical error?

Or how about, if it runs parallel to racial stereotypes. For obvious historical reasons.

Not saying I necessarily agree with these reasons, just these are the most often cited objections.

Statistical discrimination always "harms" someone relative to the world with no discrimination. Guys with mohawks would be better off if people ignored their hair style. Sure, you could give them "deeper looks", but why? That would defeat the purpose of statistically discriminating.

I happen to think that the reasons for different standards mostly lack any philosophical depth and are instead post-hoc reactions to past injustices. We frown upon racial discrimination not necessarily because racial discrimination is currently worse than any other kind, but because of what it used to be like.

David O writes:

We tend to prefer to hire someone who attends an interview in a suit and tie over someone who is wearing a T-shirt and jeans. This is because everyone can make the effort to put on a suit and tie.

I would make a distinction between mutable and immutable characteristics. Then I would recognize a few of those are a little fuzzy because while these characteristics may be changeable, they are very difficult to change.

We should avoid discriminating based on truly immutable characteristics such as race, gender and sexual orientation. Disabilities are an exception in cases where they obviously interfere with productivity. It's a little grayer when we talk about quasi-immutable characteristics like religion (it's not easy to honestly change religion) and lifestyle diseases like being a smoker (it's no easy to quit). In these cases a little discrimination may be okay, but not as much as with completely changeable characteristics.

agnostic writes:

"We should avoid discriminating based on truly immutable characteristics such as race, gender and sexual orientation."

Bryan's talking about statistical discrimination, where you're making decisions under a fair amount of uncertainty. So if your house is engulfed in flames, who do you run to for help in getting it under control until the firemen arrive -- men or women? Obviously men: taller, stronger, faster, more composed under pressure, and more likely to help strangers in need.

If you know that blacks tend to respond far better to some heart medicine than whites do, do you toss aside the patient's race when making the decision whether or not to prescribe BiDil?

No, like Phil said it has to do with the chance of making an error and the size of the consequences of the error. How you came to make the association between group membership and some other trait is irrelevant, except to allow "protected minorities" to continue getting unfair advantages.

Snorri Godhi writes:

A barbarian like me would not know anything about etiquette. As for ethics, I guess it all depends on intentions, and therefore it is completely outside the scope of the Law: off the top of my head, I would consider unethical discrimination that is made with the purpose of hurting someone, instead of helping someone [who could be me]. Of course, when discrimination helps someone, it also hurts someone else; but as I said, the distinction between ethical and unethical is all about intent.

Even to this rule there will be exceptions, e.g. I think it ethical to discriminate against someone who got away with murder, even though this discrimination is made with the sole intent of hurting the murderer.

Mind you, I am neither American nor German, so the issue of racial discrimination does not loom large in my mind.

David O writes:

@agnostic

"Obviously men: taller, stronger, faster, more composed under pressure, and more likely to help strangers in need."

Fire departments should select employees based on these qualities. I'm not sure there are these differences between employees who have gone through a selection process. If the fire department doesn't have an adequate selection process (perhaps they accept too many women), perhaps statistical discrimination may be justified.

"No, like Phil said it has to do with the chance of making an error and the size of the consequences of the error."

But it needs to be weighed against the psychological harm it causes, which in most cases outweighs these characteristics. A person who is discriminated against, particularly because of characteristics that may be products of residual racism/patriarchy/homophobia by the dominant group, will understandably suffer more than just the foregone job, and rebel.

I would also argue that from a political economy standpoint, libertarians should oppose statistical discrimination. The idea of equality of opportunity, that anyone from any background can become successful, is one of the best safeguards against redistributive socialism. Destroy it and watch how the politics becomes a discussion of class warfare.

I'd also wonder whether you hold this position because you're privileged or if you'd choose it coming from a veil of ignorance.

Discriminating in medicine for the patient's benefit is different than discriminating for your own benefit. There is no psychological cost in the former, there is a large psychological and social cost in the latter.

stephen writes:

It seems the "wrongness" of discrimination is inversely proportional to its usefulness. The more you know about the individual(s) the less you need statistics. It would be silly the assume someone is going to take on the average value of the population when you know they are in the 95th percentile, for instance. Usually we don't know this, so statistical assumptions are fine, as in all of your examples.

Biagio writes:

If I can quote the Onion: "Stereotypes are a real time saver".

ajb writes:

Not only do I believe in statistical discrimination, I support racial profiling at airports to ease security hassles. And unlike Bryan I support statistical discrimination to determine who should enter the country. Where the benefits to American citizens are low in dollar terms I'm especially in favor of citizen preferences being the main determinants of who gets to enter given the inadequate measures of social externalities of immigrants from low human capital societies.

Oliver Beatson writes:

I think that discrimination is probably only unethical (if at all) when it is intentionally harmful.

We can easily form the dichotomy, that there must exist two sets of traits according to which it is possible for a given employer (E) to discriminate against a given person (P) for a given job (J):

  • i) traits that (E) believes to statistically affect (P)'s ability to do (J); and:

  • ii) traits that (E) believes to be irrelevant to (P)'s ability to do (J).
Is it not the case that under any rational employer's favoured circumstances, the average employer (given that one must have at least some sense to get to a position of employing others) wouldn't choose to discriminate according to the latter set, and having exercised the extent of their beliefs in informing the composition and content of each set, discern the optimal employee according to the content of these sets to the best of their means? I would have thought this obviously true. To impose the requirement that the employer disregard their beliefs regarding any set of arbitrary forbidden discriminants is surely a tortuous request of self-harm, especially given that the same effects will surely come from the free distribution of information on the market, given the fact that it is in the interests of any employer to optimise the best discriminations and exorcise the worst.

(I believe this mechanism and these principles function outside of employment discrimination. I believe the fact that the argument doesn't name arbitrary traits lends it favour. The whole argument disregards any utilitarian calculations of hurt feelings, which might significantly skew the ethics, although I intuit that there are equal and opposite reasons why hurt feelings are nothing less than virtuous when (if only when) they are the product of free actions.)

It is surely only when the employer disregards the data available (the product of their beliefs), for the sake of some such unreasonable emotion as spite, that we can consider an unethical action to have occurred, because it is then that discrimination has occurred in the absence of reason and in the favour of unthinking, mutual destruction.

Zdeno writes:

How about this: Statistical discrimination is only appropriate when you are rationally applying statistical discrimination.

For example if my house is on fire and I have to choose one of two people to come help, and I know nothing about them but their genders, I will choose the man. If I am staffing a fire department, I can probably devise a program of physical and mental testing that eliminates any effect of the gender dummy on predicting fire-fighting prowess. Given the natural differences between men and women, my fire department will wind up predominantly male, but there's no reason female outliers shouldn't be hired.

Randy writes:

Why involve the law? That is, just let people discriminate as they wish, and understand that the result will be what it should be. The problem isn't when individuals discriminate, but when the law discriminates.

Zdeno writes:

Regarding immutability: In a game of pickup basketball with unknown players, I will pick the taller, ceteris paribus. Unethical? Also, I only date babes. IOW, I discriminate based on immutable markers of genetic fitness. It is morally wrong to do so?

I've already offered my suggestion for an ethical criteria for when SD is appropriate, but I'll try and clear up all this cognitive dissonance with a theory of when SD is acceptable in reality:

Statistical discrimination will be forbidden when outlawing it results in the transfer of power and privilege to people and groups allied with the political left.

If you disagree, perhaps you could come with a counterexample...

megapolisomancy writes:

Most statistical discrimination involves the withholding of a benefit, not imposing a harm. A free society should permit such discrimination because it improves rational decision making.

Ben O'Neill has done some useful work on statistical discrimination and anti-discrimination laws from a libertarian perspective.

Carlsson writes:

Statistical discrimination makes a lot of sense if rationally applied, but think of the data requirements in order to apply the theory correctly. You need a whole lot more than the means of two (or more) distributions, you'll need variances too, along with skewness. You'll need to be able to compute probabilities of Type I and II errors, obviously.

If you can't do that, you're just going by hunches, and don't confuse that with statistical discrimination. Not that hunches are bad, rules of thumb can be very useful when you have limited information. But if you place a huge value on one type of error (e.g., accepting when false), hunches can be dangerous.

What most people call statistical discrimination is just the application of rules of thumb, based on little or no data. Prejudice is not statistics.

Philo writes:

You wrote: "Judging everyone as an individual is expensive, and relying on statistical generalizations is a cheap and effective alternative." But what do you mean by "judging someone as an individual"?

The examples of "judging" you gave were examples of *predicting behavior*. True, you can wait to see what someone does; but that isn't *predicting*. If you want to estimate beforehand what someone will do, you must rely on correlations between qualities present at an earlier time (now) and actions at a later time (the future). Predictive inference *must be* statistical.

Predictive inference does vary in *breadth*. If all I know about someone is that he has property P, and I want to know whether he will in the future display behavior B, I must rely on the correlation between P and B. If I also knew that he had properties Q, R, and S, I could rely on the correlation between P&Q&R&S and B; this would generally give me a firmer basis for my expectation about B. But getting that extra information about his properties might be costly. (Is narrowing the class to which someone is assigned what you meant by "judging him as an individual"?)

Relying on statistical inference is not unethical; it is simply rational. But etiquette will sometimes require that we conceal some statistical inference we have made about someone from that person; we ought not needlessly to hurt other people's feelings.

John writes:

It's perfectly fine to discriminate. Rules of thumb are fine. Incorrect rules are fine. Predjudice is fine.

In other words, I have no business telling you what method you must use when you make decisions. Poor methods may lead you to inefficient outcomes but that's your problem, not mine (barring force or fraud, but those aren't under consideration here).

This seems to me to be the obvious position for any libertarian.

David C writes:

We should avoid discriminating based on truly immutable characteristics such as race, gender and sexual orientation.

Actually, all of those things can be changed. Blacks can wear makeup or bleach their skin. Men can have surgery to become women. Gays can take hormone supplements to change their orientation although the effects of such supplements are debateable.

The thing that doesn't make any sense to me except from a respect for tradition stand point is that people with tatoos are so widely discriminated against in the workplace.

Steve Sailer writes:

Our society has gone backwards at avoiding unfairness to individuals. For example, federal civil service examinations as a way to avoid hiring on ethnic, nepotistic, partisan, or corrupt grounds began under the Chester Arthur Administration, and was progressively rationalized up through the carefully validated PACE exam introduced in the 1970s.

The outgoing Carter Administration through out the federal civil service exam in a consent agreement to the Luevano Case in January 1981, promising that a federal civil service exam would be reinstated once somebody came up with one that didn't have Disparate Impact on legally protected groups. That was 29 years ago and it hasn't happened yet.

liberty writes:

Philo has it exactly right. The question in the post is about prediction, which one can only do by using information about past behavior either of that person or of the qualities that the person has (and their correlation with past behavior of others).

If you do not know the person then you do not have past experience of his or her behavior (and hence cannot "judge the individual" in that way), so you really only have statistical inference based on known characteristics. These may be something we might think of as "judging the individual" if the characteristic is "he seems nice because he said hello and smiled when he walked in" but this is still a statistical discrimination (e.g., "people that say hello and smile are usually nice in other ways, hence probably nice to work with, hence a better hire...")

If the characteristic is less "personal" in some way, it may *seem* more like statistical (or plain-old) discrimination - e.g., if its a broader category like gender, hairstyle, age, height, weight, race, etc.

I think the ethical question probably comes in, as Phil said at the top of the thread, when variation is high and the cost/benefit is skewed. It's not about innate vs. not as sometimes this makes sense, as some have pointed out. The little old lady really probably isn't going to mug you, and the cost/benefit and variation indicate that the discrimination is probably OK.

Matt Zwolinski writes:

Frederick Schauer's _Profiles, Probabilities, and Steretypes_ provides an excellent discussion of some of the legal/moral issues involved.

Tracy W writes:

Ajb: Not only do I believe in statistical discrimination, I support racial profiling at airports to ease security hassles.

I'm always puzzled by the support for racial profiling at airport security. Let's say that security at airports stops paying attention to elderly Asian ladies. The moment the terrorists find about is, their obvious response is to plant the bombs/guns/knives on elderly Asian ladies. One horrible idea - find an elderly Asian grandmother, kidnap her gorgeous two-year-old daughter, and threaten her life unless grandma carries a bomb/gun/knife onto the plane. Send elderly Asian grandmother one of the her granddaughters' fingers as a sign you're serious. Perhaps to reduce the odds of grandma going to the police, disguise it somehow and make up some cover story so grandma can hope that she's not actually sacrificing lots of people's lives for her granddaughter's, and rely on self-deception to carry her through. And I think I got this idea originally from a movie trailer, so it's very likely to be something the terrorists will think of.

William L. writes:

The real problem with this idea is that it usually relies on flawed logic and observations and frankly is a way to avoid being successful.

The "discrimination" is highly dependant on the assumptions that are used and the questions that are asked. Generally people do not take the time and consideration in asking the right questions or realizing the limitations of the results. Need I point out that correlation is a necessary but not a sufficient condition to make causal inferences with reasonable confidence? Need I point out the problematic issues with factual reporting, data collection, bias, and possibilities of self-fulfilling predictions?

I hope that I don't have to.

By basing your assumptions on what the majority thinks and does, you simply maintain the status quo. If you want to be average this is the way to go - but frankly some of us would rather use a nuanced approach so that we can tease out the possibilities in life and account for black swans, white knights, and little old punk grandmothers who will make our law practices rock despite having checkered pasts.

W.L.

v writes:

Mohawks aren't exogenous.

Æternitatis writes:

William L. writes (italics added):

Need I point out that correlation is a necessary but not a sufficient condition to make causal inferences with reasonable confidence?
Need I point out that causal inferences are completely irrelevant to the issue of statistical discrimination?

It appears I do: For observable surface characteristic A to be a valid tool in making estimates about hidden characteristic B (i.e., to be able to accurately statistically discriminate), all that is required is that A and B are correlated.

Whether A caused B, B caused A, they were both caused by some external factor C, or whether it is a complex, endogenous admixture of all of the above, is completely irrelevant.

Practitioners of statistical discrimination should take offense at your implication of that they believe the contrary.

Æternitatis writes:

Tracy W writes:

I'm always puzzled by the support for racial profiling at airport security. Let's say that security at airports stops paying attention to elderly Asian ladies. The moment the terrorists find about is, their obvious response is to plant the bombs/guns/knives on elderly Asian ladies.

I've always been puzzled that this counter-argument is repeated and considered strikingly brilliant by otherwise sensible people. Let me explain why it is not.

To start with what we know: The vast majority of operatives of terrorist organizations seeking to murder Americans (and others) identified from past attempts are young Arab Muslim men. And most of the rest are at least three of the four.

From this it follows that persons of this description must be among the most plentiful, available, and easy to use resources available for such organizations. In other words, this demographic has the ratio of reward (in terms of dead or terrorized infidels) to organizational effort (in terms of money, time, and other resources). If another demographic had a better ratio, the terrorist organization would already have switched to them and costlessly increased their effectiveness.

That does not mean that there aren't blond-haired blue-eyed Norwegian grannies who converted to Islam and are now all hot to kill for Allah—surely there must be. Or Asian grannies from your movie trailer. What it does mean is that they must be much rarer, harder to use, and generally more expensive means of murdering infidels. If Osama Bin Laden had a granny brigade at his disposal he would already have unleashed it, rather than relying on such a non-diverse and identifiable set of operatives.

So what happens if we institute profiling? Young Arab Muslim men become much more likely to get caught and hence less useful tools. At the same time grannies, given the same amount of total enforcement effort and hassle, become at least a little less likely to get caught and hence more useful tools.

These two effects, while in opposite directions, are not of equal size: The terrorist organization is much more hurt by the declining effectiveness of its chief type of asset than it is helped by the increased effectiveness of a type of recruit it already hardly used because of other reasons. So while there is presumably a shift in the terrorist organization's allocation of efforts, it will on net be worse off (and we better off) after the shift.

How much this hurts them (and helps us) depends on how much more expensive (in the above sense) grannies were for the terrorist organization than young Arab Muslim men before priofiling. If currently there was only a slight advantage to using the latter, then the net harm on terrorist organizations of profiling would also be small and so would the net help for us.

While that is possible, this seems unlikely. Terrorist organizations are likely to very much prefer using their current demographic (and hence be hurt a great deal by profiling) for several reasons.

First, there is a substantial individual dispersion within each demographic. If the difference in average between recruits from the two demographics were small, the distributions would likely overlap and grannies would be underrepresented in the sample of terrorist, but still a substantial presence. Instead, we see them hardly at all. That means that the difference in average between the two demographics is likely large and shifting from one to the other would impose a large cost on the terrorist organization.

Second, the difference in inherent characteristics between the demographics as to suitability as terrorist agents are likely to be substantial. While there likely are some socially contingent factors, others are so fundamental and significant that I'd estimate them to have a large effect. Lutherans are inherently less likely to consider the majority population of Europe and the U.S. to be infidels worthy of death than Muslims are. Arabs and Muslims today have a far greater cultural acceptance and valorisation of suicide attacks and even terrorism in general. Norwegians are far less likely to feel—rightly or wrongly—profoundly racially alienated from Americans than Arabs are. And throughout history and almost all known cultures, the young and men have always shown far greater propensity—again for good or ill—for the sort of physically dangerous but potentially glorious pursuits like suicide terrorism.

Finally, and most importantly, it does not matter if my predictions—admittedly based in part on broad cultural stereotypes—for the effectiveness of racial/cultural/gender/religious profiling are correct because the practice itself will demonstrate the correct point for stopping:

At the point of maximum harm to terrorist organization efforts to kill us—and for us the optimum amount of profiling—grannies and young Arab Muslim men and everybody else will have become equally (in)effective. At that point, we would expect to observe terrorist organizations to use all demographics equally. By no means should one pursue profiling beyond that point as it would only help the terrorists.

In other words, keep profiling until the demographics of apprehended terrorists look like America. If it is good enough for the U.S. cabinet, it is good enough for Al Qaeda. And just think how pleased their diversity management consultants will be!

And I think I got this idea originally from a movie trailer, so it's very likely to be something the terrorists will think of.
Sure sounds like it. Yet, dare I suggest that movie trailers are not necessarily the most accurate tool for assessing the opportunity set of terrorist organizations?
Comments for this entry have been closed
Return to top