Bryan Caplan  

Why Experiments are Misleading

PRINT
Eternal Temptation... Health Care Op-Eds...

Tyler Cowen discusses an interview with Harvard's Mullainathan, in which he talks about a bank's randomized marketing experiment:

“What we found stunned me,” he says. “We found that any one of these things had an effect equal to one to five percentage points of interest! A woman’s photo instead of a man’s increased demand among men by as much as dropping the interest rate five points! These things are not small. And this is very much an economic problem. We are talking about big loans here; customers would end up with monthly loan payments of around 10 percent of their annual income. You’d think that if you really needed the money enough to pay this interest rate, you’re not going to be affected by a photo. The photo, cell phone lottery, simple or complicated table, and deadline all had effects on loan applications comparable to interest. Interest rate may not even be the third most important factor. As an economist, even when you think psychology is important, you don’t think it’s this important. And changing interest rates is expensive, but these psychological elements cost nothing.”

But if that's true, then wouldn't every bank already be maxing out on psychological elements? In actual markets, every glass has already been labelled "half full." If you want to loan more money, then, you have to do something costly - like cut interest rates.


Comments and Sharing





COMMENTS (10 to date)
Karl Smith writes:

Perhaps also the pyschological elements serve as signals whose information content is diluted as more and more firms use them.

conchis writes:

"But if that's true, then wouldn't every bank already be maxing out on psychological elements?"

This strikes me as one of those ridiculous sentences that only an economist could come up with. We have this strange instinct that no new information can possibly be both true and useful, because if it were, people would already have exploited it.

But banks would already be maxing out on psychological elements only if what Mullainathan says is true and banks know it's true. The thing is, if they knew it were true already, then they would have had no need to be involved in this experiment.

Zac writes:

I agree with conchis whole-heartedly.

Given what we know about psychology.. the assumption that any firm is "maxing out on psychological elements" in almost any industry.. is shaky, at best.

tiltman writes:

Why do the previous commentors find it hard to believe banks may already be aware of these "psychological effects"? How much money do banks spend developing marketing materials and strategies?

We don't care if banks explicitly know these effects. How do they behave? How do they allocate their resources?

We should also take note that this experiment was not conducted in a US market.

Bob Knaus writes:

At the risk of sounding a bit un-PC, I think the key phrase is "developing countries" which is where this guys specializes in marketing plans. These people haven't developed marketing burnout yet, unlike consumers in the US who get 3 direct mails and a dozen spams every day offering financial services.

One of the reasons we enjoy looking at vintage ads is that their marketing ploys are so transparent as to be endearing. Grandma bought soap because it FLOATED??? HAHAHA!!!

Not to worry, more than just the pets.com sock puppets will be fodder for the laughter of your grandchildren.

rakehell writes:

"But if that's true, then wouldn't every bank already be maxing out on psychological elements?"

In every market there will be unsophisticated players. Suckers in other words. So some banks may be ignorant or even a bit "above" these tactics. The savvy player nowadays can search the internet looking for these spreads. For instance, a year ago I was shopping for CDs and seeing some vast differences on rates.

"This strikes me as one of those ridiculous sentences that only an economist could come up with. We have this strange instinct that no new information can possibly be both true and useful, because if it were, people would already have exploited it."

Yes, efficient markets is the butt of the old joke about the two University of Chicago professors who see the ten dollar bill laying on the ground. But somewhere in one of my texts an author points out that it is the action of all the various players in the market, some wise, some not so wise, that creates the efficient market.

conchis writes:

I don't doubt that some banks are aware of, and consciously exploit psychological elements. But Bryan's argument requires that "every bank" be doing so, and we have a fairly clear counterexample to that: the bank in question was neither aware of, nor (presumably) investing in this sort of thing.

Unless, of course by "bank" we were supposed to read "US bank".

Even then, the point remains that we don't know whether US banks are already aware of these effects (or, more implausibly, are acting "as if" they were). Whether they are or not is fundamentally an empirical question. But too often economists just assume, as I think Bryan is guilty of doing here, that people/firms must be aware of stuff - because that's what our models say. That strikes me as both a very strange, and quite dangerous way of reasoning.

AlanR writes:

Bryan echoes the economist in the old joke who won't exert the energy to pick up a dollar in the road, because it can't really be there, because if it were somebody would have picked it up already. In other words, if somebody does some properly conceived and conducted research that conflicts with a prior theory of yours, maybe you should think about the theory rather than simply reiterate it. Here, the problem with the theory seems obvious: Bryan accepts for the sake of argument that the market is made up of irrational humans but not that the banks themselves are no more than communities of the same. There is little about the history of banks and banking anywhere which would lead you to assume that today's banks are temples of rationality - even if theory tells you that they should be, and even if the very worst do indeed go out of business.

tiltman writes:

I'm not suggesting that more study isn't warranted. In fact I think it would be very interesting, and it is an empiricle question. But in doing such a study in a competitive market my hypothesis would be that the interest rate begins to matter much more. Theory doesn't prevent me from being curious about this, it tells me to be a litte cautious in excepting this one study's implications on all markets. I think that was Caplan's point.

Mike writes:

Bryan, your comment doesn't explain your title.

You suggest that all possible practical knowledge is already known and applied in practice. If that were true, then experiments would be pointless rather than misleading.

Comments for this entry have been closed
Return to top