David R. Henderson  

Statistical Significance is Not the Same as Economic Significance

PRINT
The Corruption of Politics... Balance Sheet Anxiety (What al...
They [Hong Ru and Antoinette Schoar] find that less-educated households were offered higher late fees, over-limit fees, and default penalty rates, as well as more upfront inducements, such as low introductory APRs, cash back, and waivers of annual fees. In contrast, more highly educated households were offered cards with front-loaded features such as stable regular purchasing APRs and low late fees and over-limit fees.
This is from Deborah Kreuze,"Do Credit Card Companies Screen for Behavioral Biases?", The NBER Digest, September 2016.

The whole study is here (although it is gated).

The authors' answer to the title question is yes.

It makes sense. If people pay less attention to complex terms offered by credit card companies, or any other companies, it is not surprising that credit card companies want to take advantage of this. However, there's one big offsetting factor. Even though credit card issuers want to take advantage of the gullible, they are competing to take advantage. So whereas a monopoly credit card issuer might be able to make hundreds of dollars more annually on gullible credit-card holders, firms competing with each other would probably not be able to. Why? As competitors attempt to take advantage of the gullible, competition drives margins down. How high or low? It's an empirical question.

Unfortunately, the authors of the study don't answer that empirical question. In their various tables of results in the appendix, they focus solely on statistical significance. But how economically significant are the results? Fortunately, although the authors of the study don't answer that, the NBER writer who wrote up the results, Deborah Kreuze, does.

She doesn't answer it directly in her text, but she inserts a figure that cannot be found in the original 58-page NBER study. The source of the figure is the authors' calculations, calculations that are not apparent in the study. As expected, the biggest gap in late fees is the gap between those who did not graduate from high school and those who are post-college graduates. Similarly with the "over-limit fee."

And what are the gaps?

Drum roll please.

. The gap in late fees between those who did not graduate from high school and post-college graduates was (from eye-balling the figure): 50 cents. Not 50 dollars. 50 cents.

. The gap in over-limit fees between those who did not graduate from high school and post-college graduates was (from eye-balling the figure): $1.60.

Statistical significance does not equal economic significance.


Comments and Sharing


CATEGORIES: Competition , Finance




COMMENTS (17 to date)
Greg G writes:

Even if the fee difference had been higher there is good reason to think that less educated customers are likely to be a worse credit risk than better educated customers. That alone would justify some difference in fees.

Brad D writes:

I can't help but wonder if the omission of these facts was intentional? No good economist evaluating "fees" would omit the magnitude of said "fees" unless the study was done for other than academic purposes.

Jeff writes:

This is the precise argument made in a series of articles and books by the brilliant economist Deirdre McCloskey. Back in the 1980s. How has this not sunk in yet? I met the fine lady at the Minnnesota Economic Association a couple years back, where she was the keynote speaker. I remember all the papers presented that day showed statistical significance but not economic significance. I guess they didn't know that McCloskey would be there?

In fact, this is the main argument against certain econometric techniques like those applied to panel date and really big data sets. It's easy to get statistical significance because you can get really precise point estimates (low standard errors). However, the parameters can still be really small even if precisely estimated.

Economists are not statisticians and we have to always remember that we're dealing with real people making real decisions. Size matters.

Ken P writes:

I run into a similar thing as a scientist. Often people present resukts that are statistically significant but not biologically relevant. Sometimes they are so caught up in supporting the story they are so sure exists thatvthey dont adequately question the meaning if their results.

David R. Henderson writes:

@Brad D,
I can't help but wonder if the omission of these facts was intentional?
My guess is that it was.
No good economist evaluating "fees" would omit the magnitude of said "fees" unless the study was done for other than academic purposes.
There I disagree and think the opposite is more likely the case. There’s such pressure in academia to publish statistically significant results that many academics focus on just that. My guess is that Deborah Kreuze, the person who did the write-up, wanted to be able to tell a general audience why the results mattered and asked the authors to translate it into dollar amounts. The results were puny but the study was done.

Scott Sumner writes:

I'm glad someone spends the time digging through the data to establish these facts. Too bad the original authors made it so difficult.

JLV writes:

Two things:

1) The figure in the NBER digest visualizes the same information as figure 1 in the paper (but looks much better.)

2) FWIW, the authors do make claims about economic vs. statistical significance when talking about cash-back programs (p. 26 of the NBER working paper)

Mario Rizzo writes:

If you find more of these, you are "duty bound" to let me and others know.

Nick writes:
She doesn't answer it directly in her text, but she inserts a figure that cannot be found in the original 58-page NBER study. The source of the figure is the authors' calculations, calculations that are not apparent in the study.

See Table 4, where the numbers from which it appears the digest figure was made were pulled. Substantive issues aside, it is not correct that these calculations are not apparent in the study. I don't have the time to read the paper in detail but it seems the authors were reasonably transparent.

The more important issue is the claim that the numbers are not economically significant. The question of what is economically significant is a valid question to ask. The authors should have some discussion of this point, although it seems there is a prior literature, so perhaps they felt it was unnecessary.

But you also need some discussion on this point if you want to argue that 50 cents (or $1.50 in over limit fees, or a 0.5% point increase in the APR, from the paper) are not economically significant.

Why, exactly are these not economically significant? Back up what you say! What would count as economically significant, to you, and why did you choose that number? It's not obvious to me at all why 50 cents is economically insignificant.

Let's say an unsophisticated consumer has a credit card for a year, pays the late fee and over-limit fee every month, and goes into $2000 worth of credit card debt. Then a stupidly rough back-of-the-envelope calculation suggests that that consumer pays an extra $30 a year due to this discrimination. Let's say 10 million credit card holders are affected, then this discrimination results in $300 million extracted from unsophisticated consumers.

Are these numbers realistic? Is that a big number? I have no idea! But if I were interested in criticizing their work on the basis that these numbers are not economically significant, I'd do a bare minimum more work so I did have some idea.

Nick writes:

By the way, you respond to Brad D. by questioning the ethical conduct of the researchers. Regardless of whether you're right, that is bad form, but especially so here since their paper appears to contain the magnitudes, and the digest made available for the public is quite explicit about the magnitudes.

For all I know, you're right, but again -- bad form! Make sure you have a better understanding of the paper before lobbying accusations.

Mark V Anderson writes:

I second Nick's request. When I read the original posting, my first thought was 50 cents per what? $1.60 per what? I think Nick assumes this is per month. IS this correct? Even $30 per year sounds pretty tiny, but it is somewhat significant for the very poor. IT would be worth stating what is significant. Although the point of David's posting is correct in that the original study should have also had that information.

I also wonder if Greg G is correct that the higher fees are justified because of higher credit risk. But there isn't enough information here to know.

David R. Henderson writes:

@JLV,
1) The figure in the NBER digest visualizes the same information as figure 1 in the paper (but looks much better.)
Thanks for calling my attention to figure 1 in the paper. It doesn’t give quite the same information because it omits a crucial item: units. There is no dollar sign on the y-axis in the authors’ Figure 1.
2) FWIW, the authors do make claims about economic vs. statistical significance when talking about cash-back programs (p. 26 of the NBER working paper)
Right, but they don’t tell us why they think those numbers are economically significant.

David R. Henderson writes:

@Nick,
By the way, you respond to Brad D. by questioning the ethical conduct of the researchers.
Not quite that extreme. I did say “I guess” because I was not, and am not, sure. But you’re right that it’s bad form even to make that guess.

Swami writes:

I would like to take exception with this study or its alleged conclusions at a broader level. First, I have zero experience in underwriting or marketing credit cards. As such I may be missing some of the nuances of the study and conclusions. However, I have decades of expertise in underwriting and marketing insurance products.

In marketing a product offer, the profitability of the offer is a factor of the take rate, the revenue stream of the customer, the retention (persistence) of the customer, and the profit. All of these are going to differ dramatically based upon the characteristics of the customer. Uneconomic offers will not be mailed long term.

In insurance, different segments are clearly attracted by different things. Some like low initial premiums, others place more value on added benefits such as air miles. You would be crazy to mail an offer for travel benefits to people who rarely fly anywhere. And you would be crazy to mail an offer with high upfront fees to people with short time horizons and little disposable income. By crazy, I mean that the offer would be an economic failure. It wouldn't pay for itself, thus a company would not want to do it, and if they did try it they would cancel it over time as they intensely study the economics of every marketing campaign, especially mailers.

You are right that competition between companies would tend to attract the profit per customer to whatever the risk adjusted return to capital is. But note that the optimal structure of an offer will differ dramatically for both the customer and the company. There are multiple ways to get an acceptable rate of return. For customers who are tight on money and have shorter time horizons the optimal package may very well be low up front fees, high late fees and overcharges with no added fringe benefits.

As to the question of whether marketers understand behavioral economics. Of course we do. You could just ask us. I used to pay to have Dan Ariely teach our marketers his insights. I considered his book on irrationality required reading for my product development team and the top marketing experts. However, the empirics of direct marketing almost make the knowledge irrelevant at least over the long term. Over time, companies only have to track acceptance rates, customer premium and persistence, and so forth. This is all done on a smaller than zip code level with every bit of demographic info we can possibly get (that is allowed by law).

Let me be really clear. For an offer to continue over time, it needs to be a product that is profitable, that customers like. Different customers like different things, and products will differ in profitability based upon who it is sold to. This study just profiles the obvious. People who were unable to graduate high school are patently obviously not the people to get high up front fee credit cards, with mileage teasers, and no penalties for not paying monthly. Similarly, a high late fee card, low initial fee card would never make money if sold to people who never pay late.

One last comment, do note that customers with problems keeping to long term financial reliability BENEFIT in some ways from higher fees and overcharges. It increases the net cost of such actions and thus helps them to avoid the even worse pain of losing all control over their balance. It is the rational, logical, even Pareto optimal way of offering the product to the company and customers mutual satisfaction.

But again, I may be missing some nuance in credit cards, I know insurance.

Swami writes:

Here is a more concise way to think of the issue...

Consider a Venn Diagram which includes the following two groups:

1). Offers which the company can make money on over time, and
2). Offers which the consumer will like enough to respond to and contractually agree to

When you combine the two requirements, you get a certain type of offer which is a win win with some segments and other offers which are win wins with other segments.

Nick writes:

@Swami

All interesting points, and the paper itself has an in-depth discussion of many of these issues. The paper is gated, but by googling the title you can find an older working paper. A broad comment: I think you're right to be puzzled that economists do such statistical work but do not seem very interested in actually asking the people who work in these industries what they do.

You say as insurance underwriters you would condition on zip code and demographic info. as allowed by law. I have one point and one question: First, what are the relevant laws, i.e., what can and cannot you condition on? I didn't see a discussion of this in the paper but it seems interesting.

Second, the authors condition only on educational attainment. To the extent that these are just imperfect proxies for all the other demographics which credit card offers condition on, we should expect the magnitude of the statistical estimates in the paper to be biased downward.

Swami writes:

The laws in property casualty insurance differ dramatically by state as insurance is almost exclusively controlled on a state basis. In general though no company underwrites auto insurance by zip code (they do underwrite property based upon proximity to hazard or risk exposure such as flood or hurricane). However, if allowed in that state, rates differ dramatically by zip based upon actual loss experience. Rates also differ in auto based on age, sex, marital, driving record, car type, limits and MOST IMPORTANTLY based on CREDIT SCORE.

Most companies, where allowed by the state, underwrite and price using proprietary algorithms based on the details of what they call financial responsibility. I assure you the significance of this factor is simply mind blowing. Companies without credit reports simply can't compete with companies that use them.

It is important to stress that underwriting and marketing are two different strategies but that both can lead to the same outcome. You can agree to accept someone from anywhere, but only mail offers and advertise in target zip codes. Agency companies don't have to redline an area, they just avoid putting agents (or more accurately agents choose not to set up location) in that area. Again, assuming this is kosher in that state.

I will read the full early draft, but will just add now that there is nothing intrinsically superior or morally preferable about selling credit cards with higher or lower fees, especially since the company has to get the required return of lost fees by higher interest rates and or annual fees. The paper highlight seems to imply that high fees are wrong or deceptive. I don't believe that holds up to scrutiny. I believe if given a choice many, many customers would choose the lower annual fee and higher risk of late fees. I enforce their freedom to do so and the freedom of the company to offer it. I believe it is superior to a world where it is prohibited.

Comments for this entry have been closed
Return to top