Arnold Kling  

An AP Stats Lecture

Hearing Post... No Further Questions, Your Hon...

Because yesterday's hearing ran so late, I missed my AP stats class. Here is the way I would explain what went on to my students.

At Freddie Mac, we used the concept of Type I and Type II errors. If a $100,000 loan defaults, you lose about $50,000. If the borrower pays on time, your profit is about $100. So approving a bad loan is a much costlier error than rejecting a good loan. In terms of type I and type II errors, we have:

DecisionLoan RepaysLoan Defaults
Approve LoancorrectType I error
Deny LoanType II errorcorrect

Given the costs involved, you want to set alpha (the probability of type I error) to a low number, say .0001, or 0.01 percent. That is, you want to experience very few defaults.

Every applicant has a credit score, called a FICO score (FICO is an acronym for Fair, Isaac Company, the name of the leading credit scoring firm). Oversimplifying, let us say that a credit score of 660 is the cutoff that gives a loan a default probability of .0001.

In some sense, the null hypothesis is that the loan should be denied. You reject the null hypothesis (approve the loan) if the FICO score is above 660. You fail to reject the null hypothesis (that is, you deny the loan) if the FICO score is below 660.

Using this rule, If the FICO score is above 660 but the loan later defaults, you will commit a Type I error. If the FICO score is below 660 but the loan later gets made by another company and the borrower repays on time, you will have committed a Type II error.

Suppose that a loan with a 630 FICO score has a probability of default of .06. Since you deny all of these loans, you have a probability of a Type II error on these loans of (1-..06) = .94. We say that the probability of Type II error is beta. We say that (1 - beta) is the Power of the test. We calculate Power based on a specific alternative, namely the alternative that the FICO score is 630. In this example, the power is pretty low--we have a pretty high probability of making a Type II error.

Other things equal, if you try to make fewer Type II errors, you will make more Type I errors. For example, you could reduce Type II errors by approving loans with FICO scores of 630, but that means you will experience more Type I errors.

In 2004-2007, Freddie Mac and Fannie Mae lowered their lending standards. This meant approving loans with lower FICO scores, along with other methods (mortgage underwriting is based on a number of factors). At the hearing, Congressmen were asking (not in these words), why did you succumb to pressure to reduce Type II errors and increase Type I errors?

The CEO's denied that they were under pressure from Congress to approve more loans to meet "affordable housing" goals. Assuming that they were not lying under oath, that would say that the CEO's never received a phone call from Congressmen asking them to approve more loans. However, one suspects that they felt pressure indirectly, because they know that they need friends in Congress, and the friendship of some Congressmen, particularly from urban districts, depends on how well Freddie and Fannie serve those markets.

The CEO's said that the market was changing. In the past, when they made Type II errors, nobody saw them. If Freddie and Fannie denied a loan, then the loan was not made. Wall Street was now making loans to borrowers with lower FICO scores, and these borrowers were not defaulting. The Type II errors that Freddie and Fannie were making were more visible than they had ever been before. This made the CEO's question their own credit policies, and they decided to loosen up.

As it turned out, the success of Wall Street's looser lending policies had been due mostly to luck--rapidly rising house prices. Once house prices stopped rising, Wall Street's loans started defaulting. The large number of Type I errors had been exposed. Meanwhile, Freddie and Fannie had loosened up at just about the worst possible time--just as house prices were reaching their peak. They made a lot of Type I errors, for which we as taxpayers are going to pay a steep cost.

TRACKBACKS (1 to date)
TrackBack URL:
The author at Club for Growth in a related article titled Exposing Type I Errors to the Market writes:
    I was no fan of my statistics class in college, but now that I don't have the pressure associated with taking tests, I enjoyed this brief explanation by Arnold Kling of how banks committed too many Type I errors when making loan decisions.... [Tracked on December 10, 2008 10:16 AM]
COMMENTS (4 to date)
stephen writes:

hilarious, i have a stats final on friday. powers, box plots, correlation, regression, null hypotheses, t statistics, z statistics, chi-squared tests.....ahhhhhhhhhh!!!!

El Presidente writes:

Given the magnitude of the current wave of defaults, and the multiplier effect that it produces, there may well be a good number of people who would have cleared whatever threshold you set at the time their loan was made but would not clear the same bar now. If these people default, then worrying about whether we made Type I errors and Type II errors is a little less important than a static analysis would suggest. The problem is that consumers were betting on a traditional business cycle which would have produced inflation to stabilize some of the impact of the increased financing on aggregate demand. They were receiving mixed pricing signals. Many well-payed economists thought that the world had changed. Or, to put it another way, they believed that stabilizing housing prices were indicative of a new plateau, not an approaching cliff. I disagreed, but that's just me, and who am I?

We have to consider the environment in order to interpret people's responses to it. The fact that two sets of people were interpreting the same environment in two different, yet symbiotically destructive ways should make us rethink pricing signals more than Type I versus Type II errors. If the lender doesn't have the appropriate information to set the bar at an appropriate level to minimize defaults, the consumer certainly doesn't have enough information to decide whether or not to try to jump over it. How and why would they be receiving pricing information that would lead both parties to believe this was a good idea?

Larry writes:

@El - You say that people who passed the test would have defaulted anyway. I say that loaning money to test failers catalyzed the problems that now result in problems with test passers' loans. Those loans helped inflate the bubble. No bubble; no problem.

El Presidente writes:



What if we had some mild inflation two years ago to devalue the existing ARMs and tame the asset bubble? Bubble goes bye-bye and lending slows as rates rise. I'm not saying we should have wished for inflation to solve this problem. I'm simply trying to say that there are a number of possible scenarios that could have occurred, and very many people chose the wrong one.

In retrospect, you are correct. There was a bubble and then a crash. Marginal loans contributed to it. My assessment was correct during the run-up too; that this was all getting out of hand and was a pretty ominous circumstance that could lead to very negative consequences. I don't think we disagree about what occurred, or even why. I assume you aren't stuck with a financial mess, and neither am I. Good for us.

What I'm saying though is that understanding people were responding to signals helps us to think about what signals we are sending now and would like to send in the future. It also helps us understand what expectations they have and gives us a better sense of our audience. This should all help us to craft better policy. I think we could have done much better given what we knew at the time. Many of us chose not to. I'm just doing a little postmortem examination.

The reason I thought this was all a losing proposition is that I think we did run up against a resource constraint: time. With two-income households, ballooning consumer debt, and stagnant real wages, we had already nearly maxed-out our capacity to work and maintain any sort of a healthy lifestyle. Borrowing against future wages that were not going to materialize was not wise. Doing so for consumption instead of investment was doubly foolish on the part of many. I watched them do it, in person no less. That's what troubles me. I think they'd do it again.

Comments for this entry have been closed
Return to top