David R. Henderson  

Decisions and Outcomes

A few thoughts on Greece... The Environmental Continuum: ...


In yesterday's post in which I applied the Third Pillar of Economic Wisdom to some decisions made near the end of the Superbowl game, I didn't challenge the conventional wisdom that say that Seahawks coach Pete Carroll made a bad decision about the play call on the second down.

Matt Yglesias, though, does a good job of that.

But here's the bigger point I want to make and it applies whether the decision is by a football coach in a big game or you in your big game called life.

The point is this: it's important to distinguish between decisions and outcomes. We all know why there is so much criticism of Pete Carroll: because his play decision for that second down led to a bad outcome.

But that doesn't mean it was a bad decision.

Good decisions, in a probabilistic world, that is, our world, sometimes lead to bad outcomes. And bad decisions sometimes lead to good outcomes.

Ask yourself this: would there be all this second-guessing of Pete Carroll if the Seahawks receiver had caught the ball and marched into the end zone, putting the score at 31-28 (with a converted touchdown), with only 20 seconds left to play? Well, if we take all of these commentators at their word, that's exactly what they should be doing the day after the Superbowl: excoriating Pete Carroll for his risky decision that won him the Superbowl. What are the odds that more than a handful would do that? Close to zero. And what does that mean? That they aren't making a clear distinction between good decisions and good outcomes.

Here's how my co-author Charley Hooper and I put it in our book, Making Great Decisions in Business and Life:

There is a difference between a good decision and [a] good outcome. Brian bought a house right after he started a new job as a consultant. He almost ran into trouble due to a dry spell of consulting work. Things turned out well, and he ended up selling the house for a $100,000 profit just a couple of years later because the real estate market appreciated wildly. Many people would say that, in buying the house, Brian made a good decision.

We're not so sure. A good decision is one that you would choose to make again and again, even though you occasionally get stuck with a bad outcome. Brian was fortunate, but would he buy a similar house under the same circumstances again? The answer to that question would help determine whether or not he made a good decision.

We then give an example of a teacher giving two kids a choice of a number between 1 and 10, with the kid who chooses the number closer to the teacher's number getting some special treatment. If Addie chooses first and chooses 2, what is Greer's best choice? It's obviously to choose 3, because then the probability that he will win is 80%. But Greer chooses 1. And he wins, because the teacher chose 1. His probability of winning was only 10%. Good choice? No way. (And, of course, if Addie has to choose first, her best option is to choose 5 or 6.)

Over at 538.com, Benjamin Morris does a beautiful analysis in which he makes the clear distinction between decisions and outcomes. His bottom line, which he makes persuasively: the coach who botched the call was Bill Belichick. On the other hand, Adam Kilgore of the WaPost makes a reasonable case that Belichick, by not calling a timeout, severely limited Carroll's options. This is why I loved that particular Superbowl.

Comments and Sharing

COMMENTS (26 to date)
AS writes:

Agreed. Ex-post optimality is irrelevant, since we can never directly optimize outcomes. We can only optimize expected outcomes. In roulette, the only ex-post optimal bet is to bet 100% of your chips on the number that the roulette wheel lands on. But this information is unknown at the time of betting, so the only ex-ante optimal bet is $0.

Rich Berger writes:

Note that no one is questioning Seattle's decision to go for a touchdown at the end of the first half rather than the conventional field goal try. Worked out but could easily have led to no points instead of a fairly sure three points.

Masterleep writes:

Another way to describe this is that there is a difference between a good decision and a lucky decision.

David R. Henderson writes:

@Rich Berger,
Note that no one is questioning Seattle's decision to go for a touchdown at the end of the first half rather than the conventional field goal try. Worked out but could easily have led to no points instead of a fairly sure three points.
Good point. And at the time I thought it was a bad decision. I still do. It was a bad decision with a good outcome.

Scott Sumner writes:

David, Very good post---that distinction between good decisions and good outcomes is little understood. Suppose Hoover had made decisions that made the Great Depression 25% less bad. He still would have been blamed for the Depression. The decision might even have been viewed as incorrect.

The WaPost story was particularly interesting.

Yancey Ward writes:

After the miracle reception, Seattle had time to run four rushing plays from scrimmage to get the go-ahead touchdown. However, they burned one of two remaining timeouts before running the first of those four plays, and they were in no hurry to run the second play (at least 30 seconds ran off the clock before the snap).

There were two gambles in play after that first running play- Seattle was gambling that they could score and leave New England less than 20 seconds to respond, and after Seattle refused to call timeout after the 1st play, Belichick was calculating that Seattle could only rush the ball two more times rather than three in the time that would be left (even if Seattle burned the last TO). This is why he believed a pass play was coming on 2nd down, not a rush- a reasonable guess. I suspect that Seattle had thought New England would stop the clock after each play- I know it surprised me that NE didn't call timeout.

Rob Rawlings writes:

People in the IT world used to say "no-one ever got fired for going IBM"

Would people be criticizing Carroll if he had gone for the more obvious move and it had failed ?

Probably not.

Should that have been something he should have taken into account ?

robbbbbb writes:

I'm a Seahawks fan, with an immense amount of emotional investment in the outcome. But I've been trying to be logical about the whole thing, too, even though I'm very disappointed in the outcome. To answer some points in order:

(1) As to whether or not the Seahawks should have called run or pass, I remain somewhat agnostic. I've read some good analyses about it, and I think I can see the justification for a pass.

(1a) In the moment, I was thinking: Run to Lynch. Hand the man the ball. This is his moment, he was made for it, go for it. The line was set to run read-option, and that would have been a perfect call for the moment. They had the timeout and everything, so even a failure wasn't terrible.

(1a1) And Lynch ran 24 times that day, and was stopped for no gain twice. He did not lose yardage. He pulled out a yard or more on 22 of his 24 plays.

(1b) I thought that particular pass play, however, was terrible. It's a throw to the middle of the field. Risks are higher. Plus, you're going right into the teeth of the defense.

(1b1) What's a better call? If you're going to throw, then you must run play action. Make it a run-pass option, bootleg, move the pocket, something. Everyone in the stadium thinks Lynch is getting the ball. Make him part of the play.

(1b2) Or put the ball up on a fade. Chris Matthews had been owning the Pats on those plays all day. That's a much less risky play than a throw over the middle.

(2) As to the call at the end of the half, I liked it. Best resource I can think of is Brian Burke's 4th down calculator. It's not a perfect sim, but if you put in 10 yards to go, with 6 seconds left on the clock at the end of the half, down by 7, then the model says that the kick and going for it are about a push. There's a slight edge in win probability for the kick (0.32 to 0.30), but it's close enough that you can argue either way. And the Seahawks obviously found a matchup they liked.

Plus, I think they had time to run one play and then kick the FG if it failed. There was 0:02 on the clock after Matthews caught the ball, after all.

As disappointed as I am in the outcome, it was a good game. And the Seahawks are still going to be good for a long time. We've got a franchise quarterback, and In Russell We Trust.

Dan W. writes:

The choice to pass was not a bad call. In fact probability theory supports the choice to pass. The bad decision was to call for a pass play that gave the defender an opportunity to intercept the ball. This was a risk the Seahawks could not afford to take. On 4th down, yes. On 2nd down the first priority of the pass play should have been to minimize the chance of interception.

The real burden of the play needs to rest on the quarterback Wilson. He is the one who threw a less than perfect pass. He is the one who could have decided not to throw the ball to the spot he threw it to. If there is a need to blame then the responsibility is on the quarterback.

Michael Terry writes:

Ack, no! Benjamin Morris's analysis was not beautiful, it's totally missing the point!

The anger in Seattle isn't that a pass was called, it's the type of pass. A slant over the middle is Seattle's worst pass play. Russell Wilson is short and this doesn't play to his strengths. Last week at Green Bay, Seattle's first INT was a bobble by Kearse on a slant. This isn't even the first goal line slant pick Wilson has thrown this year: Lynch bobbled one into an INT earlier this season.

Seattle doesn't throw that many slants to begin with because throwing short over the middle of the field gives many defenders a shot at a bad throw or deflection.

Any other pass would have been better. Roll Wilson out and let him have some discretion to throw it away. A timing route left him no discretion. There were 3 downs left with plenty of safer options to try.

That's not even to mention that New England's cornerback Browner played for Seattle last year and immediately diagnosed the play, and demolished Kearse, leaving his safety clear to make the play. Seattle clearly should have known that Browner might see that.

Benjamin Morris likes to be contrary. He has a long series arguing Dennis Rodman is the best basketball player in NBA history. A lot of people in the media want to say the Pats won this game rather than Seattle lost it. But that leads to bad analysis. Even in the section where he says, "Well, if you want to give Lynch greater weight for probability for making the one yard, because he's better than most backs, then the odds to run are slightly better", he's still being a bit disingenuous. Seattle was the #1 ranked team in power run situations last year and the Patriots were #30 defending them. It was a clear advantage even just to run the ball.

But again, the outrage isn't run vs pass, it's the pass play called.

David R. Henderson writes:

@robbbb, Dan W., and Michael Terry,
Excellent analyses. I hereby resign as an expert on football.
My main point remains, though, as I gather all three of you recognize implicitly: it is important to distinguish between good outcomes and good decisions.
@Scott Sumner,

Devin writes:

Your broader point is well taken and it is a most understood and underutilized way of thinking about decisions.

However, given the situation, it was simply the wrong decision, with respect to probability, and it's just the wrong call with respect to situational football.

See espn for some interesting data on pass v. run and turnovers close to goal:

But they dont mention the total probability calculation for running on 2nd and 3rd down, then run or passing on 4th, or any combination of run/pass after 2nd down. Of course there are Bayesian considerations here, and the NFL is increasingly using pass plays at the 1 yard line, but any expected value calculation will come back decidedly RUN RUN, then whether PASS/RUN PASS/RUN on the other downs is getting into the weeds.

From a football standpoint, while Lynch has been poor this year running it in from the 1, it's way too small a sample to go against game momentum, and the fact that he already ran one in. Or if not Lynch, run a play action bootleg for Wilson.

The risk/reward of passing just doesn't make sense, especially if you factor public harassment. No one would fault you for going to Lynch 3 times and failing to punch it in.

Chris writes:

As a card player, it always amazes me when people don't understand that the right decision can lead to a bad outcome.

Charley Hooper writes:

Question: Wouldn't the score have been 31 to 28 with a Seattle touchdown? And hence a New England field goal would tie the game.

Perhaps someone can tell me why I'm wrong, but I think the best play for a team with a running quarterback (e.g., Seattle and San Francisco) is to fake a handoff to the running back and then have the quarterback roll out, with the choice to run it in, pass it to a teammate, or throw the ball away out of the end zone.

A really bad Seattle "decision"? Encroachment with NE on the 1-yard line. A safety would have given Seattle two points and the ball back.

Charley Hooper writes:


Good point. Here's another (extreme) example from Making Great Decisions in Business and Life:

Consider the following gambling situation. You are playing poker and get a hand with the eight, nine, ten, jack, and queen of spades. Out of 2,598,960 possible poker hands, only six hands will beat yours—straight flushes in diamonds, hearts, and clubs that are king or ace high. Three hands will tie yours—queen-high straight flushes in diamonds, hearts, and clubs—but we will ignore those for now. So out of 2.6 million possible hands, only six feasible hands can beat yours. You bet everything you have only to be shocked when another player puts down his hand with a royal flush (ace high) in diamonds. You made a great decision but had a bad outcome. If by some amazing chance you ended up with the same hand later that night or later in life, would you still bet everything? You should, because the odds are massively in your favor. With such a high straight flush, your probability of winning is 99.99977 percent. Losing once was a fluke. A smart decision-maker would bet heavily on this hand again and again.
David R. Henderson writes:

@Charley Hooper,
Question: Wouldn't the score have been 31 to 28 with a Seattle touchdown? And hence a New England field goal would tie the game.
Oops. Yes. Higher mathematics. Correction made.

Michael Byrnes writes:

I'm a Patriots fan, so this might just be my Patriots=colored glasses talking, but I didn't see a huge problem with Carroll's call.

It very nearly was a touchdown pass. I rate Butler's interception as one of the great defensive plays I have ever seen in a football game.

He was ready for that play, he sold out for it (had the receiver faked and run a fade he would have been all alone in the end zone), and he had help from Browner who held his ground when he was supposed to be pushed into Butler's path.

But even so there were many things that could have gone wrong. His timing and angle were perfect. If he gets there any earlier, he's looking at pass interference. Any later, nothing he can do but try to force a fumble. And he came in at an angle where he could go shoulder to shoulder with Lockett WHILE making a play on the ball. And he was able to hang on to the ball through the contact. Yes, he was braced for the contact, but that doesn't mean it was easy to catch a fottbal while doing so.

The most likely outcomes of that play were, in no particular order, third down and touchdown. And that is with a corner who is ready for that play, plus good work from Browner - otherwise that's an easy touchdown.

I find the clock issue more interesting. I've seen a few comments from people covering the game like, "Pete Carroll should have called time out, or at least not let so much time run off the clock... AND Belichick DEFINiTELY should have called time out!" Wait, what? It has to be one or the other - they cannot BOTH have been wrong to let the clock run. If Pete Carroll should have stopped the clock, the Belichick was right to let it run (and vice versa).

Again, it could be my Pats-colored glasses, but I have convinced myself that letting the clock run was the right move for Belichick. I heard someone mention that Carroll was looking to the Pats sideline waiting for the Belichick time out - and I think a lot of coaches would have called one. Seattle didn't have a plan ready, they didn't want to burn their final time out, they didn't want to risk getting stopped twice and running out of time before their 4th down play. If Belichick had called the time out, it would have let them off the hook. So I'm calling it a masterful decision by Belichick.

Yet, without a once-in-a-lifeime great defensive play, Seattle either wins the game right then and there or they get 2 more shots from inside the one.

David R. Henderson writes:

@Michael Byrnes,
VERY nice analysis. You’re also backing up my view that Butler took a huge risk and it paid off. Indeed, we might even say that it was Butler who made a bad decision.
Quick question, since you obviously know the game better than I: I hear commentators use the term “sold out,” and they, like you, don’t use it in the pejorative sense that I’m used to. What does it mean in football?

J Scheppers writes:

Dr. Henderson,

Great topic, great point and even greater comments. I was going to make a comment about the wisdom of the call, but your comments above eliquently cover everything and more than I was going to say. From my perspective 95% of the comments were superb.

Thank you Dr. Henderson for orchestrating.

ZC writes:

"Sold out" as used in this context means completely committed to, put it all on the line.

Great discussion, by the way. Sports are often used to facilitate teaching life lessons, and the difference between decisions and outcomes is an invaluable one.

sards writes:

All good poker players know that you cannot judge the quality of a decision by its outcome. In fact, in the poker community, there is an official name for that mistake: "results-oriented thinking".

Michael Byrnes writes:

David H wrote:

"I hear commentators use the term “sold out,” and they, like you, don’t use it in the pejorative sense that I’m used to. What does it mean in football?"

In this case it meant that he made a trade off - he gave himself a better chnce to stop the slant over the middle at the cost of a worse chance to stop a pass to the outside.

Butler said after the game that he had practiced just this situation in the lead up to game (indeed that he was burned by a backup Pats WR who did not dress for the game), he noticed that the Seattle WR had was looking inside, and he saw Wilson looking at the receiver before the play.

ESPN had a great "sports science" video on this play. At the snap, the WR planted his foot to drive inside, and before he had even taken a step Butler had planted his foot to move to cover him. It would have cost the WR some time to change direction but probably would have cost Butler even more time unless he read the play as an obvious fake. Also, Butler hit him with 800 lbs. of force as he made his play on the ball.

Was Butler's play a mistake? That depends on whether his intuition can be trusted. If he's any slower to the ball than he was, it's an easy TD for Seattle.

Chris writes:


Yes, in bridge it's called "resulting"

Charley Hooper writes:

Here's a good game theoretic view of the last seconds of the Super Bowl by David Leonhardt.

As I learned it in grad school, you decide beforehand how often you should be unpredictable. For instance, the optimal strategy might be to be unpredictable 25 percent of the time. In that case, for each play you call, you look at the second hand of a watch (or some other device) and, if the second hand is between the 9 and 12, you call a different, and seemingly sub-optimal, play, just to keep the other team off balance.

Charley Hooper writes:

Oops. The article was written by Justin Wolfers.

David R. Henderson writes:

@Michael Byrnes,
Thanks for the explanation of “sold out."

Comments for this entry have been closed
Return to top