Bryan Caplan  

The Most Convincing Figure in Irrational Exuberance

Hard Asia, Soft Europe... Is Stanley Fischer too complac...
I never got around to reading the previous editions of Robert Shiller's Irrational Exurberance, but I've finished reading the new third edition cover-to-cover.


Rather than review the whole book, consider the single most convincing figure.  At first glance, it's easy to miss the point.

Shiller begins with an obvious observation: "The dividend value is extremely steady and trend-like, partly because the calculations for present value use data over a range far into the future and partly because dividends have not moved very dramatically."

So what?  Here's what:
[S]tock prices appear too volatile to be considered in accord with efficient markets.  Assuming that stock prices are supposed to be an optimal predictor of the dividend present value, then they should not jump around erratically if the true fundamental value is growing along a smooth trend.
The next three sentences are truly profound:
Only if the public could predict the future perfectly should the price be as volatile as the present value, and in that case it should match up perfectly with the present value.  If the public cannot predict well, then the forecast should move around a lot less than the present value.  But that's not what we see in Figure 11.2. [emphasis mine]
By analogy, suppose you graphed actual temperature against predicted temperature.  Which should be more volatile?  Actual temperature, of course.  Predicted temperature should be heavily based on long-run average temperatures, which don't jump around much.  (Indeed, if you look more than ten days into the future, weather sites often just report historic means, putting zero weight on current weather).  Actual temperatures, however, are routinely surprising. 

Now ask yourself: In Shiller's figure, which line is analogous to actual temperature?  Actual present values of dividends, of course.  Then which line is analogous to predicted temperature?  Observed stock prices - the market's best guess of the present value of dividends.  Yet contrary to the basic logic of forecasting, the latter is jumpy even though the former is steady. 

This is a classic case of, "If you're not confused, you don't understand what's going on."  Shiller's attempts to resolve these puzzles leave me dissatisfied, but kudos to him for opening my eyes to the enormity of the oddity.

COMMENTS (39 to date)
Kevin Erdmann writes:

This seems like weak sauce to me. Weather reverts strongly to the mean. That's why weather forecasts are less volatile than actual weather. There are two major negative shifts in trend before 1950. First, WW I, the first time the entire globe was engaged in war. Is it that difficult to imagine that investors would doubt reversion to the mean in that case? In fact, several economies didn't revert to the mean, so there is a bit of survivorship bias there.
Then, the rise of fascism and Communism, culminated in WW II. Again, it hardly seems crazy for investors to doubt mean reversion when, literally, a majority of the world was ruled by madmen bent on world domination and destruction of markets. And, again several other economies did fail to revert to the mean.
There is only one remaining shift in valuations - in the late 1970s (and possibly the most recent period. (In fact, I think this would be more intuitive if the discount rate of dividends was set to match with the periods of higher valuations, with the low valuation periods appearing as negative valuation shocks.) There are many moving parts here, including highly persistent trends in federal policies, Federal Reserve policies, and demographics, which operate with such long and persistent trends that they may not be fully arbitraged. (Are baby boomers irrationally exuberant, by definition, just because they all happen to need fixed income in 30 years, when they will be incapable of being productive?) Plus there are huge swings in leverage, due to inflation premiums, tax laws, etc., which means that the "Actual stock price" is not measuring a stable security over time. For nonfinancial corporations, in the late 1970's, debt / enterprise value was close to 50%. It is now in the 20%s. A security representing 75% of a firm is a very different thing than a security representing 50% of a firm, and would call for a dynamic discount rate.

Equities earn a premium because, among other things, underlying factors don't always revert to the mean, and equity is the overwhelming recipient of risk and change, so minor changes in possible outcomes will be greatly multiplied in their valuations. And, some of the recent fluctuations are reflections of short term disequilibrium, similarly to unemployment, where dysfunctional credit markets keep prices from efficiency until a demand crisis recovers.

There certainly are instances of group bias that can occur. But, I think those who point to "Irrational Exuberance" in these cases may be thrown off by cognitive biases, like attribution errors and outcome bias, than the markets they claim to judge in hindsight.

ilya writes:

Stocks are different from weather: a 10-year average for the latter is more-or-less known in advance (we call it a 'climate'), a 10-year average for the former is very much unknown.

With that in mind, I present a simple explanation of the fact that predictions of something whose average we don't really know can be more volatile than reality.

The basic reason relates to math and is very simple: with incomplete information, any imperfections in data and random events are magnified.

Why would small deviations be magnified into big ones? Well, suppose the company had 5% less revenue in one quarter compared to the expectations. Investors now expect that there is a story behind this drop.

Most likely, they apply Bayesian thinking: some change happened, and we need to reevaluate our models. Perhaps the company revenues will drop by 5% each quarter (if the first derivative of revenue will remain constant), or perhaps they will drop by 5% first quarter, by 10% second quarter, by 15% third quarter etc. (if the second derivative of revenue will remain constant).

This process is magnifies by humans' herd mentality and existence of a large class of professionals who create those models (e.g. A's dividend is lower because their expansion in China is slowing).

If you take into account this permanent change, perhaps the future discounted cash flow will actually drop by 50%. Now, different investors will have different internal models, but overall it's understandable if price drops by 20%.

Of course, it's also possible that 5% drop was a random event that doesn't happen again. In that case you see the investors' expectations returning back, overshooting by 20% in the other direction.

So you see totally rational investors changing the company's valuation based on some future expectations that are based on the models that use as an input higher derivatives of the dividend numbers. So the end result, not surprisingly, is more volatile than the numbers themselves.

3rdMoment writes:

Agreed it's a puzzle.

One thing to note is that, unlike with your temperature example, BOTH lines in the graph contain an endogenous element, since when you pick a discount rate to create the "present value of dividends" as an estimate of "fair value," it will depend on what discount rate you think other people in the present and future will use, which will depend on what they think other people will use, and so on. So it's a complicated equilibrium.

Indeed, Shiller and Campbell and others have lots of sophisticated research that attempts to decompose the stock movements into "dividend news" (or perhaps "earnings news") and "discount rate news" (or "rate of return news") and they generally find that most movements are driven by the latter.

But it's hard to see exactly what the rate of return should be or will be in the real world. In fact the whole idea of the "equity premium puzzle" says that this return has seemed weirdly high, so how confident can we be about it in the future?

Also, time-varying probability of extreme economic disasters seems more plausible than time-varying probability of extreme weather events. See here for an argument:

Eric Rall writes:

How does Shiller measure the present value of dividends? I see that he seems to have a data point for 2013, despite only having 1-2 years of actual dividends to discount. Is he using an estimate or proxy to supplement actual dividend data? Also, is he including other cash-to-investors events (liquidations, mergers and acquisitions, etc) in the dividend stream? Is the present value adjusted for taxes? Is the discount rate constant or is it based on market interest rates?

Another thing that might be going on is that the stock price is not just an estimate of the present value of dividends and firm dissolutions. It's an estimate of the present value, adjusted for perceived risk and liquidity. The present value may be stable, while liquidity or risk might vary wildly over the short term, and only long-term investors can ride out the variations and enjoy the long-term regression to the trend of present value.

charlie writes:

This insight is also the premise of his famous 1981 paper, I believe.

Charlie writes:

John Cochrane had an interesting comment (the whole post is worth reading):

"He called the dividend line "the actual market if everyone knew the future" and the "true value."

(A minor thought. Really? Would the world really be working right if that's what stock prices had all the return and no risk? If we have an equity premium puzzle now, imagine what it would look like with no risk! If prices have no risk so we should discount dividends with riskfree rates, the major failure of today's markets is not the volatility of the price-dividend ratio, it's the level, which should be many times higher?)"

Jody writes:

Seems like we should only expect stability if the underlying PSST are stable (to use Arnold's term).

Imagine what would happen to the weather if the Rockies suddenly went away or if the sun was 2x hotter, or if South America connected to Antarctica.

Similarly, the underlying patterns of the economy are constantly changing and the stock price discovery mechanism is having to constantly attempt to detect these changes.

There are still some broader trends - technology, population, and productivity march onward - but there's a lot of underlying change being reflected in the stock prices.

vikingvista writes:

There is no universal truth of forecasting here. It depends entirely on the forecasting model being used. Weathermen typically use historical means as a model, following the textbooks they learned from. But they don't have to. A rogue weatherman inspired both by the accepted (therefore low cost) poor forecasting of existing models and some great payoff for occasionally beating all other predictions, might devise a more volatile model.

It just so happens that stock forecasters use models that weigh heavily current states that are considered unique compared to historical states, typically using parameters that are closer to each forecaster's unique specialization. This is so, because stock forecasters actually do have a very strong incentive to beat the conventional wisdom.

dangerman writes:

Unlike with weather predictions, with the stock market there's money to be made **in the volatility itself.**

Chuck writes:

[Comment removed pending confirmation of email address. This is your final notice. We have attempted to contact you several times previously. Email the to request restoring this comment. A valid email address is required to post comments on EconLog and EconTalk.--Econlib Ed.]

Rich Berger writes:

Two things:

As others have noted, the discount rate to apply in calculating present values is variable over time.

What about earnings? Dividends are paid out of earnings, and earnings have been highly variable over time. If the only factor in the stock price were dividends, stocks that paid no dividends would have no value.

John Thacker writes:
In Shiller's figure, which line is analogous to actual temperature? Actual present values of dividends, of course.

Doesn't that contradict what you quote Shiller himself saying, Bryan?:

"The dividend value is extremely steady and trend-like, partly because the calculations for present value use data over a range far into the future and partly because dividends have not moved very dramatically."

The "actual present values of dividends" are the market (and executives') best guess as to what dividends will be far in the future. (More precisely, a summation of them with discounting, so a stochastic integral.) It's an average smoothed over a long period of time-- it is comparable in temperatures to, long-run temperature averages rather than a daily temperature measure. Climate models will certainly show today's best guess for long-run temperature averages increasing.

Jeff writes:

I'm with Erdmann on this score. Shiller's basic story is coming from this 1980 paper of his:

"Do Stock Prices Move Too Much to be Justified by Subsequent Changes in Dividends?"

The only thing I'd add to Kevin's comment is that Shiller doesn't appear to be accounting for stock buybacks, which are much more important than are dividends, especially these days.

AS writes:

The dashed line should be total earnings, not dividends. Dividends are a subset of total earnings, and their determination is somewhat arbitrary. Dividends as a share of total earnings has fallen since WW2, making it a weak proxy.

Stock price is not just a function of the present value of future earnings, but also investor's demand for liquidity and risk, which can change, especially during crises when unemployment is high. The risk of a market crash is interactive with the risk of unemployment: if both occur at the same time, you get very low utility. So when risk of unemployment increases, the appetite for equity risk and hence the stock price falls.

Charlie writes:

"The only thing I'd add to Kevin's comment is that Shiller doesn't appear to be accounting for stock buybacks, which are much more important than are dividends, especially these days."

No this isn't right. Any time you see a reputable Finance professor presenting this type of data, the dividend always includes stock buybacks, as well, also mergers , share swaps, liquidations--any investor cash flow.

Here is the CRSP definition:

Dividend Cash Amount

Dividend Cash Amount is the US dollar value per share of distributions resulting from cash dividends, spin-offs, mergers, exchanges, reorganizations, liquidations, and rights issues. Dividend Cash Amount includes the cash value of ordinary and non-ordinary (return of capital) dividends. When the distribution is paid in shares of a trading security, the Dividend Cash Amount is set to the exchange ratio times the price of the security at the close of the Ex-Distribution Date.

In a distribution where a limited percentage of shares are accepted in exchange for cash, the Dividend Cash Amount is set to the offer price, and the value must be adjusted using the Factor to Adjust Price. These are identified by a Distribution Code with the first digit 6 and a Factor to Adjust Price between -1 and 0. Note: regular income dividends for ADRs use the gross.

Floccina writes:

Yes, the only reason that I feel comfortable buying stocks (and I did stop buying in late 1999 until the crash), is that I think stock are almost always under valued.

Kevin Erdmann writes:

Charlie, are you sure? That CRSP description doesn't seem to include open market buybacks. And the dividend column in Shiller's data ( ) seems to track dividends only.

I don't think it needs to be included, though, if Shiller is tracking Dividends per share.

Jeff writes:


That's the correct CRSP definition, however I'm not convinced that's what Shiller is using:

That said, prices are usually adjusted for all those items by the researcher (no quibble there). But it still doesn't adjust for the buybacks, which is my concern with more recent data.

Finally, the CRSP adjustments for mergers are very suspect. When I compare CRSP delisting returns due to mergers, for example, with SDC merger data, there are significant discrepancies. In fact, there's probably a research study there.


What about the 70%+ firms that don't pay dividends? Their value is in the stock prices, but it's not the dividend stream. Unless Shiller is restricting his data to only dividend payers, I think this is a significant problem.

3rdMoment writes:

Totally agree that you should really include buybacks in measuring payouts to shareholders, especially with recent data. But that doesn't get rid of the excess volatility:

While the actual impact of this result for asset returns will be studied empirically in the next section, this finding tends to support the existing literature that relies on stationarity of dividend yields. Consider models that include dividend price ratios in VAR frameworks and exploit their implications for long-horizon expected returns (e.g., Campbell and Shiller (1988a) and Cochrane (1998)). If one uses total payout as aggregate distributions to shareholders, one will reach conclusions that are similar to those of this earlier literature with respect to volatility of returns and its decomposition into time-varying risk premiums versus cash flow risk.
Jeff writes:

Also, on Shiller's dividend data: he interpolates the dividend data from quarterly or annual data. In other words, the data are total dividends, unweighted by firm size. But the price index is weighted, so just be careful if you are using his data in any analysis.

3rdMoment writes:

How does interpolating quarterly index dividends to monthly have anything to do with making them unweighted? That makes no sense to me.

Kevin Erdmann writes:


I don't consider non-dividend payers to be a problem when looking at macro-level data.

I do wonder, considering all these issues, though, where the P* value comes from for recent years.

Jeff writes:
How does interpolating quarterly index dividends to monthly have anything to do with making them unweighted? That makes no sense to me.

I said only so people who wanted to look at the data would be aware, because Shiller doesn't mention anything about the dividend stream (other than the interpolation) on his data webpage. So if, for example, you wanted to calculate a dividend yield, it would be misleading to do it with the data so provided.

AlexR writes:


"If you're confused, you haven't read the critiques of Shiller by Gene Fama and John Cochrane."

AS writes:

The entire dividend debate above can be settled rather quickly by simply using total earnings instead of dividends. Earnings are what matter. Dividends are just one method of transmitting earnings to shareholders. Dividends are arbitrarily more stable than earnings, so any data on dividends is going to understate the true volatility of earnings. Get a proper chart comparing stock price to earnings, and then we might have something.

R Richard Schweitzer writes:

is there not a question to be asked about the (time spectrum) "rate of change" in the present value trend line - and the factors that affect that rate of change?

Charlie writes:

"But it still doesn't adjust for the buybacks, which is my concern with more recent data."

If it's an open market buyback, no adjustment is needed. It's just a sale of stock at the market price. The market price of a share is the value of the company divided buy the number of shares. If there are less shares in circulation and no changes to future cash flows, the future dividend per share will be higher.

So a share buyback basically raises the future expected dividends per share. It's just like a dividend increase from Shiller's perspective.

Charlie writes:

You can construct this same data with earnings from Shiller's website. It will look very, very similar. The fact is that stocks are much, much more volatile than dividends or earnings. It's just a fact.

The fact leads to the inescapable conclusion that expected returns vary over time. Shiller thinks they vary over time for irrational reasons (panics and manias). Cochrane thinks they vary over time due to rational reasons, risk (most people want an extra premium to hold stocks in 2009 versus 2005).

But it is a robust fact. Actually, think about bond prices move more than they "should" too right. They also have time varying expected returns, though not as much.

Jason K writes:

Aren't there multiple factors at play? On any given day, the stock market's primary purpose isn't to predict the present value of stocks. Traders go to the market for liquidity. It's an exchange first, rather than a prediction market. At the technical level, prices reflect the supply and demand for securities at given moments, which may have little to do with future dividends.

Technical trading aspects should dominant fundamental price movements in the short term. Let's assume the "rational" price movement is 6.7% real return annually. That fundamental price movement on the Dow would be around 3 points a day. Would volatility significantly outside this daily range be irrational? I have trouble imagining billions of shares exchanging hands with this constraint.

Even for a well-funded investor, there's no practical way to arbitrage some theoretical present value calculation against the actual price on a short-term basis. It seems that rational fundamental investors would step in and counteract trading noise only when the cumulative differences becomes large. Nothing irrational about that.

Jeff writes:


After checking into it, I realize I was mistaken. Buybacks are in the dividend stream. Even if they weren't, they didn't become a big deal until quite recently, so it wouldn't solve the issue.

You are quite right - this is all about time-varying returns. Cochrane's 2011 AFA presidential address makes a very interesting point though, that gets at the heart of the distinction. If Shiller wants to say it's irrational, and others want to say it's rational, there is no way to operationally separate these using just price data.

Caplan was right - if you aren't confused, you don't know what's going on.

Jon Murphy writes:

I've read this multiple times but am still confused. I'll admit that stock are not, by any stretch, my area of expertise.

So, why are dividends analogous to actual and prices analogous to future?

Jeff writes:

Jon Murphy,

John Cochrane has a really nice summary here:

Jon Murphy writes:


Thank you!

My head still hurts, but slightly less now :-)

Phil Birnbaum writes:

I don't think it's necessarily true that expectations should be smoother than reality. If a large part of "expectations" include the chance of an important low-probability event, expectations can be MORE volatile.

Take, for instance, those experimental elections futures markets. The present value of "Obama Shares" is constant back into the past, because we know he won! The price of the futures, on the other hand, MUST fluctuate with polls and events.

If you want an example with no irrationality at all ... imagine that the stock market changes with the roll of a die. It is possible to perfectly forecast the present value of future payouts. But that expectation changes every day, while the PV, in retrospect, is smooth. You don't need irrationality, just uncertainty.

I think in those two quotes you include, Shiller is just plain wrong.

Rich Berger writes:


I agree about the earnings versus dividends. For example I found
this. Earnings are far more volatile than dividends, which can be managed. I don't know why commenters are ignoring this. Take two stocks with identical dividends but one has twice the earnings of the other. Would you pay the same for both stocks?

Incidentally, I get the same disconnect and volatility using Shiller's data which is here.

[typo in html fixed--Econlib Ed.]

Charlie writes:

The value of a stock comes from the cash flows that you get from the stock. The cash flow comes from either selling the share or collecting the dividend (defined as above broadly to include all cashflows to shareholders like liquidation).

The reason a stock has value is because it does or might some day in the future pay a cash flow. If we knew a stock would never pay a dividend, no one would ever buy it.

Earnings are not cash flows to shareholders. Earnings can be reinvested in the firm. Suppose a firm has $10 in earnings. It reinvests all in the firm at a 0% return. Next year the firm has the same $10 in earnings. If you discount the earnings, you've double counted that $10. That first $10 is gone and of no value to shareholders. So there are major theoretical problems with discounting earnings.

That said, it won't even matter. While earnings are more volatile than dividends, stock returns are much, much more volatile than that.

RICh Berger writes:


If a firm has $10 in earnings and earns 0% on those earnings, it still has the original $10, plus the $10 it earns this year. Your example would only make sense if the firm earned -100%.

Your first paragraph was correct, however.

Lawrence D'Anna writes:

I have a guess as to what's going on.

Say you tried to set up a mutual fund based on this theory. You'd look at the long term trend and buy whenever current prices are below trend, and sell whenever they were above trend. So you're holding positions who's market value at current prices are getting worse, and you're going to continue doubling down on those as time goes by, and eventually, make 10 or 15 years later, you'll be vindicated when the market value goes back to trend. In real life, if you try this, you're investors or creditors are probably going to dump you way before the market vindicates you.

So the reason prices are volatile is because of uncertainty about the medium-term behavior of prices, and liquidity constraints.

Prices are volatile because of expectations that prices will be volatile.

Evan writes:

A few commentors above have linked to Cochrane and Fama critiques. Bottom line of those critiques is that the discount rate investors are using to value future dividends is a function of broader economic conditions. Investors require higher returns during bad times.

It's not that investors are changing their expectations of future dividends so wildly; it's that they're rationally updating their discount rates to reflect new information.

Comments for this entry have been closed
Return to top