The amount of oil pumped out of the ground doesn’t just depend on the current price. If I don’t pump the oil today, I can pump it tomorrow. Tomorrow’s price matters, bubble or no bubble.
Let’s start with the market when there is no bubble, and, as economists do, let me make this as simple as possible. Let’s assume there is no uncertainty about how much oil is in the ground, or how much people will want to buy at any price, and no cost of extraction. What should happen then is the expected rate of increase in the real price of oil should equal the real interest rate. Why? If I sell my oil today, I can take the proceeds and get the real interest rate. If I don’t sell my oil, its price goes up at the real interest rate. The incentive to “hoard” is exactly balanced by the incentive to sell, and any individual producer is indifferent between selling now or later.
This is what is known as the Hotelling rule for pricing oil in the ground.
Engel continues, contra Krugman,
In this case, there is no excess supply of oil. End users buy as much as they want at the market price, and producers pump out exactly that much. Ultimately the level of the price is determined by the condition that, as the price rises at the rate of interest forever, the sum of demand over the current year and all future years equals the amount in the ground.
Later, he says,
if the run-up in prices were too rapid, so that the “expected” growth rate of the price exceeded the interest rate, there would be a strong disincentive to sell any oil. Producers would want to keep the oil in the ground, and, as Paul Krugman has argued, speculators would have an incentive to hoard oil. We see very little of that type of behavior going on, as Krugman has noted.
To drive this sort of speculative hoarding, we would need the futures price to be above the spot price by more than the real interest rate. That was not the case last time I looked.
As Engel points out, the description that bets fits the data is one in which the market has been continually surprised by fundamental factors. These could include anything. I can think of all sorts of possibilities. Suppose that the market thought that ethanol mandates would restrain oil demand, but that instead it failed to do so, or perhaps even increased demand indirectly because it takes energy to run farm equipment.
Anyway, there are all sorts of possible surprises.