the actual data we have on crude oil don’t show the signatures of a market driven by speculative demand. Inventory data don’t show a big accumulation; and the market has mostly been in backwardation, not contango.
Backwardation means that futures prices are below spot prices. Contango means the opposite. See this post from four years ago.
My model of the oil market treats inventories and oil in the ground as the same. I don't care whether it is sitting in a storage tank or sitting under the Saudi sand--it's all part of the stock of oil. To look for shifts between under-sand oil and in-tank oil as evidence one way or the other on speculation does not strike me as compelling.
To a first approximation, there is just one price of oil, and it has to equate demand and supply for all time. If that price is too low, then at the consumption rates consistent with that price we will run out of oil. The holders of oil will have foregone large future profits and sold too much oil today. If the price of oil is too high, then producers will keep too much oil in the ground, the price will eventually fall, and they will have foregone large current profits.
To a next approximation, there is a path of future prices of oil. As long as extraction costs are not decreasing and the interest rate is positive, that path ought to embody a gradual increase in the price of oil. That increase will be higher the higher the nominal interest rate (so it should be rather low now). This is called the Hotelling principle.
If oil prices are expected to decrease (backwardation), then you want to extract the oil now, sell it, and put the proceeds into interest-bearing assets. With backwardation, all stocks of oil, whether in storage tankers or under the Saudi sands, should head toward zero.
The preceding paragraph implies that we should never see backwardation in commmodity markets. However, we do see it (Keynes used the phrase "normal backwardation.")
Economists explain backwardation using the term "convenience yield," which is something of a fudge factor. People don't sell their stocks of a commodity when there is backwardation, hence they must get some utility from holding the stock, so let's call this "convenience yield."
Perhaps a more plausible story would be option value. If you retain your oil even when there is backwardation, you have the option of selling it later. On average, it looks like you will forego profits by selling it later, but with volatile prices there is the chance that you will get to sell it at a much higher price. If you sell the oil now, you do not have such an option. When oil prices are volatile, the option value is high, so you keep your stock of oil even if the futures price is below the spot price. For this story to work, there has to be an asymmetry in possible oil prices. That is, you have to believe that there is a floor, but not a ceiling. I suspect that the floor has to be rather high in order to matter--zero probably will not do.
In short, my model of the oil market is this:
1. Speculators determine "the" price of oil by placing bets in the futures markets that reflect expectations for the entire future path of demand, discovery of new oil reserves, development of alternative energy, etc. There is no such thing as a "signature" for when future-oriented speculation is determining the price. Future-oriented speculation always determines the price--high, low, or in between.
2. The relationship between spot and futures prices should be based on the Hotelling principle, which would imply that futures prices would be above spot prices, with the differential approximating the rate of interest. If the spot price were too low relative the futures price, producers would extract little oil, if any. If the spot price were too high relative to the futures price, producers would use up their oil too quickly.
3. If there is a floor on oil prices without a ceiling, then keeping oil in the ground potentially has option value. With volatile oil prices, there is an additional incentive to leave oil in the ground. For a given level of futures prices, the option value causes spot prices to be higher than what the Hotelling principle would predict.
UPDATE: One commenter and one emailer have pointed out that there are substantial adjustment costs in oil production, so that oil in the ground and oil inventories are not the same. You cannot suddenly extract from a given field at a higher rate without risking having the formation collapse. You cannot "cap" an oil well without having to re-drill it later.
I can see how this would affect short-run dynamics. The flow supply of oil would be less responsive to the differential between futures and spot prices than if production changes were frictionless. This in turn would mean that we could see more "slack" in the futures-spot relationship (wider movements between contango and backwardation).
However, it does not change my view that overall the price of oil is determined by speculation that incorporates expectations of future supply and demand. And since this is always the case, I cannot imagine what "symptoms" would show that today we are seeing speculation but yesterday we were not.