Arnold Kling  

Dinner with James Manzi

Paul Krugman on IS-LM... The Age of Contestability...

The publishers of Uncontrolled, his forthcoming book, hosted a dinner attended by about a dozen conservative media luminaries, as well as by yours truly. We received advance review copies prior to the dinner. Uncontrolled will not be available until May 1.

I have tried to refrain from referring to it until closer to publication date, but I cannot maintain this self-imposed embargo. Indeed, when I invoked the Hill criteria here and here, I was referring to methodological principles that I discovered from reading Uncontrolled, which I believe should be part of the econometrics curriculum.

In fact, I could make a case for replacing, or at least preceding, the econometrics course with a course on empirical methods in social science. And I could make a case for putting large chunks of Uncontrolled on the reading list.

Manzi is a fan of randomized controlled experiments in business and public policy (in the latter, examples include the Rand health care study and the Wisconsin income-maintenance studies). I believe that decision-makers will resist this approach, for the same reason that they resist Robin Hanson's suggestion to use prediction markets. That is, decisions are not necessarily about achieving results. They are often about establishing the status of the decision-maker. For a decision-maker to conduct experiments or to employ prediction markets is to admit ignorance and doubt, which lowers the decision-maker's status. (Compare the status of economists who give a confident-sounding answer with those of us who say, "There is no way to know" when asked questions like, "How many jobs will the stimulus bill create?")

I think that the real benefit of Manzi's book will be in the way it reinforces Hayekian conservatism*. Manzi and Hayek would say that the emergent order of society includes both embedded wisdom and embedded error. That is, people have developed habits, norms, and formal institutions, many of which promote the general welfare but some of which do not.

The application of social science to public policy is an attempt to use conscious knowledge to replace embedded error. What I call Hayekian conservatism is the view that social scientists know so little that these attempts are more likely to undermine embedded wisdom than to correct embedded error. Therefore, policy ought to be cautious.

(*Hayek famously wrote that "I am not a conservative." However, one interpretation of that statement is that he was referring to European conservatism, meaning preserving a certain hierarchical order. I would say that a cautious approach to implementing top-down policy change is both Hayekian and conservative.)

Hayek would argue that there is embedded wisdom in society that is beyond the understanding of social scientists. Call these the "unknown knowns," if you will. That is, social scientists are not aware of how habits and institutions work, but these habits and institutions have, through evolution, accumulated tacit knowledge. As social scientists, we can make some guesses about how property rights, the rule of law, trust, and trade contribute to higher per capita GDP. However, we are not able to explicitly "fix" underdeveloped countries. Richer countries are richer in the unknown knowns.

Manzi argues that social scientists have developed very little knowledge that is valid and useful in fields such as criminology, education, and other policy areas (William Easterly would stress foreign aid as another example, and I would stress macroeconomics as another.) I think Manzi makes a profound, compelling, and insightful case, so that Uncontrolled is a valuable book.

When Manzi argues for randomized controlled trials, he makes a modest case. He says that social scientists could learn a bit more, and therefore make policy a bit more effective. At the dinner, I think a number of us were skeptical that if Manzi's call were heeded the intended consequences would be accomplished and unintended consequences would be avoided.

COMMENTS (8 to date)
Mike Rulle writes:

This book sounds exciting. I have commented here before that smaller sized experiments at the micro level have at least a plausible probability of creating progress (by rejection or acceptance) that are believable.

As far as scientific method goes, Macro is virtually an impossible subject to scientifically test. Super string physics it is often criticized as being "not even wrong" because there is no possible way to test it.

I wonder sometimes if Macroeconomics is "not even wrong". I think not. It falls under the category of potential knowledge, because while very difficult to test, one can at least imagine plausible tests. So not quite science, but still potential knowledge. After all, many truths exist which cannot be potentially falsified ("last night I was home")? With this book however, scientific theory may be applicable to economics.

Will definitely read it.

jc writes:

An oldie but goodie (well, "oldie" by internet standards). Short read and, perhaps, a preview of Uncontrolled?

jc writes:

Oops. I see that it was originally linked by Arnold to begin with. In fact, that's probably how I found it. :)

Steve Sailer writes:

I managed BehaviorScan marketing research controlled experiments in the early 1980s. BehaviorScan is a remarkable real world laboratory test in which households are shown different commercials for consumer packaged goods in their own homes on their own TV sets and then have their purchasing behavior at all the grocery stores in town tracked for up to a year. It was an astonishingly advanced system 30 years ago.

Corporate American embraced BehaviorScan enthusiastically in the first half of the 1980s. But, then clients got kind of bored with BehaviorScan experiments after about 1986, and revenue fell off to a lower level.

Dry Creek Boy writes:

Sort of appropriate that the book will be published on May Day.

Mike Riddiford writes:

The introduction to the linked article compellingly argues that, to use the old joke, for each expert in social science, there is often an equal but opposite expert. As such, breaking this deadlock with a conclusive experiment is an attractive idea.

My interest is in the application to business, rather than public policy. I can see how the discussed experiments with, for example, different color envelopes containing marketing offers could be useful (e.g. easy to keep everything else constant). Does anyone know if Manzi looks at more complex experiments to test strategic options?

Gian writes:

Consistent application of Hayekian principles should have made Hayek a European Conservative i.e. a Tory rather than an American Conservative i,e. a Whig.

How did Hayek came to the conclusion that "a certain hierarchical order" that was present in all European nations since forever was an embedded error and not embedded wisdom?

How does one ever decide between embedded error and wisdom? One can not do randomized controlled trials (RCT)for most things and even if one could, one must interpret the results by one's prior beliefs.

Even setting up of a RCT requires a great deal of assumptions and priors. Witness fate of a lot of RCT in medicine. There is scope for a lot of bias and unforeseen effects.

David Oliver writes:

Embace the so-called Hill Causal Criteria (Hill himself advised they weren't) as you would a timber wolf.

Some years ago when causality in the courtroom had become a complete farce, with all manner of demonstrably false claims of eg. industrially-generated cancer producing mega-verdicts and thus the mega-fees that made the trial lawyers among the primary bank rollers of the Left, those of us trying to expose junk science seized upon the Hill criteria as a way to focus courts' attention on statistical inference and how it's sensibly done. It worked. For awhile. Then the Left fought back.

Petrified by the prospect of letting the data speak Truth, in the form of defensible causal inferences, to Power, organizations like Soros' Open Society began funding attacks on Karl Popper (thereby proving that politics trump philosophy) in particular and empiricism in general. Ultimately the Hill causal criteria were by some courts reduced to a criterion - plausibility. Now all it takes is a credentialed academic willing to testify (at a rate of $650/hr) that in his/her opinion the causal claim advanced by plaintiff is plausible and you're good to go.

So, rather than wandering down this well-trod path I'd suggest you look instead to the evidence-based medicine movement. Start with the the U.S. Preventive Services Task Force ( and heed their call for pre-research protocal publication and complete transparency.

P.S.The Left has even mounted attacks on randomized controlled trials. Ultimately it's just the old case of Empiricism v. Rationalism IMHO.

Comments for this entry have been closed
Return to top