The gold standard of modern social science is the bona fide experiment, also known as a “random controlled trial.” “The stuff going on at the Poverty Action Lab” is the modal answer to the standard GMU lunch question, “What’s the most important work going on in social science today?” And yet… real-world policy-makers continue to neglect, evade, and actively oppose experimental tests of efficacy. In Adapt, Undercover Economist Tim Harford explains why:
[P]oliticians resist pilot schemes with objective measures of success. This is partly because politicians are in a hurry: they expect to hold on to a role for two or four years, not long enough for most experiments to deliver meaningful results.* Even more politically inconvenient is the fact that half of the pilot schemes will fail… so the pilot will simply produce stark evidence of that failure.
Harford continues:
This is our fault as much as the fault of our politicians. We should tolerate, even celebrate, any politicians who test their ideas robustly enough to prove that some of them don’t work. But, of course, we do not.
This is all a nice example of a theme I’ve been pushing for a while: Most political “agency failures” actually reflect principal negligence:
[P]olitical agency problems are often a byproduct of voter irrationality. The principals give their agents grossly suboptimal incentives, then complain that the agents fail to carry out their assignments.
For example, a key feature of the main models in PA is that there is no pay-for-performance. No matter how good or bad a job a politician does, he gets the same compensation.
Admittedly, this is a standard feature of modern democracies. But why is it a standard feature? Because it is too hard to evaluate politicians’ job performance? If so, using re-election as a carrot is equally misguided. Because it is too hard to assign optimal weights to various aspects of job performance? If so, one could simply “let the people decide” the optimal weights by basing bonuses on approval ratings. Because politicians’ actions have long-run consequences? If so, bonuses could be a function of long-run consequences…
Some better arguments against paying politicians for performance may exist (Hart et al. 1997). However, the flimsiness of the leading objections should open us up to a simple alternative: Pay-for-performance is a good idea, but the public is too irrational to accept it.
The lead article in the latest Journal of Political Economy presents a model where voter turnout automatically adjusts to overcome ideological bias. What’s next: A model where it’s rational for voters to punish politicians for using RCTs?
* Harford’s own footnote: “Donald Green, Professor of Political Science at Yale, tells me that one question in the social sciences has been thoroughly tested with field experiments: how to get out the vote. So politicians can use rigorous evaluation methods when it suits them.”
READER COMMENTS
Nathan Smith
May 30 2011 at 3:06pm
Rational irrationality of voters is indeed a compelling insight, albeit it kicks the ball out of economists’ court. There are lots of ways to be irrational. Why do people pick some rather than others? The answers to that in *Myth of the Rational Voter* seem a little bit hand-wavey to me. Democracy is a little bit like the bumblebee: all the equations say it shouldn’t be able to fly, yet it does. (At least sometimes.)
Maybe the secret is TRADITION. It takes a long time to see what works, so we should be very wary of changing things. And we are. The most successful democracies have a lot of tradition: venerated old constitutions, the thousand-year-old tradition of the common law, political parties that have lasted for centuries. Democracy works when democrats hold things sacred and are deferential.
Robin Hanson
May 30 2011 at 3:14pm
But business executives are also reluctant to authorize randomized trials, or prediction markets, also because they risk giving unpleasant news.
Tomasz Wegrzanowski
May 30 2011 at 11:29pm
What’s the track record of RCT in social science? They’re a tiny fraction of social science research for a good reason.
For most political questions there’s no way to properly randomize, or to have a proper control group, or to even properly measure complex outcomes. There’s fallacy of composition problem, race to the bottom problem etc. on top of that.
Even if you assumed long term interested, perfectly rational, perfectly benevolent government, there would still be a lot of reasons not to do RCTs – scientists have much easier questions, different incentives, and they very rarely do, so why do you expect politicians to?
Also, our political systems are based on tradition. We tend to keep what worked in the past and are resistant to change – this is mostly a good thing. Political RCTs are a new idea with zero track record and dubious theoretical basis, so why do you expect widespread adoption of them?
A better question is – what would be the easiest showcase of power of political RCTs when implemented on a small scale basis? If you got one shot, political RCT with few problems, unknown issue of high impact, what would that be? Then focus on it.
Edward Gaffney
May 31 2011 at 5:16am
My impression is that blatantly unequal treatment of two similar cohorts of people, in the granting of money or benefits by the government, really is anathema to many people in a modern liberal democracy. The group that gets less money or benefits see the scheme as arbitrary unfair treatment by the state, with an obvious set of beneficiaries to rally their families or public opinion against. People fear the material and social status costs of unequal treatment by the state, even when the inequality is randomised and arbitrary.
Perhaps this explains why interesting RCTs thrive in poverty scholarship, where we can’t get the government benefits to everybody anyway, and they can’t substitute with private behaviour due to undeveloped markets. Treatments in rich countries would either be interesting but repugnant to deny to the control group (differential awarding of medical care), or acceptable to deny but uninteresting (differential awarding of Lamborghinis).
Surely people “use” re-election for the primary reason that we elect our politicians before their terms of office, rather than for the primary reason that it can be a performance-related bonus after their terms of office.
Ruth Stewart
May 31 2011 at 11:43am
On the point that politicians can use rigorous evaluation methods when it suits them… I think we need to remember that that isn’t entirely unreasonable. Indeed it’s logical. If I conduct a systematic review or a randomised control trial and I do it to a high standard and disseminate it effectively, I get to keep my job, get more funding, receive praise from those who matter to me. A politician needs to weigh up what counts as success in their field of work and act accordingly. They won’t get praise for following unpopular (yet rigorous) evidence. Instead they may lose their job.
If we want politicians to pay attention to RCTs (or other forms of rigorous evidence) then we need to do more than present them with the evidence and expect them to jump. They need support from the system. And we need to start facilitating a change in society where citizens demand evidence for decisions and challenge politicians who don’t make evidence-based decisions.
How many of us would publish a paper if the chances were you would be unemployed by the end of the year as a result of your paper? Few I imagine. Then let’s not judge others who hesitate to do what’s right because they want to keep their jobs.
liberty
Jun 1 2011 at 6:28am
I’m not sure that this is as true in the UK as USA. There seem to be many tests of efficacy in UK and policies are in fact reversed when they come out poorly. I can think of 3 potential contributing factors: culture/personality of the English, the longer recent history of public programs which made this more important to do (and which in turn affects culture/personality), and the size of the country.
Jim Manzi
Jun 2 2011 at 8:33pm
Robin Hanson,
I know personally of thousands of randomized trials that have been executed by dozens of the world’s largest corporations over the past 10 years using my company’s software. One company, Capital One, executes tens of thousands of randomized trials per year, every year.
Edward Gaffney / liberty,
I’ve tried to catalog every social science (excluding medial science, therapeutics and psychology) RCT with at least 100 test subjects reported through recognized journals ever done. There are no more than a few thousand; my best guess is that the total is on the order of 1,000. A majority have been done in the US, the rest of the Anglosphere accounts for the bulk of the remainder.
Comments are closed.