Bryan Caplan  

Critique My First Serious Survey

PRINT
Divisions on Immigration... Is Bush Stupid?...

I've never written a serious survey before. Sure, I wrote the Libertarian Purity Test twelve years ago, but that was just for fun. But now I've written a first draft of a survey on which my co-author Ilya Somin and I plan to base a series of academic articles.

I've already used the Survey of Americans and Economists on the Economy to test for systematically biased beliefs about economics. Now Somin and I are designing a survey to test for systematically biased beliefs about political responsibility. The idea: Ask both the general public and Ph.D. political scientists about who in government has power over what, and see if their average beliefs are the same.

Who cares? Well, one big idea in economics and political science is that voters don't need to understand policy; they can just re-elect incumbents when there is prosperity and peace, and throw the bums out if there is depression and war. Sounds good, but what if an incumbent has nothing to do with a particular problem? As Achen and Bartels put it:

If jobs have been lost in a recession, something is wrong, but is that the president’s fault? If it is not, then voting on the basis of economic results may be no more rational than killing the pharaoh when the Nile does not flood, as some scholars believe occurred in ancient Egypt, or voting against Woodrow Wilson when sharks attack the Jersey shore, as we believe happened in 1916. (citations omitted)

In any case, I'm attaching a draft of the survey. Please tell me what you think. Specifically:

1. Any suggestions for how to improve the wording of the questions?

2. Any political actors I should add or remove?

3. Any outcome variables I should add or remove?

Who in Government Has Power Over What?: Draft Survey

In the United States, there are several different branches of the federal government, including the President, Congress, courts, and government agencies. Some of these branches may have more power than others to get the outcomes they want. Others may have a lot of power over some outcomes, but little power over others. I am going to ask you some questions about how powerful you think the different branches of the government are.

1. The President

Tell me if you think that the President has a lot of power, some power, a little power, or no power over:
a. How well the economy does during the next two years
b. How well the economy will be doing twenty years from now
c. The size of government
d. How long people in the United States live
e. Whether we are at peace or war
f. Whether we win the War on Terrorism
g. Crime rates
h. The environment

[Repeat same questions for...]

2. Congress
3. The courts
4. The Federal Reserve
5. The Department of Defense
6. The Environmental Protection Agency
7. The Food and Drug Administration


Comments and Sharing





COMMENTS (16 to date)
Eric Crampton writes:

Anything getting at federal/state/local distinction?

Dr. T writes:

I recommend setting up each question as a grid with the power choices across the top and the a-h choices on the left side. Respondents would check the appropriate cells in the grid.

I also recommend being more specific on the last 2 choices:
g. Murder and violent crime rates
h. The quality of the environment

Finally, I believe that it is of no value to link all of the items 2-7 to all of the choices listed in question 1. For example, is it necessary to ask whether people believe the Federal Reserve has power over longevity or the war on terror? Leaving the survey as is will make it too long and make it seem somewhat absurd.

Daniel Lurker writes:

What about USDA, EPA, and Health and Human Services?

aaron writes:

I will try to take a look at it later.

I don't know if I am really up for reading an academic paper.

sa writes:

Q.the relative position of US vis-a-vis other nations in the short/long run?

Andrej Salner writes:

I do not know about your intended sample size and whether it will allow the testing of following effects but I have a gut feeling answers may change depending on the order in which you list the institutions (president, congress vs congress, president).

I would also maybe consider "transposing" the whole thing and asking in each row about a specific area of responsibility with institutions in columns. The questions would then encourage respondents to consider more explicitly the relative weights (addressing more or less directly the question of which institution has in the respondent's view the most influence on the given area).

Chris writes:

Hello,

You should include a "don't know" category. You might also think about splitting Congress into House of Reps and Senate.

I think the state level question might not work given some of the categories you ask. If you do include state/municipal level, then you might have to ask about Governor, legislature, etc. at those lower levels.

Varying the question order should be a given to avoid question order bias.

Finally, you might consider eliminating sentences two and three in the intro spiel.

Hope this suggestions help.

Cheers,

Chris

Nathan Smith writes:

First, eliminate questions that have obvious "no" answers. For example, how is the Department of Defense going to impact life expectancy? Or the environment? Of course you could come up with stories, but it's not obvious, and if people say "yes," you don't know if people are really expressing a point of view or being clever.

Second, you might want to distinguish between upside/downside influence. A person might think "Yes, the president can do a lot of harm to the economy if he does something really stupid like fix prices or plunge us into pointless wars, but he can't do much good, except in the sense that minimizing harm is doing good." Or they might think, "The President is the most powerful man in the world; if he wants me to have a job, he can make that happen, and if I don't have one, it's his fault." These are very different points of view, but either would lead to a "yes" answer to your question.

Third, come at each question in different ways. You could also ask "Which of the following agents in the government has most power over..." and repeat the list of public concerns.

Fourth, you might want to distinguish between power and responsibility. People might think, for example, that the courts do have some power over whether we win the war on terror because in protecting civil liberties they make it harder for our intelligence services to operate, but that it's not their responsibility to win the war on terror, but to protect our civil liberties, so they should go on protecting our civil liberties regardless of whether it hurts the war on terror or not.

Fifth, your list of concerns shows your libertarian bias! What about health care? What about gas prices? You ask about the economy, but break it down: what about jobs? What about inflation? What about the advance of technology? What about income distribution? What about culture? What about immigration? There are probably a lot of people out there who think that President Bush is personally to blame for high gas prices. Or that teenagers engage in more sex nowadays because Congress isn't doing its job. (Speaking of which "public schools" should be in your list of governmental organizations. And "the police.")

Sixth, you may know this already, but you should ask control questions, like party affiliation, indicators of social conservatism, and whether they think the government is doing a good job. You also might want to ask, when people think the government has power over or is responsible for something, whether it is doing its job well. If people think the government has power over a lot of things, that might mean, "Things are going just great, and it's all because our wise leaders are governing us so well!" Or it might mean, "Things are falling apart, and it's all because of the negligence and incomptence of Bush and the Republicans!"

Phil writes:

I believe Congress has much power over how well the short-term economy does in the negative sense (they could legislate Marxism), but only a little bit of power in the positive sense (even if they abolish all regulation, the short-term economy will improve only so much).

How should I answer the question?

Chris writes:

Who in Government Has Influence Over What?: Draft Survey

Power may not be the right word. “Influence” may (or may not be) better.

I think you need to go into much, much more detail about the structure of government.
If you are going to survey the general public, you should be prepared to explain the structure of government in detail. I think you will have to confront the fact that many people do not have any idea about the structure of the government or the separation of power.

The survey should be written at no higher than a 10th grade reading level. Be careful about the length of the survey. The longer it is the less participation. It always seems longer to the participant than it does to the researcher.

In the United States, there are three different branches of the federal government. The three branches are the Executive, the Legislative, and the Judicial.

The Executive branch is the President and the federal agencies he oversees. These federal agencies include the Department of Homeland Security, the Department of Transportation, the Department of Justice, the Department of Agriculture, the Department of Housing and Urban Development, the Environmental Protection Agency, The Department of the Interior, the Department of Energy, The State Department, the Department of Defense, the Federal Bureau of Investigation (FBI), the Central Intelligence Agency (CIA), and the Federal Reserve.

The Legislative branch is the Congress. There are two parts of Congress, the House of Representative and the Senate. Members of the House of Representatives are elected for two year terms. The number of Representatives in each State depends on the population of the State. In all there are 435 members of the House of Representatives. Members of the Senate are elected for six year terms. Each State has two Senators. There are a total of 100 Senators.

The Judicial branch is the courts. The court system includes the Federal Appeals Courts and the Supreme Court. (include a fuller description)

Some of these branches may have more influence than others to get the outcomes they want. Others may have a lot of influence over some outcomes, but little influence over others.

I am going to ask you some questions about how much influence you think the different branches of the government are.
1. The President
Tell me if you think that the President has a lot of influence, some influence, a little influence, or no influence over:

a. How well the economy does during the next two years
• A lot of influence
• Some influence
• A little influence
• No influence
• Don’t know

b. How well the economy will be doing twenty years from now
• A lot of influence
• Some influence
• A little influence
• No influence
• Don’t know

c. The size of government
• A lot of influence
• Some influence
• A little influence
• No influence
• Don’t know

d. How long people in the United States live
• A lot of influence
• Some influence
• A little influence
• No influence
• Don’t know

e. Whether we are at peace or war
• A lot of influence
• Some influence
• A little influence
• No influence
• Don’t know

f. Whether we win the War on Terrorism
• A lot of influence
• Some influence
• A little influence
• No influence
• Don’t know

g. Crime rates
• A lot of influence
• Some influence
• A little influence
• No influence
• Don’t know

h. The environment
• A lot of influence
• Some influence
• A little influence
• No influence
• Don’t know


[Repeat same questions for...]
2. Congress
3. The courts
4. The Federal Reserve
5. The Department of Defense
6. The Environmental Protection Agency
7. The Food and Drug Administration

morganja writes:

I think it would clear up some confusion if you added a sentence or two that seperates the concept of 'should' and 'does'. A person could read this survey and think that he is answering what the constitution provides as power rather than who actually is exercising power over each issue.

Marko writes:

Although the political scientists probably know more about the division of power between various institutions than any other profession, I am not really sure that they are the ones who are the most capable of answering to questions such as "does FED have a lot of power over how well the economy does during the next two years?"

I think it would be good, if that is possible, to include some economists and lawyers in your survey. It would be really interesting to see what are the differences between these professions as well as between professionals and general public. I can't wait to see the results!

t writes:

I took a class in test theory once. Items are evaluated on two dimensions: difficulty and discriminatory power.

A difficult question is one that only a few persons will get right. Getting it right is not defined as answering correctly in most tests, but as answering in the same way as everybody else. You want questions of medium difficulty, to maximise variance.

If everyone agrees that none of your actors has any degree of control over long-term growth rates you gain no information from asking in the first place and should drop the question.

A question with high discriminatory power is one that sorts people well into groups. This is usually measured by correlating a question's results with the overall result of the test and selecting items which highly correlate. The rationale is that the test discriminates better than any given question.

Robert writes:

I'm concerned that "power" might be a poor word choice. Personally, I have no idea to answer many of the questions, because (for example) I believe that the President could do plenty to trash the economy, but is relatively limited in his power to improve it. Is this power or no?

If what you want to measure is who blames whom for what, then maybe "responsibility" is the better concept. Do you consider the President very responsible, somewhat responsible, slightly responsible, not at all responsible, for the performance of the economy, etc.

PJens writes:

My suggestion is to change the order of what each section of government has possible influence over, the a b c d's. You are interested in the economic issues, so you listed them first. Most people will have to think about that in order to answer the question. If they have to think too hard, or long, they may get discouraged. I believe it is better to ask the broader interest question first. I.e. How much influence does the President have over the crime rate?

Put the thought requiring questions in the middle of the survey. Keep it as short as possible. I personally hate long surveys and often quit in the middle when they take too much time.

Lee Templeton writes:

I think this is pretty good -- far better and more subtle than most of the polls/surveys to which I am subjected. The results should be very interesting.

Comments for this entry have been closed
Return to top