Scott Sumner  

Jonathan Chait's bizarre idea of an honest poll

Cotton, and Slaves, and Arroga... Formal and Informal Violence A...

Here's Jonathan Chait:

And Reason's poll does yield many findings that align millennials more closely with right-wing economic thought than with left-wing economic thought. It does so through the use of crafted language. As noted above, Pew's poll asks a basic smaller-government-fewer-services/bigger-government-more-services question, finding millennial voters far to the left of older segments of the electorate. Reason asks the same question. But it also asks another version of this question, where respondents are asked if they want bigger government with high taxes or smaller government with low taxes. As often happens in polling, the change in wording produces a dramatically different answer, increasing the small-government share from 43 percent to 57 percent.
I can imagine three types of polls on big government:

1. Do you favor really high taxes?
2. Do you believe the government should provide lots of services?
3. Do you favor lots of government services paid for with high taxes?

The first would have clear conservative bias, and get an anti-government result. The second would have liberal bias, and get a pro-government result.

The third option seems the fairest. Of course if you simply ask people if they want lots of free goodies, they are going to say "yes." And if you ask them if they want to pay lots of taxes they will say "no."

And yet somehow Chait regards a relatively honest poll question, pointing out both sides of the "big government" question (tax and spend) is obviously biased, and simply asking people if they want lots of freebies is a fair question. Can someone explain Chait's reasoning to me?

Comments and Sharing

CATEGORIES: Political Economy

COMMENTS (22 to date)
A 24 year old millenial. writes:

He doesn't like that we're naturally libertarian.

Pajser writes:

Pew's poll compares different generations. Trend can be observed even if questions are very biased, as long as same questions are given to all generations. Reason-Rope surveyed only one age group, so their conclusions are more sensitive on form of the question.

pdh writes:

trick question! he's not reasoning.

Steve S writes:

I think you might be giving Reason the benefit of the doubt, their question is more in line with your question #1.

Asking if I want "bigger government with high taxes" doesn't really paint a clear picture of the benefits I'll be receiving for my tax dollars. It's almost a tautology (big government = government with lots of money).

Dan C writes:

It leaves out the (fantasy) option "Do you want big government and low taxes?" Also another option would be, "Do you want big government as long as it's paid for by someone else?" Which is option championed by Obama, at least in rhetoric if not reality.

Thomas Sewell writes:

Chait is reasoning from the standard left-leaning poll wording of "smaller-government-fewer-services/bigger-government-more-services" that mentioning taxes is clearly more right-leaning.

Of course, that leaves out that the standard wording of the question is left-leaning, as it pretends the choice doesn't imply a change in taxes.

You won't convince Chait of that, though.

Granite26 writes:


Are you saying that Pew compared the dumb kids of today with the same question asked of the dumb kids of 30 years ago, rather than the dumb kids of today with the slightly less dumb adults of today?

If so, good point

JLV writes:

When people hear "higher taxes", they really think "higher taxes on me". So if you were to change it to "bigger government and higher taxes on the rich" you'll get a different result than "bigger government and higher taxes". This has the added benefit of actually being the liberal position as well.

BC writes:

Perhaps, this is related to Scott's earlier post about liberals and free lunches. Rather than arguing for particular trade-offs --- more services vs. lower taxes in this case, liberals have a tendency to deny that a trade off even exists.

In the same Chait column, he presents the following as another example of a "loaded" question: "If you had to choose, would you rather live in a society where the gap between rich and poor is small regardless of achievement, or a society where wealth is distributed according to one’s achievement?" Again, one could only view this question as "loaded" if one denies that there is a trade off between pursuing equality of results and upholding meritocracy.

Chait's point about needing to compare results across generations to draw conclusions about generations is a valid one. Scott's framing, however, would be the correct question to ask all generations. Perhaps, younger generations are less likely to instinctively understand the connection between more government services and higher taxes, for example because they have less experience in listening to politicians' sales pitches. That might explain why the Pew poll's less balanced wording leads to such different results from the Reason poll.

Scott Sumner writes:

Pajser, All true, but not relevant to this post. I never claimed everything Chait said was wrong, just that I was puzzled by one particular claim. His article convinced me that 57% of millennials oppose big government in a balanced question. So he pushed me in the direction of Reason magazine, as I never would have assumed it was that high.

Steve, I don't follow your logic at all. Big government can be described as lots of government services or lots of spending. They are two sides of the same coin. How is describing just one side fairer than describing both?

JLV, You said:

"When people hear "higher taxes", they really think "higher taxes on me"."

Which is of course the truth. 100% of big government economies get there with high taxes on average people, like VATs. Not one high tax European economy gets there with an American-style tax system.

DW writes:

A government that provides a lot of services is a big government but a big government is not necessarily one that provides a lot of services. It could be big because of waste, or subsidies to private companies, or military spending (which may or may not be a "service" depending on your view of foreign policy.)

The bigger government/higher taxes questions is therefore NOT the same as the more services/higher taxes question.

Mike writes:

I am pretty dubious about the premise here that we can figure out which question is most fair using armchair reasoning. For example, let's say that empirically, the 'services' framing correlates very well with other measures of economic liberalism while the 'taxes' framing doesn't, perhaps because a subgroup of very liberal people gets all confused and emotional when you start saying 'high taxes' and stop reasoning all together. (Yes I am deliberately constructing an example I hope appeals to small government types -- I am a liberal, for the record). The 'services' framing would then be the better question to ask, no? Now I myself am not super familiar with the empirical literature on survey construction, but I do expect that the Pew people are! It could be that they picked their framing because they are biased liberals, but it also seems possible to me that their framing is well-grounded. That really leads us back to Chait's broader point which is: Pew are the only people who work on this topic who non-experts should trust. And I think you agree with that, don't you Scott? So I think if you want to make this argument, the burden is on you to come up with empirical evidence (although I understand that off the cuff blog posts are not usually held to that standard -- maybe we can treat this post as a highly preliminary outline of an argument. But then of course liberals should not be expected to find it convincing!).

Michael Stack writes:

I don't think he's arguing that it is 'biased' in the sense of inaccurate, or misleading. Rather, I think he is simply pointing out that that the wording of the question has a big impact on the results of the survey.

Unfortunately Jonathan doesn't address this issue directly but hints at it here:

When I wrote that Ekins does advocacy polling, I did not mean that she is incompetent, dishonest, or reporting inaccurate numbers. Advocacy polling is an honorable profession with perfectly well-qualified polling professionals. I have no reason to doubt Ekins’s credentials.

Still, it is important to understand the job of an advocacy pollster. The most important fact about polling is that, since most people's political opinions are inconsistent and malleable, responses can vary wildly with even slight changes to the wording of a question, or even to the order of questions. Advocacy groups commission polls that, using what is often very neutral-sounding language, invariably produce results that present public opinion in a favorable light for their sponsor. There are right-wing, left-wing, and centrist versions of these kinds of polls. They exist as a publicity device to show that their sponsors’ agenda is popular.

Even if you believe the Reason survey question wording is the most accurate, Chait's observation is still correct.

Michael Stack writes:

Oops, I meant to add this at the end of my previous comment:

Even though Chait is correct about the survey's sensitivity to question wording, it is not clear to me that this supports his conclusion that younger folks are no more libertarian than in the past. I also agree with Scott that the Reason survey question wording is the most accurate of the examples listed, though that may say more about my politics than how well the question was written.

Peter Drake writes:

Scott, I think the point is "big government" is somewhat value-neutral. It could be big, sclerotic and annoying, or it could be big, responsive and wonderful. Just saying "big" doesn't really imply much benefit at all. I'd argue small is similar. Is it Singapore small or Somalia small?

A more fair question would be "a government that provides many services and collects high taxes versus a government that provides fewer services and collects lower taxes".

Roger Sweeny writes:

So shouldn't everyone be asking, "Would you prefer a small government with fewer services and lower taxes or a large government with more services and higher taxes?"?

Aaron Zierman writes:

It strikes me as sad that there is such differences when the wording is changed. Apparently there aren't a lot of thinkers out there who can apply logic to whichever question they are presented.

This being true, do we not have to go even a step further in explaining the question? What exactly might this "big government" or "small government" look like? Big government has more services, yes, but it is also more intrusive. How much do we have to explain?

In other words, what real value do any of these polls have?

MG writes:

Scott, Thomas Sewell has provided the simplest, yet most reasonable answer. In a world where so much of the public debate has been framed by Left-Liberal messaging, anything leaning to the center will be perceived right wing (notice how Libertarians are routinely called conservatives) and anything leaning right of center, extreme. He probably would also say ABC, CBS, NBC, CNN news and the NYT are centrist.

Mark Bahner writes:
Also another option would be, "Do you want big government as long as it's paid for by someone else?"


Scott Sumner writes:

Mike, You said:

"That really leads us back to Chait's broader point which is: Pew are the only people who work on this topic who non-experts should trust. And I think you agree with that, don't you Scott?"

I couldn't disagree more strongly. I trust experts on nothing. Expert favored the policies that caused WWI, the Great Depression, the Great Inflation, the Great Recession, etc. Why would anyone trust experts? The people running the ECB are experts.

Sorry, but if you come over here telling me my opinion is wrong because "experts" don't agree with me, you are kind of missing the point of blogging. You need reasons.

And there can't be any empirical evidence for the accuracy of policy preference polls because there is no way to measure public opinion other than by polls. So the tests of accuracy are merely circular reasoning. (In contrast, you can test election outcome polls.)

Michael Stack, I think if you look at the words "crafted" and "basic" in the quotation I provide, it's very clear he thinks the Pew poll asks the more straightforward question. In any case, 99% of readers would have assumed he meant that.

Peter, I completely disagree. Suppose someone supported a big government that provided lots of nice services, but nonetheless opposed any sort of big government in the US because they correctly assumed it would inevitably be bloated and wasteful. I'd call that person libertarian, whereas Chait and Pew would presumably call them a left-winger. Which is correct?

BTW, that hypothetical person I described is me.

The poll should simply ask about big government, or high government spending, with no editorializing about where the money goes.

Aaron, I complete agree, and will do a post on this soon.

Mark V Anderson writes:

Thank you Scott on the expert issue.

I wouldn't quite say that I trust experts on nothing. Experts should always be consulted and their opinions weighted a bit higher than others. But we should never EVER accept expert opinion without question. This has constantly gotten this country into trouble.

ScottA writes:

Chait is actually following a lot of existing literature on this. For whatever reason, most polls do operate this way (more government/less government in isolation, without a willingness to pay component). Probably more path-dependency than anything else; a lot of prominent poll questions were designed in the 50s/60s and it's tough to change them (we lose comparability over time). The spending questions (spend more/spend less) definitely go back that far. The solution is probably to work out a statistical correction to account for the extent of over-estimated support for government spending so we can keep the existing time-series and get a realistic idea of how these things actually move over time.

Comments for this entry have been closed
Return to top