In 1975, Ann Landers asked her readers a hypothetical question: If you had you had your life to live over again, would you still have children? Over 10,000 replied. 70% said no.
I've seen several random "child-free" activists cite Landers' survey as proof that they know the One True Way. It turns out, however, that this famous survey is a textbook case of selection bias. Statistican David Bellhouse at the University of Western Ontario has a whole lecture about it here. Long story short: Landers' respondents were self-selected for extreme crankiness. Here's what follow-up surveys found:
1. Good Housekeeping repeated Landers' survey with its own self-selected sample. 95% said they would do it over again.
2. The Kansas City Star surveyed a random sample of Kansas City residents. 94% said they would do it over again.
3. The clincher: Newsday did a national random sample. 91% said they would do it over again.
Ann Landers herself came close to describing what is one of the major problems with her survey. At first, in one column she listed potential psychological and cultural reasons for the negative response. Then in a later column she wrote what is perhaps the real reason for her negative response.
“I believe the logical explanation for this phenomenon is (a) the hurt, angry and disenchanted tend to write more readily than the contented, and (b) people tell me things they wouldn’t dare tell anyone else.”
Her choice (a) is the real culprit. The responses to the survey were not a true cross section of society and reflected the opinions of only those who felt strongly enough to write in. [footnotes omitted]
Now you could say that random sampling does nothing to cure social desirability bias: People deny regret because they're supposed to say that they Love Their Kids More Than Anything.
That's true, but there are two countervailing biases to consider.
First, when an answer is rare enough, many of the rare answers are false positives caused by respondent confusion. Imagine asking 1000 people, "Are you a billionaire?" I bet at least 1% (way above the true rate in the population) would say yes out of sheer incomprehension.
Second, there is a segment of the population that gives socially undesirable responses just to be curmudgeons. I came across a random webpage where a guy said that his mom, who had seven kids, liked to say "If I had to do it over again, I would have raised hogs." I've met people like this before [shudder], and my best guess is that they're just being ornery.