Forbid Or Allow?

By Kathy Frankovic, CBS News director of surveys

I couldn't find any polls conducted about the speech, and whether Americans (or New Yorkers) would have been willing to have Ahmadinejad speak at Columbia. But measuring a question like that may not be as easy as it seems. Questions about who has the right to speak have generated experiments that date back decades.

As far back as 1941, in fact. That's when Donald Rugg published an article in Public Opinion Quarterly, reporting on a test of two questions about speeches against democracy. He did what's known as a "split-ballot experiment." One half of the sample, randomly selected, was asked: "Do you think the United States should allow public speeches against democracy?" The other: "Do you think the United States should forbid public speeches against democracy?" It's fairly easy to divide a sample into random halves. Rugg could have had his interviewer alternate the question from respondent to respondent, or flip a coin to determine which question to ask. Nowadays, you can use the last digit of a phone number (odd vs. even), or just have a computer program randomly make the assignment.

Questions about allowing or forbidding speeches against democracy seem like mirror images of each other, but what turned Rugg's experiment in question wording into a journal article was that they were not. Seventy-five percent of Americans would "not allow" speeches against democracy, but only 54 percent would "forbid" them. The experiment became part of survey lore. Howard Schuman and Stanley Presser reported their 1970's experiments with those two words in their 1981 book, Questions and Answers in Attitude Surveys. Hans Hippler and Norbert Schwartz tested the forbid-allow issue in 1982 in West Germany, and Bregje Holleman did so later in the Netherlands. Philip Gendall and Janet Hoek report finding forbid-allow differences in New Zealand.

In New Zealand in 1989, 13 percent agreed that "the law should allow public speeches which promote racism." When the statement was turned around, so was the answer. Thirty-one percent disagreed with the statement that "the law should forbid public speeches which promote racism."

That same year, in the U.S. General Social Survey, 19 percent said "yes" when asked "Do you think the law should forbid marriages between blacks and whites?" But 32 percent said "no" when asked "Do you think the law should allow marriages between blacks and whites?"

Some of these differences are not just because people are forced to choose between yes and no. Similar results have been found on some questions that give respondents a scale of possible answers.

What is going on? One easy answer is that although it may appear that forbid and allow are two sides of the same coin, they aren't. "Forbid" is a strong negative; while "allow" might imply not just permission, but perhaps even approval. Another is that people without strong opinions about a subject will not endorse either option, causing the difference.

The good news is that these double-digit differences are the exception, and not the rule. And that the practice of conducting split-ballot experiments continues.

CBS News and The New York Times routinely add these sorts of experiments to our questionnaires. When candidates use different versions of their names (like "Hillary" and "Hillary Rodham" Clinton or "Rudolph" and "Rudy" Giuliani), we want to find out whether the name makes a difference. Our good news is that when we have conducted such experiments recently, it hasn't.

And things sometimes change. For example, we have asked Americans to tell us whether the war in Iraq has been worth the costs, defining the outcome as "the result of the war" to one randomly-selected half of the sample and "removing Saddam Hussein from power" to the other half. Earlier in the war, those different wordings resulted in different answers, with more Americans describing the war as worth the cost when we asked about "removing Saddam" than when we asked about "the result of the war" in general. But in more recent polls, there have been small or no significant differences in these questions.

So that 1940 experiment about "speeches against democracy" might not get the same results today as it did then. In a 1989 General Social Survey experiment about a speech by an "admitted communist," 34 percent said they would "forbid" such a speech, while 35 percent said they would "not allow" it - no significant difference at all.

Learning what we can about the impact of question wording makes polling both more interesting and more challenging. And had we polled on Ahmadinejad's speech last week, we might very well have found America's answer - however we asked the question!
By Kathy Frankovic