Winter 2010

Survey says... or does it?

It was a result guaranteed to make headlines. A 2006 survey conducted by the American Medical Association (AMA) found that college women were engaging in disturbing rates of binge drinking and unprotected sex over spring break.

There was only one problem. It wasn’t true. Or at least the responses weren’t representative—since three quarters of the 644 women surveyed had never gone on a spring break trip. What they had done was volunteered to answer a number of questions as part of an online survey panel that researchers incorrectly billed as a random sample of young females in the United States. So the news stories and TV reports touting an epidemic of “girls gone wild,” were actually based on flawed data. The AMA ultimately changed the wording in its methodology, but the damage had been done.

The story illustrates what Notre Dame researchers Ken Kelley and Timothy Gilbride see as one of the bigger challenges in evaluating surveys today. You have to understand the mechanics of how they’re put together, notes Gilbride, who teaches market research and studies statistical methods for gauging consumer preferences. “Because whether or not you have a job in
marketing, you’ll be hearing about things like samples all your life.” To help, he and Kelley agreed to explore some common misinterpretations and errors in surveys and offer some tips on how to spot them.

Rule #1:
 
Watch for sloppy samples

It’s difficult to measure the likes and dislikes of an entire population. But as the AMA report showed, not all samples are equal. In order for the results of a poll or survey to be statistically applicable to the group you’re studying, the sample has to be random. This means each person in the population of interest (e.g., college age women, retirees, voters, etc.) must have an equal chance of being selected.

Unfortunately, in this era of rapid technological change, “random” often goes out the window, notes Kelley, assistant professor of management, who teaches statistics and specializes in research design and statistical analysis. “For example, an Internet survey may inadvertently bias results because senior citizens are less likely to be online compared to their younger counterparts,” he says. “Or the senior citizens who are online may be more likely to take the survey than their younger counterparts, so their responses could be overrepresented in the population.”

Either way, you run the risk of getting a skewed picture of the group you’re trying to assess. This is currently an issue in telephone surveys. Most of the databases from which random numbers are drawn include only landlines since cell phones are often unlisted. As a result, as much as 20 percent of the mobile-only population, which tends to be young, single and low-income, might be excluded from selection.

Meanwhile, in a TV call-in poll or online survey, you get a sample that tends to reflect the opinions of those who feel very strongly about the issue. Gilbride acknowledges the media has gotten better at seeing such surveys for what they are, but not always. Two years ago, the American Association for Public Opinion Research (AAPOR)—which refers to such self-selected opinion polls as SLOPS—took MSNBC.com to task for publishing a survey purporting to show that men were the preferred leaders in the workplace. In her letter to the editor, AAPOR President-elect Nancy Mathiowetz wrote: “Perhaps a better title for this story would have been ‘Men rule—at least according to a lot of people who decided to take part in our online survey and who may or may not be representative of anything.’”

Rule #2:  

Weigh the responses

Another clue to the reliability of a survey is its response rate. If only 15 people respond out of hundreds surveyed, it’s risky to project their views to the total population of interest.

It doesn’t help that responses are becoming more difficult to obtain. The proliferation of online surveys—which once drew a 90 percent response—has caused people to tune out, with participation now closer to 10-30 percent. This mirrors what has happened with phone surveys, where because of irritation, lack of time or perceived bias of the pollster, more people are hanging up. “The thing that is more alarming is the refusal rate,” says Gilbride. “You’re not sure if those who don’t participate have systematically different views.”

This situation has prompted the use of panels, like the one from which the AMA drew its survey participants. But since this is done by finding groups of people who will agree to take part in surveys, the results must be used with caution, Gilbride says. If you know every month you have to report what your consumption patterns are, you might start thinking differently than the average consumer or changing your behavior since you have to justify your purchases.


Rule #3:  

Pay attention to all the numbers

How big should the sample be? Most political polls include at least 800 people. The AAPOR suggests 1,000 when looking to find out about national opinion. Even when you have an optimum number, a different sample taken from the same population is almost certain to give you different results. Because of this variability, which is larger with smaller sample sizes, a margin of error is calculated to give a more accurate picture. The margin of error is usually noted in the fine print of the survey methodology in a range, such as plus or minus 4 points. “People do tend to ignore it, but that’s a mistake,” says Kelley.


Rule #4:
 
Make sure you trust who’s asking the questions

Who is behind the survey? Was the research done independently by a reputable organization? If you’re looking at a Gallup Poll or a brand-name market research firm or university survey which adheres to very high standards, it’s easier to have a greater comfort level with the results.

If you have your doubts or think the survey-backer is an organization with an agenda, go back to the methodology. Get a copy of the questions if you can, says Kelley. “If a group with a vested interest in the outcome of the survey wrote the questions and/or the pollster script,” he adds, “there may be some inherent issues with the wording and or voice inflection used by those conducting the poll.” (See below). Reliable surveys typically adhere to fairly strict guidelines in terms of how they ask questions and collect responses. What’s important, both researchers agree, is to consider the merits of each one on an individual basis.

“One error is you don’t believe any surveys,” says Gilbride, “the other error is to believe them all.”

—Kathleen Murray (’82) is a business writer based in Vienna, Va.