As an article that was published on the Chicago Tribune website explains, “Arguably the most famous headline in the newspaper’s 150-year history, DEWEY DEFEATS TRUMAN is every publisher’s nightmare on every election night. Like most newspapers, the Tribune, which had dismissed him on its editorial page as a “nincompoop,” was lulled into a false sense of security by polls that repeatedly predicted a Dewey victory. Critically important, though, was a printers’ strike, which forced the paper to go to press hours before it normally would. As the first-edition deadline approached, managing editor J. Loy “Pat” Maloney had to make the headline call, although many East Coast tallies were not yet in. Maloney banked on the track record of Arthur Sears Henning, the paper’s longtime Washington correspondent. Henning said Dewey. Henning was rarely wrong. Besides, Life magazine had just carried a big photo of Dewey with the caption ‘The next President of the United States.’”
“The ink was hardly dry on 150,000 copies of the paper when radio bulletins reported that the race was surprisingly close,” the article continues. “The headline was changed to DEMOCRATS MAKE SWEEP OF STATE OFFICES for the second edition. Truman went on to take Illinois and much of the Midwest in this whopping election surprise. Radio comedian Fred Allen noted Truman was the “first president to lose in a Gallup and win in a walk.” The Tribune blamed the pollsters for its mistake.”
The Problems With the 1948 Gallup Poll
Many statisticians and historians have voiced their opinions about what lead to the surprise victory by Truman and what Gallup did wrong.
Some of the issues were a result of the unique nature of voting (e.g., voter turnout, which way the undecided voters ultimately voted, whether or not supporters of a third-party candidate ultimately changed their minds, etc.)
However, many people who have written about the election explain that one of the biggest problems was the fact that Gallup quit polling voters two weeks before the election. The pollsters assumed that the 14% of voters who said that they were undecided at the time the survey data was collected would vote the same way as the respondents who said that they had made a decision about how they would vote in the election. Add in the other issues that I just listed in the last paragraph, and you can see why it is not so surprising the actual election turned out differently than the pollsters predicted.
The last issue with the 1948 Gallup poll is the one that can be generalized to all types of surveys, not just political polling. The issue: quota sampling.
The practice of quota sampling begins by trying to determine what types of respondents might influence the survey results and then setting quotas to ensure that there are enough members from each subgroup represented in the final data set. In theory, this is a not such a bad idea.
However, it is the second part of quota sampling that is at issue. When using quota sampling, interviewers may be told that they need to interview a certain number of people from each subgroup, but they still get to choose who they interview. Because the interviewer gets to choose who to interview, there is going to be a built-in bias because not everyone has a chance of being selected. For example, the interviewer might interview the exact number of people to meet a gender quota, but the people who they choose might over-represent people fitting into another category not accounted for (e.g., geography, fluency in the English language, what type of car they drive, etc.) that can influence the dependent variable being measured.
The Takeaway for Marketers
While brands that are doing surveys for marketing purposes are not going to have their survey results scrutinized in the same way that presidential polls are, the results of their surveys might be as important or even more important than those conducted on election night.
If the brand is making business decisions that can influence the success or failure of a marketing campaign or maybe even the business itself, then it is extremely important to make sure that the survey data is accurate.
Given all the Do-It-Yourself (DIY) research tools out there, it is tempting for brands to conduct their surveys in-house. However, even the most skilled marketer may overlook something that a trained individual who is dedicated to survey research has seen before and therefore wouldn’t repeat again.
When conducting a survey, there are many things that a business needs to consider, including what questions to ask and how to word the questions, when to field the study, how to administer the survey (e.g., via telephone, mail, in-person, Internet, etc.,) who to include in the sample, and how the data is analyzed and presented after the data collection is completed.
Each step is important.
As shown in this post, who to include in the sample is particularly important, as it can have a huge effect on the survey results.
Therefore, brands might want to consider hiring a strategy and research firm to assist them in their survey research efforts. If they don’t, brands might end up on the wrong side of the business of equivalent of what happened to the Chicago Tribune in the 1948 presidential election.
Photo credit: And all that Malarkey on Flickr.