Whether it’s about an upcoming election, attitudes towards race in the criminal justice system, or which celebrity will take home the award for best performer – people love polls!
But how much do they really love them? Do they love them enough to care about what goes into making them?
Otto Von Bismark once said that “Laws are like sausages, it is better not to see them being made”. However, I would like to paraphrase what Bismark said and say that polls are akin to sausages in that people like to consume them but don’t really care much to see how they’re made.
It’s easy for someone (myself included) to discover a survey or study in the news, immediately generate an opinion about the subject matter, and move on without really asking how those findings were reached. However, the information to which an individual is exposed has the probability of influencing their attitudes, opinions, and even future behaviour.
In a recent article in Public Opinion Quarterly, Kevin K. Banda, a Political Science professor at the University of Nevada-Reno stated that “citizens want to collect adequate information to make accurate decisions while minimizing the costs they face. This accuracy motivation along with citizens’ disinterest in politics encourages them to rely on easily accessible cues when forming attitudes about candidates”. These accessible cues Banda refers to are alternatively known as heuristics or a kind of cognitive short-cut.
Simply put: polls are heuristics. They’re easy to absorb snapshots of what the public thinks, and these snapshots generate opinions.
Questionnaire design is an essential part of an opinion poll. Researchers go through great pains considering how the use or removal of even a single word can influence someone’s response to a specific question. It is important for consumers of polling data to understand this and journalists can help educate readers by including question wording directly in as opposed to alongside their articles.
When Patrick Brown won the Progressive Conservative Party’s leadership contest, the Toronto Star published the findings of a Forum Research poll which asked respondents a series of questions about their attitudes towards Brown and some of the views that he has championed during his political career. The article reported attitudes towards provincial voting intentions, creationism, same-sex marriage, and the sex-education curriculum. While it is appreciated and encouraged that the Toronto Star provided a link to Forum Research’s press release, the columnist failed to disclose in their article what questions Forum asked. I use this article as an example to highlight the importance of disclosing survey questions in media publications and not as a means to criticize the Toronto Star or the respective columnist directly.
The Market Research and Intelligence Association (MRIA), the regulatory body within the polling and market research industry, states in their Code of Conduct that for all reports of survey findings the Client (in this case the Toronto Star) has released to the public, the Client must be prepared to release the following details on request:
- Sponsorship of the survey
- Dates of interviewing
- Methods of obtaining the interviews (telephone, Internet, mail or in-person)
- Population that was sampled
- Size description and nature of sample
- Size of the sample upon which the report is being released
- Exact wording of questions upon which the release is based
- An indication of what allowance should be made for sampling error
The Toronto Star article in question met three of the eight requirements above: method of contact (in this case IVR), the sample size (1,001 people), and the allowance for sampling error (considered accurate to within three percentage points, 19 times out of 20).
I understand that it’s awkward for a journalist to include methodological minutiae such as the dates that the survey was in the field or perhaps going deep into a firm’s sampling frame. Being a journalist is not easy! They are subject to tight and frequent deadlines, word counts, constant research and fact-checking, and of course writing a story that people will want to read. Question wording, however, is not a methodological triviality and if a journalist is willing to report the findings of a question, they should at least include the question that was asked.
I applaud and encourage journalists for incorporating polling data in their publications. However, my criticism lies in the fact that readers are not being told the whole story when question wording is withheld. A journalist may believe that they are informing their readers by including a firm’s press release alongside their article, however, Banda’s comments on heuristics has shown that the probability of a reader following up on external information to further their knowledge (in this case learning what questions were asked) is low.
By including what was asked directly into the article, a journalist enables a more potent heuristic for people to digest without placing a significantly greater amount of effort on the reader. If the questions asked (in this case voting intentions, same-sex marriage, etc.) were included in company with the poll’s findings in the article, then readers would be able to acquire a more comprehensive and three-dimensional view of what public opinion is like in Ontario because they have a better idea of where these attitudes and opinions are coming from.
Market researchers and pollsters spend a lot of time thinking about what questions to ask on a survey, how to frame these questions, when and where to ask these questions in the survey, and why these questions were asked. If journalists are in the business of selling sausage, they should make clear to their readers a little bit more about what goes into making them.