A Review of the Public Opinion Polling Conducted During the
2017 Calgary Election

A report submitted to the Marketing Research and Intelligence Association
Prepared by:
Christopher Adams, Paul Adams and David Zussman
May 31, 2018

 A Review of the 2017 Calgary Election
Acknowledgement
We would like to thank the MRIA for the logistical and research support that was
provided to us in the course of our review. Without the commitment of the organization
and its leadership, we would not have been able to complete our task in the time frame
that we set for ourselves. In particular, we would like to thank Kara Mitchelmore, former
CEO of MRIA, Greg Jodouin, a partner of Pace Consulting, and the following MRIA
staff: Dan Jackson, Etta Wahab, Erica Klie, and Wayne Lawrence.
While MRIA provided the research and logistical support, all of the interviews, analysis,
drafting, conclusions and recommendations are the work of the three members of the
review panel. At no point did the review panel receive any guidance or direction from
MRIA, its members or any other party.
We hope that this report will be helpful to the public opinion research community in
adding a further level of professionalism to their activities and to the many people who
use public opinion research to understand our society in all of its complexity.

Christopher Adams
Paul Adams
David Zussman

2

 Executive Summary
Accurate public opinion polling can provide voters with information about the views of
their fellow citizens and plays an important role in modern democracies. It can draw
voters’ attention to particular candidates and issues and it can help them make tactical
decisions as they consider how to vote. Polls are part of a feedback loop voters expect at
election time to understand the dynamics of the campaign
The Marketing Research and Intelligence Association (MRIA) is the industry association
representing market researchers in Canada, which includes the polling industry. In
reaction to widespread concern that the polling in the municipal election campaign in
Calgary 2017 did not accurately reflect the intentions of voters there, and in doing so
misled the public and deleteriously affected the campaign, the MRIA announced that it
would commission a study.
In January 2018, the MRIA announced that a panel had been appointed consisting of
three independent academics with experience in the polling industry to review the polling
in the Calgary 2017 campaign. The panel members were: Dr. Christopher Adams, Rector,
St. Paul’s College at the University of Manitoba; Prof. Paul Adams, Associate Professor,
School of Journalism and Communication, Carleton University; and Dr. David Zussman,
adjunct professor in the School of Public Administration at the University of Victoria.
The MRIA asked the panel to review three issues:
1. The degree of inaccuracy in the Calgary election polls;
2. The reasons for the inaccuracy;
3. Whether the polling results were adequately communicated to the general public.
With the assistance of the MRIA staff, the panel collected all publicly available
information about polls published in the 2017 Calgary campaign, media coverage of the
polls and social media interaction regarding the polls, as well as other documents such as
MRIA standards and election laws and regulations.
The panel then invited many of those prominently involved in the Calgary campaign as
well as relevant experts to meet. The invitees included the polling companies involved in
the Calgary campaign, the organizations that had commissioned the polls, the main
contestants in the Calgary mayoral campaign and their advisors, Postmedia (which had
sponsored a series of campaign polls), academics and polling experts who had
participated in the debate about the polls during the campaign, journalists, and
independent experts on polling methodology, communications and journalism. With
some important exceptions, most of the invitees agreed to meet with the panel, which
travelled to Toronto and Calgary to conduct interviews. The panel also conducted several
interviews by phone where this was most convenient.
None of polls released to the public during the Calgary election campaign approximated
the actual election result, which was that the incumbent mayor, Naheed Nenshi, won by a

3

 margin of 7.65 percentage points over his principal rival, Bill Smith. This victory came as
a surprise to many Calgarians because the most well-publicized polls during the election
campaign were those conducted by Mainstreet Research for the Postmedia newspaper
chain, publisher of Calgary’s two daily newspapers, the Calgary Herald and the Calgary
Sun. All three of Mainstreet’s published polls suggested that Nenshi was trailing by large
margins to his relatively much less known rival – at one point by nearly 17 percentage
points. The panel heard that these polls, which received the greatest media attention
during the campaign because of their number, their startling results, and their association
with the two Calgary dailies, significantly affected the course of the campaign. They
threw Nenshi’s campaign on the defensive, gave impetus to Smith’s campaign, and
possibly doomed the prospects of another candidate, André Chabot, who Mainstreet’s
poll suggested was not a close contender.
The Mainstreet polls, which were conducted by IVR, triggered an acrid debate in the
media and on social media, in which the Nenshi campaign attacked the firm’s motives
and independent academics questioned its results and methodology. Mainstreet
executives responded with unshakeable confidence in their results and attacked their
critics, often in personal terms, at one point suggesting there would be “payback” after
the election results were known. The panel found that Mainstreet’s overconfidence and
its contentious style of public debate significantly contributed to the embarrassment to the
industry when its results were proven to be radically mistaken. Mainstreet’s public
confidence during the campaign also contrasted with internal concern about its results
that led to adjustments in its methodology.
As has proven to be the case with similar inquiries into polling failures in other
jurisdictions, the panel found that the answers raised by the problems in Calgary were
complex. Rather than relying on Random Digit Dialing (RDD) the Mainstreet Research
polls drew their sample from a “directory” that seems to have greatly underrepresented
younger voters. In reaction to criticism of its first poll, Mainstreet decided to stratify its
sample by municipal ward, which inadvertently stripped out much of its cell phone
sample, exacerbating its problems with reaching a representative sample of younger
voters. When the election proved to have unexpectedly high turnout, especially among
young people, these vulnerabilities were yet further exposed.
Two other polls played significantly in the public debate during the Calgary campaign:
one commissioned by the LRT on the Green Foundation and the other by an academic
group, the Canadian Municipal Election Study (CMES). Both of these polls were
conducted online and both showed very large leads for Nenshi – outside of their stated
margin of error when compared with the eventual result. These polls tempered what
turned out to be the erroneous narrative created by the Mainstreet polls that the mayor
was likely to lose but may have added to public and media confusion.
As discussed in the report, the poll commissioned by the LRT on the Green Foundation
suffered from serious design defects, particularly concerning questionnaire sequence. The
poll commissioned by CMES, which was the most methodologically sound of those
released in Calgary, nonetheless was not perfectly suited for the purpose of public release

4

 because of its long survey period. As detailed in the full report (Section 2), none of the
polls met the MRIA’s full requirements for disclosure.
In reviewing media coverage, the panel found that Postmedia in particular was not critical
enough in its reporting of polls for which it was partially responsible. Some of the
academics we interviewed said that the Postmedia newspapers did not report on
methodological concerns they raised with reporters. Moreover, Postmedia did not share
with its own readers concerns it had about the polls and the degree to which Mainstreet
was altering its methodology to address them. The panel found that many media outlets,
including the Postmedia newspapers did report on the dramatic discrepancies between the
Mainstreet polls and others published in the campaign, but that the coverage was not
technically sophisticated and would not have left readers fully equipped to evaluate the
polls.
The panel heard from many of those it interviewed that the MRIA is well positioned to
play a more active role in creating a framework for higher standards in published polling
– a suggestion that its leaders appeared to welcome when we met with them. The panel
made dozens of recommendations aimed at illuminating a path towards that goal.
The key features of those recommendations include:
•
•
•
•
•
•

A continuous updating of industry standards, particularly around disclosure
A standardized disclosure checklist that polling firms would publish
contemporaneously with their polls and which would provide a tool of
accountability to the media, public and the MRIA
An annual report on the state of the polling industry that would evaluate the
adherence to industry standards
The development through these tools of a capacity for the MRIA to react in real
time to public or media concerns about polling methodology
Enhanced services to the media and public, in terms of understanding polls and
providing accountability
The development of codes of conduct by polling firms for their interactions with
the media and on social media

With regard to the media:
•
•
•

Full disclosure of the nature of commercial and financial relationships when
obtaining polling data on the basis of exclusivity
The application of normal journalistic context and skepticism of polls, including
those obtained through an exclusive relationship
The development of more sophisticated technical skills for the reporting of polls
(a task in which the MRIA can be helpful)

With regard to Mainstreet Research:

5

 •

That Mainstreet Research and the MRIA agree on a neutral academic auditor who
can evaluate the firm’s current practices and adherence to standards with a view
to rejoining the MRIA

6

 Introduction
Naheed Nenshi was re-elected as mayor of Calgary in the October 2017 election. While
the mayoralty race was closely followed across the country because of Calgary’s
importance as a national business centre, the election campaign became known for the
fractious public discourse that paralleled the election campaign. In particular, the
campaign was dominated by acrimonious social media commentary among a small
number of pollsters, commentators, journalists, and political activists, some of whom
were working for the two leading candidates, Naheed Nenshi and Bill Smith.
At the outset of the campaign there was little evidence that the election would become a
battleground among the various players. In fact, there appeared to be scant interest in the
election in the summer of 2017. The expectation was that, in the end, the major
opponents to the incumbent mayor would be drawn from the pool of city councillors.
The relaxed pace of the election campaign was broken when a number of polls were
published by Mainstreet Research that suggested the public was growing weary of Mayor
Nenshi’s style of governing and his ongoing disputes with the business community. All
semblance to a routine election campaign was shattered with the publication of three
Mainstreet Research polls conducted for the city’s two Postmedia newspapers, the
Calgary Herald and the Calgary Sun1, suggesting Nenshi was likely to lose the election,
and lose badly, to Bill Smith. Subsequently two competing polls were released suggesting
Nenshi in fact had a comfortable lead. The controversy that ensued among Calgary-based
interested parties initially was centred on Mainstreet Research’s methodology but it later
shifted to allegations of the personal qualifications of Mainstreet’s principals and their
business relationship with the Postmedia newspapers that sponsored the election polls.
In the end, the controversy raised many issues about the efficacy of contemporary
political polling, the role of the media in sponsoring and reporting on political polls, the
dynamics created by Twitter ‘conversations’, and the appropriate role of professional
associations in regulating the industry and their members in the event of disputes and
challenges around competency and normative behaviour.

1

The Calgary Herald and Calgary Sun are owned by Postmedia Network Canada
Corporation (also known as Postmedia) and consists of the publishing properties of the
former Canwest Corporation and the English-language operations of the former Sun
Media chain, with many operations in newspaper publishing, news gathering and internet
operations.
8

 This report is the final step in a review that was commissioned by the Marketing
Research and Intelligence Association (hereafter MRIA), which is the professional
association for public opinion and market researchers in Canada.
The report is divided into 4 sections. Section 1 sets the stage by describing the MRIA,
and the mandate and members of the review panel. It also provides an overview of the
background to the 2017 Calgary election and lays out some of the areas of concern that
framed the work of the review panel. The section closes with a brief description of the
methodology and timeline that guided the panel’s work. Section 2 looks at industry
standards used in political polling in Canada and is followed by a section that provides a
detailed description of the three Mainstreet polls, the one sponsored by a group called the
LRT on the Green Foundation, which favours an extension in the city’s light rail transit
system along the “Green Line”, and finally the one commissioned by an academic group
called the Canadian Municipal Election Study. Section 3 examines the ways in which
Mainstreet’s and the other polls were communicated by Postmedia, Mainstreet Research
and other pollsters. This part also describes how social media served as the battleground
for the vigorous debates that ensued during the election. In the final section of the report,
Section 4 includes an overview of the major findings of the review and concludes with a
series of recommendations for the MRIA.

Section 1
Setting the Stage
The Marketing Research and Intelligence Association (MRIA)

Nine days after the Calgary election, on October 26 2017, the Marketing Research and
Intelligence Association announced its intention to initiate an independent inquiry into
underperforming and conflicting election polling results published during the municipal
election in Calgary.
The MRIA is the national self-regulatory body representing all sectors of the market and
survey research industry in Canada. Its members include more than 1,200 individual
research professionals and more than 150 corporate members. These corporate members
are drawn from small to large research agencies, and include many buyers of research
services, such as financial institutions, major retailers, insurance companies,
telecommunications firms, and manufacturers.

9

 The MRIA is the national body responsible for the ‘development of robust, world-leading
industry standards’. In this role, after the election, the MRIA wanted to know more about
the information and data that the pollsters publicly released in order to support their
polling results. It also wanted to know whether industry standards for public opinion
research were met by those who published polling data during the election campaign.
In the press release, MRIA CEO, Dr. Kara Mitchelmore, said that the “MRIA believes
that what happened in Calgary needs to be better understood so that it can be prevented in
the future”. She went on to call on the pollsters involved in the election to submit their
data and methods to the independent review panel that was being organized by the
MRIA. It was anticipated that the reviews findings would help ensure that ‘polling
research remained a vital and useful tool for democracy’.
Mitchelmore further stated that: “[E]lection polls are a vital tool in democracy. Polls help
tell the election story, highlighting how things are moving and the currents and drifts in
public opinion. In doing so, they inform the electorate in their decision-making as they
head to the ballet box. In the Calgary elections, however, the wide discrepancies between
various polling studies became part of the media story and acted as a distraction rather
than an informative democratic tool.”
MRIA furthermore argued that: “[T]he reason that polls work – when they are done well
– is that they are based on reliable science. This includes rigorous adherence to standards
that have been developed over many years by the world’s best statistical experts. This
provides a scientific foundation to validate that study findings are accurate and a true
reflection of public opinion at a particular moment in time. In Canada, adherence to
MRIA's internationally-recognized standards and code of conduct has resulted in multiple
examples of election survey results that have mirrored nearly exactly the final voting
outcomes, sometimes down to the decimal.”
In the view of the MRIA, the polling results and the controversy around their publication
has had a serious impact on the credibility of the industry. Mitchelmore described it this
way:
The failure to accurately paint the mayoral election story in
Calgary has led many to question the value and robustness
of polling methodology and has shaken confidence in our
industry. It is the reality of our industry that bad election
polls or the undisciplined conduct of pollsters can tarnish the
industry’s credibility and call into question the reliability of
all survey research.

Public opinion polls are subject to the same professional, and ethical requirements as
other forms of market and social research as set out in the MRIA's Code of Conduct for
Market and Social Research. All researchers and research users must conform to this
10

 Code. The MRIA’s Polling Standards for the Canadian Marketplace apply to all instances
where Public Opinion Surveys and Polls are conducted. Specifically, the standards
document does the following 2:
•

Sets out the ethical rules that public opinion researchers must follow.

•

Highlights the key information that must be made available to maintain
transparency when the results are published;

•

Specifies standards to guide the agreements to be in place with those who
commission polls to ensure published survey results are presented in an unbiased
way;

•

Highlights the core methodological principles that apply in the design and
conduct of such research;

•

Underlines some of the additional issues that arise with specific forms of public
opinion polls;

•

Includes federal regulations that all researchers must adhere to, including the
Canada Elections Act and the CRTC 3.

Finally, the MRIA made the commitment that participants in the Calgary election would
be invited to share information with the review panel and relate their views and
perspectives regarding the matter. The MRIA would act as secretariat for the review
panel and once the work was completed, the MRIA would publish the full report.
Accordingly, on January 31, 2018 the MRIA announced the creation of a panel made up
of three academic experts who were very familiar with public opinion research and
polling standards. Their task was to “review underperforming and conflicting election
polling results published during the municipal elections in Calgary”. In particular, the
primary objectives of the review were to investigate, in an open and transparent fashion,
the following three issues:
•

The degree of inaccuracy in the Calgary election polls;

•

The reasons for their inaccuracy; and

•

Whether the polling results were adequately communicated to the general public.

2

The standards were created by the MRIA Standards Committee using a wide range of
international standards.
3
From MRIA Code of Conduct for Market and Social Research, Appendix ‘L’, Polling
Standards for the Canadian Marketplace
11

 The Review Panel
Accordingly, the independent review panel was made up of Professors Christopher
Adams, Paul Adams and David Zussman. Each of the three review team members are
university academics who, at various points in their careers, have worked in the polling
industry - but no longer have any formal or informal relationship to commercial polling
firms - and who use public opinion research in their academic and professional work. A
more detailed description of their qualifications and areas of expertise is presented below.
Christopher Adams is a Certified Market Research Professional (CMRP). He holds a PhD
from Carleton University. After completing his doctoral studies in 1995, he held senior
positions in the polling industry for a number of firms, including the Angus Reid Group
and Probe Research. In 2012, he was appointed to serve as Rector of St. Paul’s College at
the University of Manitoba where he also holds an Adjunct Professorship in Political
Studies. His areas of teaching include marketing research for MBA students as well as
quantitative research methods in business and the social sciences. He continues to be a
frequent media commentator on topics relating to polling and his academic writing has
appeared in numerous books and journals, including The Canadian Annual Review of
Politics and Public Affairs and the Journal of Parliamentary and Political Law. He is
currently writing a history of the Canadian public opinion research industry.
Paul Adams was educated at the University of Manitoba, Oxford University and
Columbia University. He is currently an associate professor in the School of Journalism
and Communication at Carleton University. He was a lecturer in Political Studies at the
University of Manitoba (1982-4), and taught journalism at Massey University in
Wellington, New Zealand (2012). As a journalist he worked in the parliamentary bureaus
of CBC Television, CBC Radio and the Globe and Mail, and also served as the Globe’s
Middle East correspondent. He was involved with the design, interpretation and reporting
of political polls with CBC and the Globe and Mail. Adams managed and presented focus
group research on Palestinian political issues -- conducted for the National Democratic
Institute -- to Palestinian political parties as well as the office of the Vice-President of the
United States. He was executive director of EKOS Research (2005-10), during which
time he helped manage polling during two elections and worked on political polls with
The Toronto Star, Globe and Mail, La Presse and CBC. He is the author of several
books, writes a weekly a column on the media for iPolitics.ca and has written recently for
healthydebate.ca, the Ottawa Citizen, and The Walrus.
David Zussman has held executive positions in academia, government and the not for
profit sector. In academia, he held the Jarislowsky Chair in Public Sector Management at
the University of Ottawa and served as the University’s Dean of the Telfer Faculty of
Management and Director of the Graduate School of Public and International Affairs.
Prior to that, he was a professor in the School of Public Administration at the University
of Victoria. In the not for profit sector, Zussman was President of the Public Policy

12

 Forum and in government he was Assistant Secretary to the federal Cabinet for
Machinery of Government and served as Commissioner of the Public Service. He has
authored many articles in refereed journals, published a number of books in Canadian
public administration and has had a regular public policy column in the Ottawa Citizen
and Canadian Government Executive. Currently, Zussman serves as the Chair of the
Board of Directors of the Canadian Centre for Ethics in Sports and is a member of the
Environment Canada and Climate Change audit committee. He is also President Emeritus
of the Public Policy Forum, an adjunct professor at the School of Public Administration
at University of Victoria and Senior Fellow at the Graduate School of Public and
International Affairs at the University of Ottawa. He was the recipient of the 2017 Vanier
Medal for Public Administration in Canada.

The 2017 Calgary Election
The 2017 municipal election was a contentious and fractious one for the citizens of
Calgary. This was due in part to the way in which the campaigns were conducted and the
issues debated by the leading candidates. The conflicts were also fueled by the general
economic climate that surrounded Alberta and Calgary in particular. After many years of
dramatic economic growth in the energy sector, by 2016 the economy was in deep
recession. The TD reported that the ‘recession gripping Alberta would likely go down in
history as one of the most severe the province had ever endured’. The report also
predicted a three per cent GDP contraction in 2016, adding up to a 6.5 per cent loss since
the downturn started in 2014.
All of this followed on the heels of one of the worst floods in 2013 when more than
100,000 Calgarians were forced to leave their homes in search of higher ground. In the
aftermath of the floods, the mayor became a household name in Calgary and across the
nation and developed a reputation as the champion of progressive policies, a supporter of
reconciliation with first nations, the Keystone XL pipeline, construction of bike lanes and
the Calgary Pride parade. He also took a strong stance against urban sprawl and the
disproportionate influence of property developers and downtown business leaders who
were championing more public investment into the construction of a new arena for the
ever-popular Calgary Flames.
During the summer months of 2017 the leading candidates organized for the upcoming
October election by selecting their campaign teams and recruiting volunteers in all of the
14 wards that made up the boundaries for the more than 666,000 eligible voters. Prior to
the campaign, on April 27, Mainstreet had reported Nenshi was dropping in popularity.
To the surprise of many observers, Mayor Nenshi’s approval rating had dropped from

13

 65% in January to 52%.4 It therefore began to look like the campaign might be
competitive.
On September 30, the Calgary Herald and Calgary Sun published a poll conducted by
Mainstreet indicating that Bill Smith, to that point a relatively little-known public figure
and first-time candidate for elected office, was leading the incumbent Nenshi by nine
percentage points among committed voters (the results of this and other polls are
provided in more detail in Section 2). The poll attracted considerable reaction from the
Nenshi election team as well as from a wide range of professional and academic
observers since Smith was an unknown quantity in Calgary municipal politics, although
he had served as the president of the Progressive Conservative Party of Alberta under Ed
Stelmach.5
On October 7, the Calgary Herald published another front-page story on the results of a
second Mainstreet Research poll. This time the gap between Smith and Nenshi grew and
the incumbent mayor was now almost 17 points behind his challenger with 31%
supporting Nenshi and 47.6% supporting Smith. Perhaps equally important, André
Chabot, a well-respected Calgary city councillor, was only able to garner 5.6% support
among committed voters rendering his candidacy for mayor unlikely and his campaign
unviable. This second poll, which also was carried on the front page of the Calgary
Herald and reported widely among other Alberta media outlets, caused great confusion
and anger among members of the Nenshi election team, political commentators,
consultants, polling professionals and academics who were closely following the election.
In their view the findings were not consistent with their own understanding of how voters
were likely to vote. The debate about the Mainstreet polls was further inflamed when
allegations were made by some Nenshi supporters that Mainstreet was ‘in cahoots’ with
the Calgary business community to unseat Nenshi.
The third and final Mainstreet poll was released on October 13, only days before the
election and it provided a third slice of evidence that Smith would win the civic election
by a comfortable margin. In this poll Smith had a lead of 13 percentage points.
In the end, the Mainstreet polls, powered by the dominance of the two city dailies, and
the ricochet effect it had through its reporting in other media outlets and social media,
created the dominant narrative for the campaign: that the election, in short order, had
become a two-horse race between Nenshi and Smith with the mayor struggling to stay
competitive. In response to the Postmedia reporting of the Mainstreet polls, two other
groups decided to release polls before the election in an effort to balance the impression
created by the Mainstreet surveys that the race was all but over. The first poll was
released by an interest group that favours an extension of Calgary’s existing Light Rail
4

Mainstreet Research, Media Release, April 27, 2017,
https://www.mainstreetresearch.ca/nenshi-approval-continues-drop/.
5
See http://www.cbc.ca/news/canada/calgary/bill-smith-running-mayor-calgary-election1.4157107.
14

 Transit (the “Green Line”), which was an important election issue. The other survey was
released by a group of academics that was conducting research on behalf of a SSHRCsupported academic project known as the Canadian Municipal Election Study (CMES).
The CMES is a longstanding research program that systematically examines municipal
elections in a number of Canadian cities. Both of these polls were released in advance of
the election in order to counter the weight of the Mainstreet polls. Contrary to the nowdominant narrative of the campaign, these two polls suggested that Nenshi had a large
lead over Smith and was on track to a significant win.
None of the polls was close to the actual result – they were all outside the margin of error
– but the LRT on the Green and CMES polls were significantly closer and correctly
identified Mayor Nenshi as being in the lead.
The debate over the polls triggered bitter and highly charged exchanges on social media,
particularly Twitter, between Mainstreet’s executives, Maggi Quito and David Valentin,
and those who questioned the company’s methodology and conclusions. The Mainstreet
polls were criticized by some academic and independent experts on methodological
grounds, especially with regard to anomalous findings with regard to women and younger
voters. Some aides to Mayor Nenshi went further and suggested Mainstreet was
manipulating its results for political or commercial ends. Mainstreet responded furiously
to all of this criticism, referring to some critics as “appalling”, referring to a rival survey
as a “pseudo-poll” and reportedly threatening to “single out” its critics after the election.
The electoral campaign, which started in earnest at a slow pace in early September,
eventually became a negative and acrimonious affair. While there were many
opportunities for voters to watch and assess the qualities of the candidates during formal
debates and in Q and A sessions, the election was often dominated by the controversy
around Mainstreet Research’s polls and the negative discourse among the key players.
On October 16, 2017, the citizens of Calgary voted to re-elect Mayor Naheed Nenshi for
a third term. Once the dust had settled, of the 387,583 votes cast (out of the 666,663
enumerated voters) Nenshi garnered 199,131 in comparison with the 169,587 received by
Bill Smith (51.4% versus 43.8%). André Chabot received 11,949. The participation rate
was 58.1%, which was high by historical standards for a Calgary municipal election.

The Aftermath of the Calgary Election
The election results of the Calgary 2017 municipal election raised significant questions
among the public, media and expert commentators about the polling industry. In
particular, it highlighted weaknesses in predicting election outcomes, methodological
problems with some current polling techniques, the conflicted role of pollsters who
increasingly act as political pundits and brand-builders rather than as dispassionate social
scientists, the lack of expertise of journalists in reporting on political polls, the
ambiguous role of media sponsors of polls, the powerful impact of social media in

15

 degrading the quality and tone of legitimate discourse, and the influence of polling on
campaigns and voter preferences.
In addition to any impact that polling may have had on the course of the campaign, there
was a general view among those we interviewed during our visits to Toronto and Calgary
that the public acrimony which characterized the interactions between Mainstreet
Research and the Calgary-based and national public opinion research communities had a
significant impact on the reputation of the public opinion industry in Canada and on some
of the polling companies and their principals (many of whom are not members of MRIA).
After the election, Mainstreet’s CEO, Quito Maggi, responded to the criticism that had
been expressed about his sampling methodology from the Calgary survey research
community, notably Brian Singh, Janet Brown, Melanee Thomas, and Duane Bratt, by
commissioning two studies. The first report, undertaken by Dr. Joseph Angolano, vicepresident of Mainstreet Research, specifically addressed the methodological issues.
Justin Ling, an independent free-lance journalist, authored the second review, which
examined the various ways in which Mainstreet’s results became part of the election
rhetoric and how they sparked such a high degree of disharmony on social media.
The Angolano report argued that special circumstances in Calgary - namely, high
numbers of younger voters who were difficult to reach and an unusually high voter
turnout - had led to the serious errors in Mainstreet’s polls. The Ling report criticized
Mainstreet’s overconfidence during the campaign and suggested that its executives had
failed to distinguish between their roles as social scientists and those of political pundits.
The panel members found these reports useful in their own inquiries and reflect on many
of the issues they raise in what follows in this report.
During the campaign, and even after the election results, Mainstreet executives made
threats of legal action against some of its critics. It subsequently decided not to pursue
this course. Maggi later apologized publicly for Mainstreet’s disrespectful behavior
during the election and he told the panel members that Mainstreet was committed to
implementing the recommendations of both the reports it had commissioned.
The panel found a residue of considerable anger among all the participants in the debate
over the polls, and this included both Maggi and his numerous critics. Mayor Nenshi and
his election team expressed frustration with Mainstreet’s role and continued to suggest, as
they did during the campaign, that there might have been a cabal among Mainstreet,
Postmedia, and elements of the Calgary business community aimed at unseating him.
They expressed bitterness at a caustic campaign, which they said included unwarranted
attacks on the mayor’s integrity as well as racist attacks on him personally -- an acrid
atmosphere to which they feel the Mainstreet polls contributed. Maggi in turn expressed
deep resentment at these attacks on his integrity. Meanwhile, the academics and pollsters
who felt menaced by Mainstreet’s words and actions during the campaign remain deeply
aggrieved, notwithstanding Mainstreet’s apologies for its behavior and suspension of any
legal action.

16

 Issues
Election polls in Canada have a reasonably good track record in guiding the public and
media towards the actual outcome.6 However, there has been a lot of concern expressed
by experts and the media about the accuracy of election polling in many jurisdictions
around the world, including Canada. For example, polls incorrectly predicted the
outcomes of the British general election of 2015, the provincial election in Alberta in
2012, the provincial election in British in 2013, and of course the Calgary municipal
election in 2017.
In addition to the particular circumstances surrounding the Calgary election the polling
industry over the past two decades has been subjected to a large number of important
developments that have dramatically transformed the traditional business model and the
ways in which data are collected and analyzed.
One of the obvious reasons for the difficulty in capturing accurate assessments of the
general population is that it is increasingly hard to draw representative samples using
traditional sampling techniques. There are many well-documented challenges such as
finding suitable respondents and encouraging them to participate in surveys. This is
especially the case for certain demographics such as young, urban and working citizens.
Moreover, every year fewer people are using landlines complicating the task of
identifying and contacting potential survey respondents. Online techniques face other
challenges such as creating a representative list from which to draw respondents.
Another challenge the public opinion research community is also facing is the increasing
reluctance of the public to participate in polls. It is not unusual to have response rates
lower than 5% in general public opinion surveys which raises the issue of how persistent
pollsters are in following up on their samples and how they replace those sampled who
are none respondents. Difficulties with reaching some demographics has in turn led at
times to an over-reliance on the use of weighting.
There are other factors not directly related to polling techniques that have complicated
the task of predicting elections, most notably a dramatic decline in participation rate in
most elections. For example, forty years ago, more than 70% of eligible voters often cast
their ballots in federal elections, while 50% is now more typical. When the percentage of
eligible voters declines it is more difficult to estimate whether the characteristics of
voters is more or less consistent with those who answer public opinion surveys.
Moreover, there may be unexpected volatility in turnout. Interestingly, perhaps due to the
controversy during the Calgary election campaign, the participation rate increased
dramatically to 58% in 2017 from the 39% in the 2013 election.

6

See, for example, Bryan Breguet http://www.tooclosetocall.ca/2018/05/can-we-trustcanadian-polls.html
17

 In addition to these challenges, there has been a dramatic change in the business model
that has traditionally supported public opinion research on political races. The low cost to
entry into the polling business due to the availability of inexpensive analytical and
processing tools, cheap data storage and pools of potential respondents, has drawn many
new entrants into the polling business. Meanwhile, cash-strapped news organizations play
less and less – sometimes nothing at all. With the increase in competition among the
pollsters, some within the industry have built their brands by becoming political
commentators and pundits in the media. It now is apparent that the role of pollster as
social scientist and methodological expert has, at times, morphed into using the role of
pundit as a marketing tool. For the public, it is often difficult to discern evidence-based
analysis from punditry. The blurring of the lines between social scientist, pundit and
brand-building entrepreneur is made more complex when polling companies enter into
‘exclusive’ contractual relationships with media outlets (even when there is no
remuneration involved). Currently, consumers of public opinion research cannot easily
appreciate when media outlets are reporting on independently conducted polls and when
they are actively promoting findings for which they have paid. In many cases, news
outlets suspend their normal critical reporting practices because they are the sponsors of a
particular poll.
The previous work of Mainstreet executives on elections for political contestants raised
suspicions in the Nenshi campaign that the firm might not be operating as a neutral
observer. Even assuming these suspicions were unjustified, the Calgary election raised
issues of transparency that current industry standards do not address.
In Canada the responsibility to address the specific issues raised by Mainstreet’s polling
of the Calgary election falls to the MRIA as the organization representing the marketing
and public opinion research sector in Canada, even though Mainstreet is not a member.
Many of those who we interviewed feel that the MRIA has the mandate to regulate and,
when necessary, mediate disputes and sanction firms when appropriate. We deal with
how the MRIA can respond to this call in the recommendations section.
Social media has redefined the ways in which citizens exchange ideas and interact with
one another. Our review of the tweets and emails that surrounded the Mainstreet polls
demonstrated a level of discourse that is unhealthy and counterproductive for developing
respectful relationships among the polling firms and for those interested in participating
in the municipal election. As a general observation, social media has become the
principal mode for most politically active people to communicate with one another and
this is especially the case for young citizens between 18-35 years. Social media as a
communications tool has eclipsed letter writing, the telephone, emails and face-to-face
conversations in terms of influence on social affairs discourse.
One further element that has redefined the polling industry in Canada is the limited
resources at the disposal of media companies (and especially the print media) to carry out
systematic surveys during elections. Large scale, longitudinal surveys are rarely
sponsored today by the media because of prohibitive costs. Instead, typically polling
18

 firms offer their media partners a low-cost alternative in exchange for publicity or air
time. By necessity, these arrangements encourage the polling firms to develop
methodologies that are inexpensive (i.e., fewer call backs, ad hoc sampling frames, novel
data collection methods) and carry with them a higher risk of error along many
dimensions. The thinness of these relationships also means that journalists tend to be less
experienced in using survey data these days than in previous years, creating an increased
risk that those reporting on the survey findings will fail to recognize problems with the
research.

Methodology and Timetable
Within the framework defined by the three issues for review, the panel members with the
help of MRIA staff, assembled and reviewed the media coverage of the Calgary election
as well as the social media commentary associated with the news reports and those
regarding the political polling. Particular attention was paid to the tweets that were shared
among commentators such as academics, professional pollsters and members of the
political adversaries.
As well, all publicly available materials and results for the three Mainstreet Research
polls, the LRT on the Green poll conducted by AskingCanadians (a data collection firm
which is owned and operated by Delvinia), and the Canadian Municipal Election Study
(CMES) carried out by Forum Research were assembled and shared among the review
panel. As well, many follow-up emails were directed to the principals of the three firms
to clarify methodology and other questions that surfaced during our work.
The Panel also benefitted considerably from the two reports commissioned by Mainstreet
Research in their effort to explain why their findings were so out of line with the eventual
election results.
During the second phase of the panel’s review, interviews were conducted with key
representatives of the firms whose polling results were publicly released during the
Calgary election, as well as with individuals and organizations that were involved in the
election and its reporting. Specifically, the review panel met in Toronto on April 5-6 with
the polling firms that are Ontario based and then in Calgary on April 16-17 with industry
representatives as well as Mayor Nenshi and members of his team. A list of those we met
with is contained in Appendix B.
The interviews with the key participants were particularly helpful in our understanding of
the issues and, as a result, we are very grateful to those who agreed to meet with us and
for their willingness to engage in very frank and informative discussions. Each of the
interviews was conducted in a neutral meeting room and notes of the conversations were

19

 taken by all three review panel members7. Unfortunately, the principals at Postmedia8 and
the Bill Smith election team declined to meet with us. However, we were able to secure
interviews with a number of informed individuals who provided us with insights into the
activities of Postmedia and the Smith campaign9.

7

Quito Maggi and Joseph Angolano (Mainstreet Research) recorded the session in the
presence of their legal counsel.
8
Postmedia cited ongoing litigation issues in declining our invitation.

20

 Section 2
The Calgary Election Polls
This section provides an overview of the polls that were released to the public during the
Calgary election. It is arranged according to the order in which each of the research firms
and their sponsoring organizations first released their polling results: Mainstreet Research
for Postmedia, Pantheon Research for Common Sense Calgary, AskingCanadians for the
LRT on the Green Foundation, and Forum Research for the Canadian Municipal Election
Study. Table 1 provides a summary of the six polls, with each in the chronological order
in which they were released to the public, including their methodologies, release dates,
and the margin for the leading candidate. (A summary of the findings for all the polls and
the final electoral results are provided in Appendix A.)

Release Date

Research Firm

Sponsor

Sample
Method

Sept 30th

Mainstreet
Research

Postmedia

N=1,000 IVR using Fielded
Mainstreet’s
Sept 28
sample

October 6th

Pantheon
Research

Common
Sense Calgary

October 7th

Mainstreet
Research

Postmedia

N=4,887
IVR
Sample
source
not stated
N=1,500 IVR using
Mainstreet’s
sample

October 11th

AskingCanadia
ns

October 13th

Mainstreet
Research

October 13

Forum
Research

LRT on the
Green
Foundation
Postmedia

N=1,004
panel

and Fielding
Dates

Fielded
Sept 29Oct 210
Fielded
Oct 3-4

Lead
Election
Candidat Result
e
Bill Smith
+8.8%

Bill Smith
+12.96%
Bill Smith
+16.6%

Naheed
Nenshi
+7.62%

Online Fielded
Oct 7-10

Naheed
Nenshi
+15%
N=1,500 IVR using Fielded
Bill Smith
Mainstreet’s
Oct 10- +11.7%
sample
11

Canadian
N=843 RDD IVR Fielded
Naheed
Municipal
recruitment with Sept 28- Nenshi
Election Study online survey
Oct 12
+16.8%
(CMES)

Table 1: Summary Table Calgary Election Publicly Released Polls
10

It is unclear whether the dates were the four dates stated here or limited to September
30 and October 1. See the discussion regarding this poll in this section.
21

 Mainstreet Research - Postmedia
During the length of the campaign, Mainstreet Research conducted three polls that were
publicly released for Postmedia, the corporate owner of the Calgary Herald and the
Calgary Sun. These were published on September 30th,11 October 7th, and October 13th,
with all three having survey results showing Bill Smith in a strong lead over the
incumbent, Neheed Nenshi. Mainstreet also conducted a fourth poll which was not
released to the public.12 Due to the review panel’s focus being on publicly released polls,
we do not discuss this fourth poll in this report.
Poll #1: Released September 30, 2017
Poll #1 involved 1,000 respondents who were interviewed using an IVR (Interactive
Voice Response) system, with all the interviews conducted on the single day of
September 28. In this poll, Bill Smith was leading Naheed Nenshi by 8.8% among
decided voters.
Table 2: Mainstreet Research Poll #1
Fielding
Dates
Fielded
Sept 28

Method
N=1,000 IVR

Naheed
Nenshi
32.9%

Bill
Smith
41.7%

Andre
Chabot
6.6%

Others

DNK

4.4%

14.4%

Four media releases were issued based on this first poll. These were issued for public
release with a media embargo date of September 30th, October 3rd, October 4th, and
October 6th, with the first release focusing on voter preferences and the other three
providing results on attitudes about municipal issues.
After the results from this first poll were released to the public, questions were raised by
third party analysts. The two most significant were as follows:
• Contrary to expectations, the poll showed women were much more supportive
of Bill Smith (48%) compared to men (35%).
• Bill Smith was only slightly behind Nenshi among young adults, with 41% of
those under 35 supporting Smith compared to 46% supporting Nenshi.
The polling firm disclosed that the data were weighted “by age and gender totals” based
on the Canada 2016 Census with the results reported to have a margin of error (MOE) of

11

Three media releases were produced based on results from this first poll, with the
second release (October 3rd) featuring results regarding issues.
12
Angolano Report, p. 8.
22

 +/- 3.1.13 In examining the unweighted versus weighted frequencies for gender and the
four age groups in Table 3, using figures provided by Mainstreet Research,14 there are
only minor differences between the unweighted numbers and weighted numbers. This
indicates that the statistical weight should not have been a significant source of error.
Table 3: Poll #1 Unweighted and Weighted - Age
Unweighted
Weighted

Male
483
495

Female
517
505

18--34
320
326

35-49
290
290

50-64
240
241

65+
150
143

Poll #2: Released October 7, 2017
Mainstreet Research’s second poll showed that Smith’s lead over Nenshi among decided
voters had increased from 8.8% to 16.6%. Table 4 shows the results for this poll.
Table 4: Mainstreet Research Poll #2
Fielding
Dates
Oct 3-4

Method
N=1,500 IVR

Naheed
Nenshi
31.0%

Bill
Smith
47.6%

Andre
Chabot
5.6%

Others

DNK

3.1%

12.8%

For this second poll, Mainstreet increased the number of completed interviews from
1,000 to 1,500, with interview quotas set for all 14 wards in the city. The survey was
fielded during October 10 and 11.15 Results were weighted by age, gender, and ward. In
order to ensure it was adequately surveying the fourteen wards, and because it was felt
that many respondents would be unsure of the ward in which they reside, Mainstreet
selected its potential respondents by matching the sample records in its directory to the
postal codes contained for each individual in its sample database. However, not all of
their sample records contained this information and the firm reported later that this
caused non-response bias in its sample design: that is, only respondents in the sample
frame who could be matched to a ward, i.e. had a postal code, were selected for an
interview.
What surprised many observers of this second poll was the extent to which Bill Smith
was leading Naheed Nenshi among young adults, a demographic many thought would be
pro-Nenshi. Mainstreet showed Smith with a commanding 36% lead over Nenshi among
those under the age of 35, and an 18.9% lead among women - two demographic groups
that were expected to lean towards Nenshi.
13

Mainstreet/Postmedia, “Calgary Sept 2017” media release, September 30, 2017. The
press release rounds the results, the more precise figures are derived from Angolano
Report.
14
Angolano, p. 6.
15
Mainstreet/Postmedia, “Calgary 2017,” media release, October 13, 2017.
23

 Table 5 shows the results for this second poll broken down specifically by gender and
age.
Table 5: Poll #2 – Results by Gender and Age
Male
45.3%
31.1%

Bill Smith
Naheed Nenshi

Female 18—34
49.8%
60.6%
30.9%
24.6%

35-49
34.6%
37.8%

50-64
45.4%
34.7%

65+
49.4%
24.8%

The aforementioned problem with the sample frame in this second Mainstreet poll
exacerbated the issue for its young adult subsample. By removing those who did not have
identifiable postal codes, close to 30 percent of potential respondents were excluded from
the sample frame, with higher rates of those with cell phones being in this group. The
number of cell phones in the sample frame went from what they claim as a “typical” 30
percent incidence to “20% cell phones or less”16 and because younger people are more
reliant on cell phones, this meant that the young people who were within the sample
frame might not have been representative of this population.
Another problem was that by stratifying the sample by ward, the firm introduced
additional error due to there being uneven turnouts from ward to ward; that is, those in
wards that produce low turnouts would be overrepresented in the polling results.17
In Table 6 are the unweighted versus weighted frequencies for the four age groups, using
figures provided by Mainstreet.18 As with the first poll, there were only minor differences
between the reported unweighted numbers and weighted numbers. Therefore, errors
relating to the young adult sample were not due to low incidence rates, but because the
young adults who responded to the survey, presumably mostly through a landline
telephone, were not representative of their demographic.
Table 6: Poll #2 Unweighted and Weighted - Age
Unweighted
Weighted

18—34
471
463

35-49
437
438

50-64
358
374

65+
234
225

Poll #3: Released October 13, 2017
Mainstreet’s third poll, showing Bill Smith still in the lead, produced the following
results:

16

Angolano, p. 7.
Angolano, p. 7.
18
Angolano, p. 7.
17

24

 Table 7: Mainstreet Research Poll #3
Fielding
Dates
Fielded
Oct 10-11

Method
N=1,500 IVR

Naheed
Nenshi
35.5%

Bill
Smith
47.2%

Andre
Chabot
5.5%

Others

DNK

1.9%

9.9%

This third poll was fielded on October 10 and 11, with a media release on October 13.19
The poll included 1,500 respondents, as with the second poll, with a stratified sample
created with interviews conducted for all 14 wards in the city. Results were weighted by
age, gender, and ward. It is unclear how the methodology for this poll differed from the
2nd poll, a poll which Mainstreet Research later recognized as having methodological
error.20
According to the firm’s post-election analysis regarding the polling and
representativeness of the sample, Joseph Angolano of Mainstreet wrote that following the
second poll, the outside “criticism was so intense that some doubts emerged amongst the
team. Specifically, some team members felt that the sample of respondents in the 18 to 34
age group was not representative of likely voters in that age cohort…”21 Angolano writes
further that in their analysis of what he refers to as the firm’s “directory sample”, only 4.7
were adults under the age of 35, compared to 23.4% that were of this age group when
sampled using a random digit dialing (RDD) methodology.22
The Mainstreet Polls: Explaining the Inaccuracies
In interviews, the review panel heard what could be described as two sets of explanations
for why the Mainstreet polls produced results that varied both from other polls as well as
the final electoral outcome.
-

Explanation Set #1:

From Mayor Nenshi and several of those associated closely with his campaign, the panel
heard allegations that the polling done by Mainstreet was based on samples drawn from
databases created specifically for voter identification activities undertaken by Mainstreet.
It also heard allegations from the Nenshi campaign that support for Bill Smith may have
been bolstered due to respondents of the Mainstreet polls having received calls
beforehand fielding “push-polls” disparaging the Nenshi campaign. There was a related
allegation that the Smith campaign may have had access to the same “directory” that
Mainstreet was using for its surveys. And, there were additional claims made during the
19

Mainstreet Research/Post Media, “Calgary 2017,” media release, October 13, 2017.
Angolano, pp. 6-7.
21
Angolano, p. 8.
22
Angolano, p. 11.
20

25

 campaign that Mainstreet Research might have been in “cahoots” with Bill Smith or his
supporters, or with Postmedia, to promote Smith’s candidacy.
Mainstreet has specifically rebutted these claims in public, in the interview with the
review panel, and in subsequent correspondence. Maggi and Angolano told us that its
directory was not acquired in connection with any voter ID activities, that it did not
participate in any such activities in the Calgary election and that its “directory” was not
available to any other entity in connection with the Calgary election. It adamantly denied
that it deliberately skewed its results.
Given the scale of reputational damage that would result from publishing polls that
turned out to be gravely misleading, it seems inherently unlikely to the panel that
Mainstreet or any other public opinion research firm would be willing to take the risk
implied by the accusations that it would deliberately falsify its results. In the aftermath of
the Calgary election, Mainstreet commissioned Justin Ling, a well-respected independent
journalist, to write an examination of the firm’s communications during the campaign.
Ling wrote: “there really is no basis for the claim that Mainstreet was in the bag for either
the Smith campaign; the Calgary Flames, whose public dispute with the mayor grabbed
headlines; or anyone else. Mainstreet had no political dog in the race.”23
The panel did not receive any substantive evidence that would lead us to demur from
Ling’s conclusion. On the balance of probabilities, it seems much more likely that
Mainstreet’s failure was a methodological one.
-

Explanation Set #2:

In addition to the independent Ling report, in the aftermath of the Calgary election,
Mainstreet commissioned an internal review of its methodology. This review was
conducted by the firm’s vice-president, Joseph Angolano, who it said was not involved in
the Calgary polling.
In his report titled “Mainstreet Research and the 2017 Calgary Election: How it went
wrong and how it can improve”, Angolano identified a number of factors he said came
into play in the failure of the Mainstreet polls – factors he summarized as the “Calgary
Effect”. In essence, he argued that there were a number of features peculiar to Calgary in
2017 that exposed weaknesses in otherwise sound Mainstreet methodology.
These included an unexpectedly “higher voter turnout”, especially among young voters,
and “a significant number of Nenshi voters” who did not respond to attempts to contact
them. Furthermore, the Mainstreet polls “likely did not contain enough cell phones to
build a reliable [sample] frame” especially in light of Calgary being one of the “youngest
cities in Canada.”24 Angolano wrote that “Mainstreet uses a mostly landline sample, and
23

Justin Ling, “The Only Poll That Matters: How Things Went Wrong in Calgary, and
How to Fix It,” report prepared for Mainstreet, n.d. 2017.
24
Angolano, p. 17.
26

 while it has used quota dialling to get a sufficient number of respondents aged 18-34,
most of these respondents were reached via landline.”25 This problem, Angolano said,
was exacerbated by the fact that Calgary is a young city, with the 2016 Census showing
Calgary’s median age being 36.9 compared to the national median of 40.6.26 This coupled
with a higher than expected turnout in 2017 in which 387,583 cast their ballot27 compared
to 262,577 in the previous 2013 election28 and 354,090 in 2010,29 meant that the nonrepresentativeness of young Calgary in the samples translated into increased threats to
validity.
In addition, in an attempt to address criticisms of its initial published poll, Mainstreet
decided to stratify its sample in the second poll by ward. For reasons explained in the
Angolano report as well as in our discussion earlier in this section, this had the effect of
stripping out most of the cell phones--which normally constitute about 30% of the
Mainstreet sample--thus exacerbating the existing methodological problems.
The panel members applaud Mainstreet Research for its diligence in trying to identify
sources of methodological error. However, we feel that Angolano’s report may have
understated one of the most important elements in its failure to capture what was
happening in Calgary, which is addressed in the following sub-section.
-

The Mainstreet “Directory”

In theory, opinion polls work according to the laws of statistics when a random selection
of respondents from a larger population is surveyed, accurately representing that
population within a statistically calculable margin of error. In the early days of scientific
polling, researchers knocked on randomly selected doors, and later, in an era where
almost everyone had a telephone landline, live operators phoned randomly selected
numbers. There were always issues of non-response bias, but the effort to capture a
random slice of the population was straightforward enough.

25

Angolano, p. 22.
Statistics Canada, “Census Profile, 2016 Census,” http://www12.statcan.gc.ca/censusrecensement/2016/dppd/prof/details/page.cfm?Lang=E&Geo1=ER&Code1=4830&Geo2=PR&Code2=48&Da
ta=Count&SearchText=calgary&SearchType=Begins&SearchPR=01&B1=All&TABID=
1
27
City of Calgary, “Official Results,”
http://www.calgary.ca/election/Pages/results/Official-Results.aspx.
28
City of Calgary, “Official Results,”
http://www.calgary.ca/election/Scripts/election/historical/2013-General-ElectionResults.htm
29
City of Calgary, “Official Results,”
http://www.calgary.ca/election/Scripts/election/historical/2010-General-ElectionResults.pdf.
26

27

 However, in modern polling, it is less straightforward. Respondents to online polls are
usually selected from a “panel” previously assembled by self-selection, phone
recruitment, or from lists acquired for other purposes, such as retail loyalty programs. In
contrast, phone polls, including IVR, are usually reliant heavily on Random Digit
Dialling (RDD) which has fewer potential complications. However, Mainstreet Research
did not employ RDD in Calgary, but rather made its IVR calls to numbers selected from
its own assembled “directory”.
In the Angolano report, Mainstreet’s directory is described somewhat mysteriously as
being “assembled from various sources.”30 When asked about this by the panel during its
interview with the firm, Mainstreet’s executives told us that the firm gets its phone
numbers from Bell Canada, Telus and other major providers and matches the numbers to
Elections Canada data. In subsequent correspondence, Maggi told us that, “we purchase
commercially available directory data through a number of brokers. We do not buy
directories for a single city or province, we buy the national directories for both Canada
and the United States on a subscription through our brokers annually, with a new sample
provided every quarter.” He said that the directory was “primarily” assembled from these
commercial sources. In a final exchange, Maggi told the review panel that it relies on a
broker who assembles data from “Randa, SSI, Exact Data, Dun and Bradstreet (for
commercial lists), Canada Post, etc.”
In the end, the review panel did not have enough insight into the assembly of the
Mainstreet directory to know to what degree it may have been a source of error, though
there is reason to think it was not a merely theoretical concern. According to Mainstreet,
in a post-election survey it undertook, and in which it added an element of RDD that it
had not employed in the Calgary election polls, there was at least one dramatic difference
in the respondents they surveyed. While just 4.7% of the directory respondents identified
themselves as being age 18-34, among the RDD sample it was 23.4%. Put another way,
this means that small, perhaps unrepresentative samples end up being massively
canvassed (or potentially weighted up) to match Census data. As discussed earlier in this
section, if Mainstreet’s reported numbers are correct, the weighting that was done for the
young adult age segment was not large (see Table 3 and Table 6), signifying that it was
the sample directory that was problematic
The review panel will address the issue of the samples in the Mainstreet polls and one
other Calgary election poll in its conclusions and recommendations.
MRIA’s Standards and the Mainstreet Research Polls
At the time of the Calgary election, Mainstreet Research was no longer a member of the
Marketing Research and Intelligence Association due to an unrelated dispute. However, it
is worthwhile reviewing the MRIA’s polling standards to see how they might have
applied to Mainstreet’s polling during the campaign.
30

Angolano, p. 5.
28

 The panel does note that in the MRIA’s “Code of Conduct for Market and Social
Research” it says that “[l]imitations and weaknesses in the poll design, its execution, and
the results must be noted in all reports and analyses.”31 We applaud Mainstreet Research
for its post-election analysis, but the panel is concerned that the firm was insufficiently
transparent regarding its sample frame and other methodological concerns at the time that
it released its polling results.
The MRIA’s “mandatory disclosure requirements for all public opinion surveys,”
provides a list of requirements for all public opinion surveys. Below we show the extent
to which the Mainstreet’s polls were compliant with these requirements:32
1. The names of the organization [sic] which conducted the poll and its sponsor. Yes,
both the firm and Postmedia were identified.
2. The dates of field work. Yes, each of the media releases provided this
information.
3. The sampling method used. Yes, the firm provided a basic description of the
methodology, that it was IVR and drawn from its directory. However, the panel
remains unclear on the precise nature of this directory and its composition.
4. The universe [is] effectively represented. Yes, this was provided, albeit there was
probably non-response bias due to lower than desired cell phone coverage.
5. The actual sample size (number of completed interviews). Yes, this was provided.
6. The initial sample size. No, this was not provided, only the number of completed
interviews. We recognize that this is not a common practice in the Canadian
industry.
7. The number of individuals who were asked to participate. No, this was not
provided, only the number of completed interviews. And, again, this is not a
common practice in the Canadian industry despite the MRIA rule.
8. The number (and respective percentage) of people who refused to participate and
who were ineligible to participate. No, this was not provided, only the number of
completed interviews.
9. The method by which the poll was conducted. Yes, IVR was clearly stated as the
methodology used.
10. Details of any strategies used to help gain cooperation. No, but not applicable.
11. The results presented in data tables, including base numbers and percentages. Yes,
these were provided.
12. Whether weighting/calibration/normalization was used to adjust the results and
the impact of the procedure on the data as measured through a weighting
efficiency calculation and disclosure of the range and variance of the weights.
This was not done in such detail, but we note that many polls, including those
released by MRIA members, commonly fall short of meeting these standards. A
31

MRIA, “MRIA Code of Conduct for Market and Social Research: Appendix L, Polling
Standards for the Canadian Marketplace,” Updated June 2017, p 5.
32
MRIA, “MRIA Code of Conduct for Market and Social Research: Appendix L, Polling
Standards for the Canadian Marketplace,” pp 6-7.

29

 statement was given for each poll that results were weighted by age, sex, and for
two of the polls, by ward.
13. The percentage of respondents who give “don’t know” answers (and in the case of
voter-intention studies, of those who say they will not vote). The firm identified
the DNKs in each of the polls, but not the “will not vote” answers.
14. The method used to recalculate data to take into account in the survey the results
of participants who expressed no opinion, were undecided or failed to respond to
any or all the survey questions. Because the firm included the DNKs in the
reported results, this is not applicable.
15. The Margin of Error if applicable. Yes, this was provided.
16. The screener questions asked during the actual survey. No, this information was
not provided in the media releases, but this is not normal practice in the industry
including among those practitioners who are MRIA members.
17. The questions asked, and the results for each question. Yes, these were provided
for the specific results for specific questions.
18. If part of an omnibus, that this is clearly indicated. The firm was not using an
omnibus, thus this was not applicable.
19. If an internet panel was used it must be indicated if the recruitment method to
build the panel is based on multiple sources or only one source. Not applicable.
20. The number of call-backs for telephone interviews, IVR or live interviewer/CATI.
This information was not provided. However, due to the narrow fielding windows
for each of the polls, there would have been little opportunity to apply effective
call-back procedures.
21. Any relevant stimuli, such as visual or sensory exhibits or show cards that were
used. Not applicable.
The MRIA states that “all researchers must” immediately notify their clients, or media,
who have published the results, of any error(s) made by the researcher.33 The firm was in
discussions with its media sponsors regarding the concerns raised by the results from the
first and second polls, and the firm altered the sampling methodologies in an attempt to
increase reliability and confidence in the results. It was only afterwards that the firm was
willing to state the possible sources for error publicly.
The MRIA also recommends that the firm provide information on whether or not it used
a “likely voter model” for their release of results. In the case of Mainstreet Research, it
does not use these models for its polling results.
Regarding a firm’s responsibilities when reporting on results, the MRIA states that “both
the researcher and the client have a responsibility to ensure the published results are not
misleading” and that the “public must be able to clearly distinguish between the survey
findings as such and any editorial or other comments based on these findings.”34 With
regard to the second poll released by Mainstreet Research, the firm overstated its findings
when it said: “These numbers point to the near certain election of Bill Smith on October
33
34

MRIA, p. 7.
MRIA, p. 8.
30

 16th.” Mainstreet Research did qualify this statement by discussing the possible effects of
campaigning and the get-out-the-vote activities in a campaign.35 Yet throughout the
campaign, Mainstreet repeatedly conflated its findings with commentary as will be
discussed in Section 3 on media and communications.

Pantheon Research – Common Sense Calgary
The panel members were unable to ascertain full details of the Pantheon Research poll,
which was put forward as an “independently commissioned” poll released by Common
Sense Calgary.36 There was insufficient information in Common Sense Calgary’s media
release. The panel’s attempts to secure interviews with Common Sense Calgary (which
now appears to have disbanded) and Pantheon Research were unsuccessful. Pantheon is
not an MRIA member. The media release for the poll states the survey was done via IVR
“between” September 29 and October 2. Due to the use of the word “between”, it remains
unclear if the fielding dates were only the two dates of September 30th and October 1st, or
the four dates of September 29th, 30th, October 1st, and October 2nd,.
Common Sense Calgary’s Twitter account, used to announce the poll, has been inactive
since December 2017. Its website has not been maintained. Table 8 provides the results
from the Common Sense Calgary poll showing Bill Smith with a 13-point lead over
Naheed Nenshi. In addition to showing results for the mayoral race, it was the only
publicly released survey to provide ward-by-ward results for the councillors’ races.
In its media release, the wording of the question for the mayoral candidate preference
was “of the following mayoral candidates, who would you be most likely to vote for?”
The order of questions was not provided, nor was there any information regarding the
source of Pantheon Research’s sample nor whether or not any specific weights were
applied to the data.
The total number of completed responses was 4,887. No information was provided on the
response rate, nor the number or percent of those who stated they were unsure of how
they would be voting or did not plan to vote.
Table 8: Common Sense Calgary - Pantheon Research
Fielding
Method
Dates
Sept 29- N=4,887 IVR
Oct 2 [?]

Naheed
Nenshi
36.76%

Bill
Smith
49.72%

Andre
Chabot
5.91%

Others

DNK

7.61%

Not
reported

35

Mainstreet/Postmedia, “Calgary Election 2017,” Media Release October 7, 2017,
The poll results were released via two avenues. Via Twitter (CommonSenseYYC) and
an undated formal media release: “Polling Indicates Major Shakeup Likely at City Hall
This Election.”
36

31

 It is clear that both Common Sense Calgary and its polling partner, Pantheon Research,
fell short of proper industry standards when releasing this poll to the public. Here we
review the MRIAs mandatory disclosure requirements when public opinion surveys are
provided and, as we did in the previous section, we provide in italics a statement on
whether or not the Common Sense Calgary poll conducted by Pantheon Research was in
accordance to MRIA standards.
1. The names of the organization [sic] which conducted the poll and its sponsor. Yes,
Pantheon identified Common Sense Calgary as its sponsor.
2. The dates of field work. Yes, the media release provided this information.
3. The sampling method used. No, the organization stated that it was IVR based, but
there is no information regarding the sample source.
4. The universe [is] effectively represented. The panel was unable to obtain a
sufficient amount of information, other than that the poll that was based with IVR
covering all the wards.
5. The actual sample size (number of completed interviews). Yes, this was provided.
6. The initial sample size. No, this was not provided, only the number of completed
interviews.
7. The number of individuals who were asked to participate. No, this was not
provided, only the number of completed interviews.
8. The number (and respective percentage) or people who refused to participate and
who were ineligible to participate. No, this was not provided, only the number of
completed interviews.
9. The method by which the poll was conducted. Yes, IVR was clearly stated as the
methodology used.
10. Details of any strategies used to help gain cooperation. No, insufficient
information was provided.
11. The results presented in data tables, including base numbers and percentages. No,
only the results were provided for each candidate. It is unclear what the base
numbers were, especially with no information about how data were weighted.
Furthermore, no demographic data information was provided.
12. Whether weighting/calibration/normalization was used to adjust the results and
the impact of the procedure on the data as measured through a weighting
efficiency calculation and disclosure of the range and variance of the weights. No
information was given regarding any weighting performed on the data.
13. The percentage of respondents who give “don’t know” answers (and in the case of
voter-intention studies, of those who say they will not vote). The percent of
“undecideds” was identified in the results; however, the “will not vote” answers
were not provided.
14. The method used to recalculate data to take into account in the survey the results
of participants who expressed no opinion, were undecided or failed to respond to
any or all the survey questions. This was not provided.
15. The Margin of Error if applicable. Yes, this was provided.
16. The screener questions asked during the actual survey. No, this information was
not provided in the media release.

32

 17. The questions asked, and the results for each question. We were unable to
determine all of the questions that were asked and their order. However, results
were provided for specific results from specific questions.
18. If part of an omnibus, that this is clearly indicated. We were unable to ascertain
any additional information regarding the design of the survey and whether or not
it was part of an omnibus.
19. If an internet panel was used it must be indicated if the recruitment method to
build the panel is based on multiple sources or only one source. Not applicable.
20. The number of call-backs for telephone interviews, IVR or live interviewer/CATI.
This information was not provided.
21. Any relevant stimuli, such as visual or sensory exhibits or show cards that were
used. Not applicable.

AskingCanadians - LRT on the Green Foundation
On October 11, 2017, an advocacy group titled LRT on the Green Foundation released a
poll conducted by AskingCanadians, a division of the Toronto-based Delvinia, an MRIA
member. LRT on the Green Foundation describes itself as “a not-for-profit organization
with the mission to unite Calgarians in the desire to have LRT [light rapid transit]
operating on the Green Line in both the North Central and South East corridors by the
year 2026. Our members represent various business, developer and community
stakeholders who have an interest in seeing LRT on the Green Line sooner, rather than
later.”37
The LRT on the Green poll was the first publicly released survey to contradict the
findings of Mainstreet Research’s first two polls in the campaign as well as the
Pantheon’s results, with the LRT on the Green’s results showing Mayor Nenshi with a
15-point lead:
Table 9: LRT on the Green - AskingCanadians
Fielding
Dates
Oct 7-10

Method
N=1,004
Online panel

Naheed
Nenshi
41%

Bill
Smith
26%

Andre
Chabot
3%

Others

DNK

2%

28%

The LRT on the Green survey was fielded online and occurred “between October 7 and
10” with the survey sample drawn from an online sample database of Canadian
consumers which is managed by Delvinia, the parent company of AskingCanadians. The
LRT on the Green designed the questionnaire and conducted their own analysis upon
receiving the data results from AskingCanadians.

37

LRTOTG, Media Release, October 11, 2017, pp. 3-4.

33

 The LRT on the Green’s media release provides a description of the poll’s sample source:
“AskingCanadians, a Delvinia company, is an online data collection firm with access to a
research community of more than one million Canadians who have opted-in to participate
in online surveys that significantly influence today’s leading brands.”38 Furthermore,
“This online survey utilizes a representative but non-random sample, therefore margin of
error is not applicable. However, a probability sample of this size would yield a margin
of error of +/- 3.1 percentage points at a 95% confidence level.”39
As per the LRT on the Green’s media release, “Respondent quotas [were] established to
ensure data are population representative… Data presented are weighted to reflect
population of adult voters 18 years and older. Data obtained are rebalanced by age and
gender as per the 2017 City of Calgary civic census.”40
The review panel, in its interview with representatives of AskingCanadians and Delvinia
learned further details about the source of Delvinia’s online sample. The firm maintains a
general population database consisting of Canadian consumers who are members of
loyalty card programs including Aeroplan, Hudson’s Bay Rewards, Petro-Points, and
Walmart. When a sample is drawn for a particular study, the sample is weighted to the
characteristics of the research population being studied. 41
Our review panel expresses strong concern about the wording and the structure of the
questionnaire. The panel determined that questions #1, #2, #3 were biased in their
wording.42 That is, they appeared designed to evoke a particular response. The wording
of the questions is as follows:
1. “City Council has approved the full 46 km route and 28 stations for the Green
Line LRT from north to southeast Calgary. This ensures that the Green Line can
be built in stages as funding becomes available. Do you support this plan that has
been approved by Council?”
2. “After extensive public consultation, the 20 km Phase One of the Green Line LRT
has been fully funded by the municipal, provincial and federal governments and
the line is under construction. Do you support proceeding with Phase One of the
Green Line LRT project?”
3. “In the Calgary municipal election, some candidates have proposed delaying the
Green Line LRT project and developing an alternative plan. Every year the
project is delayed, the City estimates construction costs will increase by $100
million per year. Do you agree with delaying the project?”
38

LRTOTG, Media Release, October 11, 2017, pp. 3.
LRTOTG, Media Release, October 11, 2017, pp. 3.
40
LRTOTG, Media Release, October 11, 2017, pp. 3. Regarding this census, see
http://www.calgary.ca/CA/city-clerks/Pages/Election-and-Information-Services/CivicCensus/2017-Results.aspx
41
Panel interview with Raj Manocha, Executive Vice Preident, Delvinia, April 17, 2018.
42
For a discussion of these types of questions see Alvin Burns and Ronald Bush,
Marketing Research: Six Edition, Boston: Prentice-Hall, 2010, pp. 309-311.
39

34

 Our panel also determined that with questions #1, #2 and #3 preceding question #4, that a
positive response was to be expected for the following:
4. Will a candidate’s position on the Green Line LRT project impact your vote in
this election?”
Furthermore, because Mayor Nenshi was on record as being a strong supporter of the
LRT on the Green project,43 the review panel believes that the subsequent question would
quite likely be influenced by the previous questions, and that Nenshi would receive
strong support among the survey’s respondents:
5. “Which Mayoral candidate do you intend to vote for?”
In reviewing the data tables that were produced by the survey results, 70% of those in the
survey said they supported the LRT on the Green Line.
The review panel was concerned that there may not have been any trained individual
responsible for the design and interpretation of the AskingCanadians-LRT on the Green
poll. The LRT on the Green group declined our invitation to meet. In our interview with
Delvinia and AskingCanadians, they said that as a data collection service only, they
relied on their client, LRT on the Green, to design the questionnaire and to do their own
analysis of the finalized data tables. AskingCanadians received the survey, tested it for
any technical difficulties, and then fielded it. AskingCanadians stated that they did not
provide any comments to its client regarding the ordering of the questions in the survey.
Furthermore, we were informed during our interview with Asking Canadians and
Delvinia representatives that LRT on the Green did not seek input from AskingCanadians
regarding the professionalism of the questionnaire.
Our panel expresses its concern that when an organization that does not employ any
individuals who are properly trained in survey methods seeks to use a sample and field
provider to conduct a survey on their behalf, problems can easily occur. (See Section 4,
Conclusions and Recommendations).
In addition, this poll raised a number of issues regarding the application of existing
MRIA standards when releasing results to the public. Here we provide in italics whether
or not the LRT on the Green poll was released in accordance to MRIA standards.
1. The names of the organization [sic] which conducted the poll and its sponsor. Yes,
both AskingCanadians and LRT on the Green Foundation were clearly identified.
2. The dates of field work. Yes, the media release provided this information.
However, by stating it was fielded “between October 7 and 10” it is unclear if
this means the fielding only consists of October 8th and 9th, or October 7th,8th,
9th and 10th.
43

See for example, http://www.cbc.ca/news/canada/calgary/lrt-province-funding-announcement-ctraingreen-line-nenshi-mason-infrastructure-1.4192673.

35

 3. The sampling method used. Yes, this is provided.
4. The universe [is] effectively represented. The media release stated that it reflected
the 2017 Calgary civic census. However, respondents were recruited based on
consumer panels that might not have reflected the full electorate.
5. The actual sample size (number of completed interviews). Yes, this was provided.
6. The initial sample size. No, this was not provided, only the number of completed
interviews.
7. The number of individuals who were asked to participate. No, this was not
provided, only the number of completed interviews.
8. The number (and respective percentage) or people who refused to participate and
who were ineligible to participate. No, this was not provided, only the number of
completed interviews.
9. The method by which the poll was conducted. Yes, an online panel was clearly
stated as the methodology used.
10. Details of any strategies used to help gain cooperation. No, insufficient
information was provided.
11. The results presented in data tables, including base numbers and percentages. Yes,
a link was provided in the media release to the data tables, which included base
numbers and percentages.
12. Whether weighting/calibration/normalization was used to adjust the results and
the impact of the procedure on the data as measured through a weighting
efficiency calculation and disclosure of the range and variance of the weights.
Basic information was provided stating that weights were assigned by age and
gender.
13. The percentage of respondents who give “don’t know” answers (and in the case of
voter-intention studies, of those who say they will not vote). The DNKs were
provided.
14. The method used to recalculate data to take into account in the survey the results
of participants who expressed no opinion, were undecided or failed to respond to
any or all the survey questions. This was not provided.
15. The Margin of Error if applicable. A statement was given that this was not a
probability sample, however, a statement was given regarding what the MOE
would be if it had been a probability sample.
16. The screener questions asked during the actual survey. No, this information was
not provided in the media release.
17. The questions asked, and the results for each question. Yes, this was provided.
18. If part of an omnibus, that this is clearly indicated. We were unable to ascertain
any additional information regarding the design of the survey and whether or not
it was part of an omnibus.
19. If an internet panel was used it must be indicated if the recruitment method to
build the panel is based on multiple sources or only one source. Yes, this was
provided.
20. The number of call-backs for telephone interviews, IVR or live interviewer/CATI.
Not applicable.
21. Any relevant stimuli, such as visual or sensory exhibits or show cards that were
used. This information was not provided.

36

 Forum - Canadian Municipal Election Study
The Canadian Municipal Election Study (CMES) is an academic project studying the
electorate in eight cities with surveys conducted during each city’s municipal elections.
The Calgary election was the first in this project, with funds provided by the Social
Sciences and Humanities Research Council of Canada (SSHRC).44 Our panel members
were able to interview the principal investigator, Dr. Michael McGregor a political
scientist based at Ryerson University as well as another member of the project team who
was directly involved in the Calgary survey project, Dr. Jack Lucas, a political scientist
based at the University of Calgary. We met also with Lorne Bozinoff of Forum Research,
an MRIA member that was the contractor conducting the poll on CMES’s behalf.
After undergoing a formal procurement process, Forum Research was commissioned to
conduct the CMES survey. The sampling methodology was based on recruiting through
random digit dialing with IVR. Respondents who agreed to participate in the survey were
then relayed to a live interviewer who would obtain the respondent’s email address, and
the survey itself was conducted online. Because the recruiting procedure was deemed to
be based on probability sampling, CMES was able to report a margin-of-error of +/3.37%, 19 times out of 20. CMES also reported the weighting: “Data have been
statistically weighted by age and gender, to ensure that the sample reflects the actual
population according to the latest Census data.”
Because the CMES survey was not launched with the intention of capturing a “snapshot
in time” but to cover the general views of the electorate over the period of the campaign,
the fielding window was quite long. The publicly released results were based on the
surveys that were collected from September 28 to October 12, with surveys continuing to
be collected after this period.45 The mayoral candidate question was as follows: “Which
mayoral candidate do you think you will vote for?” Table 10 provides the results from
this poll.
Table 10: CMES – Forum Research
Fielding
Dates
Sept 28Oct
12
Released
Oct 13

44
45

Method
N=843,
Online
survey with
RDD
using
IVR
recruitment

Naheed
Nenshi
49.8%

Bill
Smith
33.0%

Andre
Chabot
4.1%

Others

DNK

0.5%

Not
reported

Information regarding this project can be found at: https://www.cmes-eemc.ca/.
Subsequently collected surveys but were not part of any CMES media releases.
37

 The results from this CMES poll were released on October 13, the same date that
Mainstreet Research released its third and final public poll. These two polls provided
very different results, with CMES reporting Nenshi as having a 16.8% lead over Bill
Smith and the Mainstreet Research poll showing Nenshi behind Smith by 11.7%.
There appears to be a misconception about the reason why the CMES results were
released to the public. Our panel members heard from many different sources that the
directors of the CMES had not planned to release their poll, and that they were reserving
the results specifically for scholarly research and publications. In this narrative, they were
compelled to release their results when they witnessed the first two Mainstreet polls with
results that were very different from their own. However, Dr. McGregor and Dr. Lucas
told the panel members they had previously planned to release their results in order to
add to the public discussion and raise the profile of the CMES irrespective of what the
results showed.46
The CMES media release states: “This Forum Polltm and other polls may be found at
Forum’s poll archive as www.forumresearch.com/polls.asp.” However, our panel was
unable to find the release for the CMES poll at this website.
The panel members feel that proper industry standards were used for the sampling and
collection of survey results for the CMES. As we have done in the previous sections, we
review here the MRIAs mandatory disclosure requirements for publicly released results
from public opinion surveys. We provide in italics a statement on whether or not this
CMES poll conducted by Forum Research was released in accordance to each of the
MRIA standards.
1. The names of the organization [sic] which conducted the poll and its sponsor. Yes,
this was clearly stated.
2. The dates of field work. Yes, the media release provided this information.
3. The sampling method used. Yes, the organization stated that it was based on
RDD.
4. The universe [is] effectively represented. The media release states that recruiting
was done via RDD among Calgary voters.
5. The actual sample size (number of completed interviews). Yes, this was provided.
6. The initial sample size. No, this was not provided; only the number of completed
interviews.
7. The number of individuals who were asked to participate. No, this was not
provided; only the number of completed interviews.
8. The number (and respective percentage) or people who refused to participate and
who were ineligible to participate. No, this was not provided; only the number of
completed interviews.
9. The method by which the poll was conducted. Only partly. The media release
states that RDD was used to reach respondents, but not that IVR was used to
46

This was confirmed in a personal email from Dr. McGregor to Christopher Adams,
May 4, 2018.
38

 identify those willing to be surveyed. It did clearly state that the survey was an
online survey for those recruited to participate.
10. Details of any strategies used to help gain cooperation. No, but not applicable.
11. The results presented in data tables, including base numbers and percentages. Yes,
base numbers were provided along with percentages.
12. Whether weighting/calibration/normalization was used to adjust the results and
the impact of the procedure on the data as measured through a weighting
efficiency calculation and disclosure of the range and variance of the weights. It
was clearly stated that weights regarding gender and age were applied, but the
media release does not include a means to assess the unweighted numbers
compared to the weighted numbers.
13. The percentage of respondents who give “don’t know” answers (and in the case of
voter-intention studies, of those who say they will not vote). The DNKs were
identified in the results. No information was provided on those who gave “will not
vote” answers.
14. The method used to recalculate data to take into account in the survey the results
of participants who expressed no opinion, were undecided or failed to respond to
any or all the survey questions. This was not provided.
15. The Margin of Error if applicable. Yes, this was provided.
16. The screener questions asked during the actual survey. No, this information was
not provided in the media release.
17. The questions asked, and the results for each question. Results were provided for
specific results from specific questions.
18. If part of an omnibus, that this is clearly indicated. CMES was not using an
omnibus, so this is not applicable.
19. If an internet panel was used it must be indicated if the recruitment method to
build the panel is based on multiple sources or only one source. Not applicable.
20. The number of call-backs for telephone interviews, IVR or live interviewer/CATI.
This information was not provided.
21. Any relevant stimuli, such as visual or sensory exhibits or show cards that were
used. Not applicable.

39

 Section 3
Communicating the Polling Results
Prior to September of 2017, the conventional journalistic wisdom in Calgary was that
Mayor Nenshi would be reelected in a low-key, and likely low-turnout campaign.
Although the mayor’s tenure as mayor was more controversial than it had been in 2013,
he was not perceived to have a clear opponent. The two most prominent candidates other
than the mayor were André Chabot, an independent-minded conservative member of city
council, and Bill Smith, who had been president of provincial Progressive Conservative
party but who had not held elective office. The first Mainstreet Research campaign poll,
published on September 30, which suggested Smith had a significant lead over Nenshi,
with Chabot far behind, transformed the journalistic narrative of the race. It made a
potentially uninteresting race exciting with the prospect of Nenshi’s defeat, and it
relegated Chabot to the role of an also-ran in terms of media coverage (and perhaps
affected his ability to raise funds and attract volunteers). The Mainstreet polls created and
sustained the journalistic frame within which the election was reported, analyzed and
commented upon in the media.
The three published Mainstreet polls drew most of the attention during the municipal
election campaign because of the surprising news they carried that Nenshi was losing and
losing badly, because of the partnership with Postmedia newspapers which amplified
their impact, and because of Mainstreet’s confident and, as will be shown here,
combative public stance. The polls were widely covered in all the media and became the
subject of heated debate between Mainstreet, critics in the academic and polling
communities, and the Nenshi campaign. The high profile of these polls and the sense of
confidence, even certainty, Mainstreet Research conveyed during the campaign increased
the stakes when the polls turned out to be badly wrong. Meanwhile, Mainstreet’s public
bravado regarding the reliability of its methods and analysis contrasted with its internal
alterations with the methodology of its polls.
Two other polls were published, one sponsored by LRT on the Green and conducted by
AskingCanadians, and the other sponsored by the academic group, the Canadian
Municipal Election Study (CMES), conducted by Forum Research. Both of these polls,
which had their own methodological flaws as “horserace” polls, showed Mayor Nenshi in
the lead and were published to temper the narrative created by Mainstreet. These polls
were covered as news in the media mainly because they contradicted the picture of the
race created by the Mainstreet polls. However, the coverage of these polls was less
extensive in part because there was no formal relationship with a media outlet and in part
because the backers of these polls largely chose not to engage in a public debate with
Mainstreet after publication.

40

 A fourth poll was publicly released by Common Sense Calgary and was conducted by
Pantheon, who we were told was also Bill Smith’s pollster. This poll showed Smith in the
lead. It was barely covered by the media, presumably because of a reticence to publish
internal campaign polls.
Mainstreet Research was the subject of caustic criticism of its methods and its
motivations by the Nenshi campaign. In turn, Mainstreet’s executives were acidic in their
interactions in the media and on social media not only with the Nenshi campaign but also
with those in the academic and polling communities who criticized its methodology.
Some of those we interviewed said they would be reluctant to conduct polling during
future elections or make public comments on other polls as a result of their experience in
the Calgary campaign. This suggests that the heated exchanges created a chill for
academics and interest groups on matters relating to polling and elections.
Media coverage of the Mainstreet polls was very extensive, particularly in the Postmedia
newspapers, the Calgary Herald and Calgary Sun, but also in other outlets. Reporters in
general struggled to be balanced but they were constrained by an uncertain grasp of the
methodological issues and in some cases, we were told, their lack of interest in these
issues. In addition, there was a lack of transparency by Mainstreet about their
methodology and the changes they were making to it during the campaign, which may
have been abetted by executives at Postmedia.
No one we interviewed questioned that the polls and the media coverage of them deeply
affected the campaign. Representatives of the Smith campaign told reporters that the
polls had given their efforts a boost. The Nenshi campaign, we were told, developed
strategies to counter the damage resulting from the misleading polls. However, the
Chabot campaign was effectively made irrelevant, ensuring the election was perceived as
a two-way race between Smith and Nenshi. The Mainstreet polls turned out to be badly
mistaken when Mayor Nenshi won the election comfortably. The net effect of the
Calgary election for the polling industry was an embarrassment that raised serious
questions about the efficacy of polling in the minds of those in the media and presumably
the public. This in turn likely reflected on the entire research industry.

Mainstreet Research - Postmedia
According to Quito Maggi, the CEO of Mainstreet Research, its polls were conducted
under the aegis of a national polling contract with Postmedia that provided for “cost
recovery” - that is for less than commercial rates. Mr. Maggi described the arrangement
(which has since ended) as a “loss leader” for the firm. According to Mainstreet
Research, the contract provided for wide latitude to decide the number of polls, their
methodology, questions and subject matter, but editorial staff sometimes suggested topics
that Mainstreet might then incorporate into questions. The normal procedure was for
Mainstreet to provide the newspapers exclusively with the poll results along with an

41

 analytic deck and commentary a day or so ahead of publication, and Mainstreet
executives would make themselves available for interviews.
In this case, according to our interview with Mainstreet, Postmedia alerted the firm about
its desire to poll in the Calgary municipal election only a few days prior to going into the
field for the first survey - less time than it was accustomed to. Postmedia did not make
itself available to our inquiry, saying that this was due to litigation in a “separate matter”;
Postmedia operates both of Calgary’s daily newspapers, the Calgary Herald and the
Calgary Sun.
Somewhat unusually, in addition to providing Postmedia with the data prior to
publication, Mainstreet Research also sent a release to other media under an embargo not
to publish before Postmedia. This is a departure from general industry practice, which
would normally give access to non-sponsoring media only at the time of publication.
Even more unusually, Mainstreet sent copies of its release for its first Calgary election
poll to the pollsters who were working with the Smith and Nenshi campaigns. These
measures ensured that Mainstreet’s polls would receive wide coverage in all the media
and that the campaigns would be primed to respond quickly when approached by
reporters.
The headline on the Mainstreet press release for its first poll at the end of September was
a bold: “Nenshi Faces Defeat”. After the first poll was released, the Nenshi campaign
attacked Mainstreet’s methodology, its motivations and its track record and sent an email
to supporters detailing the Nenshi campaign’s position. Nenshi’s campaign pollster, Brian
Singh, a member of the MRIA, went further, suggesting that Mainstreet Research was
skewing its numbers to promote Smith’s candidacy. David Valentin, the Mainstreet
Research executive in charge of the file (who is no longer an employee of the firm)
replied by joining in the political debate: “If they have a conspiracy theory to chase,
they’re more than welcome to it,” the Calgary Herald reported him saying. “I would
suggest they go knocking on doors instead.”
In addition, a number of academics and commentators raised questions about the poll,
particularly the anomalous finding that Smith had a large lead among women. As a result
of these concerns, Maggi told us, Postmedia requested a conference call. In the course of
that conversation, Maggi suggested stratifying the sample by wards as a way to address
some of these concerns. (As noted in Mainstreet’s own review of its polls, this decision
likely compounded its methodological problems; see Section 2 of this report.)
Mainstreet’s second poll was criticized by a similar retinue of Nenshi supporters and
more neutral commentators, this time mainly for the large lead it suggested Smith had
among younger voters, in which 61 percent of those under the age of 35 were reported to
be supporting Smith, compared to only 25 percent for Nenshi.47 Later in the campaign,
Mainstreet and Postmedia conducted a second conference call to discuss the progress of
the polling.

47

Mainstreet Research/Post Media, “Calgary 2017,” media release, October 7, 2017.
42

 Quito Maggi, Mainstreet’s CEO, came to the polling industry from a background in
political research. This background created the premise for some of the allegations made
against Mainstreet’s polling by the Nenshi campaign. In an interview with our panel,
Mayor Nenshi and his campaign aides suggested that there may have been defects in the
“directory” Mainstreet Research was using to derive its sample (see Section 2), which
they believed may have been assembled as part of unrelated voter identification work
done for political parties and candidates, by Mainstreet or an affiliated company or entity.
They went beyond this to suggest that Mainstreet or an affiliated company or entity may
have shared this directory with the Smith campaign or its pollster. They said they had
some evidence of voters receiving a “push-poll” call with negative information about the
mayor just prior to being polled by Mainstreet Research.
In subsequent correspondence with the panel, Maggi denied these suggestions, saying
that Mainstreet “purchases commercially available directory data through a number of
brokers” and extracts local (Calgary) data from these. He said that the directory used in
Calgary was drawn “primarily” from these sources. (See Section 2.) He told us
Mainstreet Research was not involved in voter identification in the Calgary race and its
sample was not available to any other entity that might have been. In his original
interview with the review panel, Mr. Maggi said that Mainstreet Research did not have
any private clients in the Calgary election whatever. However, in subsequent
correspondence he revealed that Mainstreet had conducted three previously undisclosed
and unpublished ward-level polls for a “past client”, which he said did not ask questions
related to the mayoral race. The lack of transparency around the full array of a research
firm’s activities may create an atmosphere in which suspicions may ferment, justified or
not.
Publicly, Mainstreet expressed virtually absolute confidence in its polls throughout the
campaign. It said at the time and subsequently that this confidence was rooted in its track
record. However, there were many reasons during the campaign and prior to the final
result for Mainstreet to dial back its self-confidence. To name a few: the unexpected
results in some key demographics in its polls, particularly among women and young
voters; the surprise with which its initial poll was greeted not just by the Nenshi
campaign, but also reportedly the Smith campaign; the radically contrary results of the
two online polls conducted by other firms published during the election; the concern
expressed not only by partisans of the mayor but also by otherwise disinterested
academics and by Mainstreet’s partners at Postmedia; and finally, an online sample
Mainstreet conducted itself (but did not release) that showed something closer to the
actual result which Nenshi won.
Dr. Angolano’s post-election report as well as our interviews confirmed that Mainstreet
Research was scrambling throughout the campaign to determine whether there were
issues in its polling and how they might be addressed – and that there were internal
differences on these matters. Indeed, of the four polls Mainstreet conducted during the
campaign (of which three were published), no two were conducted in precisely the same
way. (The smaller online sample represented a fifth approach.)

43

 In contrast to the scrambling that was happening behind the scenes, Maggi and Valentin
expressed unstinting confidence in their findings and were caustic in answering their
critics. Little effort was made by Mainstreet executives to distinguish between those who
were making accusations of malfeasance (skewing their results to please clients known
and unknown) and those raising questions about the methodology and the results.
Mainstreet Research claimed a scientific precision belied by their internal tweaking of the
methodology. “The math is the math,” Maggi told the Calgary Herald in an article
published October 7. “That’s the only thing I can do is take the sample, weight it
accordingly, test it as many different ways as I can possibly test it, and then put out our
number. This is no different than that we have done before, time and time again.” In fact,
each poll was at least slightly different from the last.
The main thrust of independent critics of the Mainstreet Research polls, such as
University of Calgary political scientist Melanee Thomas and Mount Royal political
scientist Duane Bratt, was that the polls did not comport with what was publicly known
about the campaign and that the polls were showing anomalous results, such as a large
lead for Smith among women in the first poll and among young people in the second.
However, throughout the campaign, Maggi and Valentin expressed unstinting confidence
in their findings and were harsh and often personal in responding their critics.
Mainstreet’s press releases expressed increasing confidence in a Smith victory as the
campaign went on. The release associated with the first published poll said there was
“time for Nenshi to turn things around”. The second became more aggressive, going
beyond a statement of the poll’s findings and critiquing the mayor’s campaign in the
fashion of a pundit, saying Mr. Nenshi had failed to see “the change in the landscape and
pivot successfully”. The headline to the press release on Mainstreet’s final published poll
was “Calgary’s Next Mayor Is…” In an accompanying commentary, Maggi chose to
castigate Nenshi for his skepticism of the Mainstreet polls, and this passage deserves to
be quoted at length:
Those who followed the tracking [Maggi wrote], could see a growing discontent
between the Mayor and the Calgary public. It was not obvious to the Mayor or his
team who chose instead to attack the polls and pollsters that were simply taking
accurate measures of the mood and sentiment on numerous issues.
That decision to ignore numbers, battle with pollsters, media and numerous
community groups instead of addressing the growing discontent, will serve in
another in a long line of cautionary tales for politicians who believe they are
immune to the passing of time and the change of public sentiment.”48
Whatever had been true at the outset of the campaign, Mainstreet executives now had
personal stake in the outcome of the campaign. Perhaps needless to say, Maggi’s
commentary went well beyond the publicly available data. There would indeed ultimately
48

Mainstreet Research/Post Media, “Calgary 2017,” media release, October 13, 2017.
44

 be cautionary tales to be extracted from the election results once they were tabulated,
though not those prophesied by Maggi.
In the last weeks of the campaign, Mainstreet executives adopted an increasingly
aggressive style of personal invective in response to their critics and the polls whose
results conflicted with their own. Valentin told a radio station that he had “seen a lot of
behaviour from political scientists that I would say is quite shocking in this election
campaign, and some of it, quite frankly, is quite appalling.” Maggi’s understandable
concern about attacks on the Mainstreet Research brand and reputation contrasted with
the scathing sarcasm with which he spoke of competitors and critics. Speaking of the
Forum-CMES poll, Maggi said on Twitter that, “Buying panel responses doesn’t make
you a pollster”. On Twitter, he called the AskingCanadians-LRT on the Green poll a
“pseudo-poll”. He tweeted at Duane Bratt, saying: “Your credibility is getting stretched
the more you comment, let’s just see in 5 days.”
Valentin told a radio interviewer ominously that Mainstreet planned on “singling people
out” after the election and was reported as saying it would be “payback time”. Maggi’s
threats of legal action were taken seriously enough that at least two academics sought
legal advice from their institutions. Maggi sent a lawyer’s letter to Mr. Singh several
weeks after the election – a threat of legal action not withdrawn until December. He
informs us that he drew up another letter threatening legal action to another individual but
did not send it when that individual issued a public apology for suggesting the failures of
the Mainstreet polling resulted from intent rather than flaws in methodology.
This take-no-prisoners attitude obviously raised the stakes for Mainstreet Research and
compounded the embarrassment to the industry when the firm’s polling turned out to be
radically wrong. When Maggi was asked in our interview about the discrepancy between
Mainstreet’s increasing internal concern about its polling and its unwavering selfconfidence in public, he referred to what he called the “theatre” of politics. In his
description, politicians and pollsters play roles during the election, and one way of
approaching a pollster’s role is not to “give an inch”. Although he was harshly critical of
Brian Singh attacking Mainstreet’s poll out of political motivations, he told us in regard
to his own performance: “I am used to the theatre and I thought I was playing along.” He
later wrote to us to clarify: “I believe it is part of the business, but would certainly try to
avoid theatre or drama similar to Calgary going forward.”49
Many of those we interviewed in the course of our review were plainly bruised by the
experience of the Calgary election and the tenor of the debate over the polls, particularly
by its personal tone. This included Maggi himself.
To its credit, in the aftermath of the election, Mainstreet announced that the company
would undertake two post mortem reports. One was an internal review of the
methodology Mainstreet used in the election, conducted by Mainstreet vice-president,
Joseph Angolano. The other was an external review of its communications during the
49

Email to panel members from Quito Maggi, April 12, 2018.
45

 campaign, conducted by a freelance journalist, Justin Ling. The willingness to examine,
review and correct error is in the highest tradition of social science and the panel
members found these reports very valuable as it covered similar ground. Maggi has
apologized for his public behavior during the Calgary campaign and has said he is
committed to adopting many of the recommendations of both reviews.

Pantheon Research – Common Sense Calgary
Common Sense Calgary was a conservative-leaning advocacy group. Its website is no
longer up and the panel was unable to identify its leadership. We did not receive a reply
from Pantheon to our request to meet. Others with whom we met told us that Pantheon
was working with Nenshi’s principal opponent, Bill Smith.
The Common Sense Calgary/Pantheon poll showed a large lead for Smith, however it
was not widely reported, likely because of media reluctance to publish polls conducted by
polling companies associated with a campaign.

AskingCanadians - LRT on the Green Foundation
LRT on the Green is a public advocacy group promoting an expansion in public transit
similar to that promoted by Mayor Nenshi. LRT on the Green declined to meet with
members of the review. During our interview, Nenshi indicated that his campaign had
helped the group find funding to mount a poll in the expectation that its results would
contradict those of Mainstreet Research.
As detailed in Section 2 of this report, the AskingCanadians-LRT on the Green
poll was defective as a horserace poll because of its question order, which may have
primed respondents by raising an issue associated with Nenshi’s campaign before posing
the vote choice question. The review panel received no information on this aspect of the
poll’s design. When the poll was released, media inquiries were steered to Prof. Bratt,
who had not been involved in the design of the poll but who had had an opportunity to
review the data in preparation for media calls. Beyond conducting and releasing the poll,
LRT on the Green did not engage in the subsequent debate about its significance or
methodology.
Representatives of AskingCanadians and its parent company, Delvinia, did meet with
members of the review. Their executives emphasized that their role was to provide
sample and data collection, not research, and that questionnaire design, analysis and
communication were not within their firm’s mandate or commercial scope. The poll was
conducted at a standard commercial rate. In accordance with its standard practice as a
data collector rather than as a researcher or pollster, AskingCanadians played no role in
the public discussion of the poll or its methodology.

46

 In sum, the review was unable to ascertain who, if anyone, was responsible for the overall
technical and methodological soundness of the AskingCanadians-LRT on the Green
survey. The possibility that no one had this overall responsibility is a matter of
considerable concern. Of course, if no one individual was responsible for the poll, no one
was in a position to speak authoritatively to the media and public about its results.

Forum - Canadian Municipal Election Study
As described in the methodology section of our report, the Forum-CMES poll was not
originally designed as a horserace poll for release during the election campaign.
Academics directing the CMES designed the poll to inform an academic study of
municipal elections. It was conducted by Forum Research, which had won a bidding
process for the work.
The Canadian Municipal Election Study (CMES) is a group of young political scientists
conducting academic research on municipal elections in a number of cities across
Canada. Although there was one antecedent study of Toronto on which their project was
modelled, Calgary’s was the first election in which they polled as a group. Prof. Michael
McGregor of Ryerson University, the principal researcher, and Prof. Jack Lucas, the
principal Calgary researcher, said that because their research is publicly funded, they
consider it a duty to contribute to the public discourse. Lucas had been writing a blog
using his expertise on municipal elections that had attracted a following among some
journalists and through which they became aware that CMES was polling in Calgary. In
response to inquiries from journalists, Lucas said that the group would share their results
publicly if they could. Mayor Nenshi said that his campaign was also urging the group to
release its findings, expecting that they would contradict Mainstreet Research’s results.
McGregor and Lucas of the CMES told the review panel that because it appeared that
Mainstreet Research would be the only polling company to publish during the campaign,
they were inclined to release their own results, notwithstanding the fact that their survey
methods were likely to produce more of a time-lapse picture of the electorate than the
traditional snapshot. Although they suspected that their results might differ from
Mainstreet’s, they said they made the decision to publish before they knew for certain.
Their view was that if the results confirmed those of Mainstreet Research, it would give
the public greater confidence in the polling, but that if they did not, it might strike an
important cautionary note.
When the CMES results were released to the public, the poll showed a 17-point lead for
Nenshi. Other than introducing their poll when it was released and answering technical
questions about it, the academics behind the poll chose to remain out of the debate that
ensued. Some commentators noted that the sub-samples in the poll, particularly with
regard to women and younger voters, were more in line with standard expectations than
the Mainstreet polls had been. The seriousness and competence of the academics behind

47

 the CMES/Forum poll were attacked directly by Mainstreet Research executives and
these academics told us they were subjected to anonymous invective on social media
such as Reddit. This was not only personally distressing to them, but as young academics
who have not yet obtained tenure, they feared that the controversy might affect their
careers and their livelihoods.
As a result of the acrid atmosphere around the polling in the Calgary election, the
members of the CMES group told the review panel that they have decided they will not
publish the polls they conduct during election campaigns in other cities, and will reserve
the release until they publish their academic findings through scholarly channels, which
will be long after the campaign periods are concluded. Although the members of the
review panel fully understand this decision, the panel members were disturbed to find
that academic inquiry and timely public access to publicly funded research were among
the casualties of the debate over the polling in the Calgary.

Media Coverage
The Mainstreet Research polls dominated media coverage of polls during the campaign
because of their frequency, their surprising results, and the relationship with Postmedia,
which publishes both of the major Calgary dailies. Exclusive relationships between media
outlets and pollsters often result in the distortion of normal news values, for example
giving the stories greater prominence than they might otherwise have (almost invariably
the case), the avoidance of reporting conflicting evidence in other polls, a reluctance to
report concerns about methodology or the history of the polling company, and a
willingness to report the comments of the pollster uncritically. There is some evidence
that this distortion of news values was at play in the Calgary election.
That having been said, we did not see strong evidence that Postmedia, as some have
claimed, was in cahoots with Mainstreet to portray Mayor Nenshi’s challenger, Bill
Smith, as the frontrunner regardless of the facts. To the extent that Postmedia’s coverage
was flawed, it was flawed in the more commonplace ways that occur in these exclusive
relationships. While the Postmedia newspapers were not skeptical enough of the
Mainstreet polls, their coverage was not completely devoid of caution. For example, in its
September 30 edition, the Calgary Herald treated the finding of the first Mainstreet poll
that had Smith with a nine-point lead, with some circumspection. The front-page headline
on its news story highlighted that it was a “two-horse race”, in contrast with some other
media that emphasized the Smith lead. The Calgary Herald’s political columnist, Don
Braid, wrote in the same issue that Smith’s campaign was surprised by the size of the
lead in a campaign they thought was running about even, and commented for his own
part: “At this point it is still hard to imagine Nenshi actually losing.”
Nonetheless, we were told by Melanee Thomas, a professor of political science at the
University of Calgary who is an expert in polling methodology, that she received the
results of the first Mainstreet poll from a Calgary Herald reporter on an embargoed basis

48

 and that she raised extensive methodological concerns. Her principal concerns as she
described them to the panel, were that the results did not comport with what was known
about the campaign to that point and she raised questions about Mainstreet Research’s
sample (what Mainstreet calls its “directory” from which the sample is drawn). She said
the reporter told her flatly that the article would not deal with any methodological issues
with the poll. Thomas later disputed a direct quotation from her that was included in the
published article.
Although polling is not at the centre of his expertise, Mount Royal political scientist
Duane Bratt is a frequent commentator on Alberta and Calgary politics and was also
asked by the Calgary Herald to comment on the first poll under embargo. Bratt was
concerned with the results among women, showing a significant lead for Mr. Smith,
which did not fit with historical patterns. Bratt was also quoted in the Calgary Herald
article but without any reference to his reservations about the poll.
Unfortunately, as mentioned previously, Postmedia declined to meet with us, so panel
members were unable to hear its version of these events. We were unable to reach the
lead reporter on the polling stories, who has since left the organization.
Over the course of the campaign, both the Calgary Herald and the Calgary Sun published
articles that contained criticism of the Mainstreet Research polls from the Nenshi
campaign as well as academics and commentators. They also reported on the two online
polls that showed Nenshi in the lead. However, these articles were published with less
prominence than were the Mainstreet Research polls. When the Calgary Herald initially
broached the criticisms of the first poll, it did so in an article that placed the emphasis on
Mainstreet Research’s rejoinder rather than the critique of the polls.50 One Sun columnist
returned to the polls several times and discussed them completely uncritically. General
news articles on the campaign generally adopted the frame created by the Mainstreet
polls that Nenshi’s campaign was in trouble and that Bill Smith was his main opponent.
To the degree the newspapers explored the methodological issues raised by Mainstreet
and the two online polls, they focused mainly on mode, somewhat neglecting issues such
as the source and structure of the sample frame, question order and the dates on which the
survey was conducted. These elements seem more germane in retrospect. However, in
this respect, the Postmedia coverage did not differ substantially from the coverage of the
polls by other news organizations, which treated both the polls themselves and the
contradictions among them as news while demonstrating a faltering grasp of the potential
methodological issues.
It should be said that Postmedia was in a better position than other news organizations to
evaluate Mainstreet Research’s public claims of absolute confidence in its work. As
reported to us by Maggi, Postmedia executives requested a conference call to address the
criticisms of the first poll. In the course of that conversation, Mainstreet indicated it
50

“Pollster says track record speaks for itself after Nenshi’s campaign team questions
results”, Calgary Herald, October 1, 2017
49

 would be tweaking its methodology. To a degree, Postmedia seemed to recognize that as
a partner in the polls, it had a responsibility to ensure their soundness, or at least that it
had some common public exposure if the polls turned out to be wrong. Despite
Postmedia’s awareness that Mainstreet Research was concerned enough to make changes
to its polling, the Calgary Herald and the Calgary Sun continued to print quotations from
Mainstreet executives that proclaimed a pugnacious self-confidence about their
methodology. A news outlet’s first responsibility is not to its polling partner, to its own
reputation, or to a particular political outcome, but to its readers. The readers of the
Calgary Herald and the Calgary Sun would have been better informed if Postmedia had
chosen to be more forthcoming with them about what was happening behind the scenes.
It seems apparent in reviewing the media coverage in the newspapers as well as other
media that many journalists were struggling to find the intellectual tools with which to
evaluate the contradictory evidence. They relied on a combination of interviews with the
original pollsters themselves, other pollsters, the campaigns and academics to furnish
commentary without ever reaching a conclusion. One article in the Calgary Herald for
example, threw up its hands, saying that “pundits” thought the truth lay somewhere
between the Smith lead suggested in the Mainstreet polls and the Nenshi lead found in
the online polls. Of course, there was not enough transparency about the methods of the
Mainstreet, the CMES, or the LRT on the Green polls even for experts to explore their
strengths and weaknesses in a forensic way.
It is clear from the media coverage that journalists did not know enough about polling
techniques or have sufficient information about the polls themselves to evaluate the
contradictory polling evidence about the course of the election. Prof. Thomas told us, in
addition, that journalists at both Postmedia and other outlets were often indifferent to
methodological issues and preferred to address the polls as political stories on the shaky
assumption that the results were sound.
A number of those we interviewed pointed out that since the previous election, the
Calgary Herald and Calgary Sun newsrooms have merged and been reduced. Reporters
not normally assigned to political coverage were recruited in for election stories, and the
panel was told students were used to augment ward coverage. Other news outlets have
also reduced their newsrooms and coverage of City Hall. This may have had a number of
effects. First, polling stories were given greater prominence than other stories because
they are easy to do quickly. Second, many of the reporters covering the municipal
election normally work on other beats and thus have less municipal political background.
Third, reporters have less time to delve deeply into issues such as methodology, and less
experience in reporting on polls. Fourth, thinner newsrooms mean fewer other stories on
the campaign and on qualitative assessments of voters’ concerns, which mean reporters
and editors have less information with which to balance and contextualize the
information from the polls. All of these factors may have contributed to the Mainstreet
polls framing the journalistic narrative of an incumbent mayor running behind, perhaps
badly behind, his principal opponent.

50

 In the absence of evidence, it is impossible to know what an ordinary news consumer
might have made of all of this; but it is fair to say that the Mainstreet polls, which were
ultimately the worst prognosticators of the election, received the most prominence and
that coverage of the online polls, primarily in the Postmedia newspapers but also in other
media, received significantly less. Criticism of the Mainstreet Research polls and
coverage of the other polls in the media would have at best created confusion or cast
doubt about the value of any polling in the public mind. Certainly, even the most avid
layperson reading the media coverage of the polls would have had no basis to conclude
much at all about the course of the campaign.
Mayor Nenshi and his aides told the panel they believed that the Mainstreet polls hurt
him early in the campaign by portraying him as vulnerable, portraying Smith as his
principal opponent, and by relegating Chabot to also-ran status. However, Nenshi and his
aides indicated to us that towards the end of the campaign, the perceptions created by the
polls helped them drive up turnout, which was unusually high and much higher than
commentators had expected at the outset of the campaign. None of those we interviewed
expressed doubt that the polling coverage affected the course of the campaign and likely
the results, though there is no hard evidence that we encountered to prove these
intuitions.
In the aftermath of the election Postmedia announced that it had asked Mainstreet
Research to review its methods and its communications in the Calgary election, which as
mentioned above, it subsequently did. Mainstreet Research informed the panel that it is
no longer in a relationship with Postmedia.

51

 Section 4
Conclusion & Recommendations

1. Methodology
The review panel was struck by the fact that all the polls published in the Calgary
election had significant methodological flaws as horserace polls and fell short of the
MRIA’s existing standards of disclosure. The Forum-CMES poll was the most sound
methodologically, though it had been designed for a different purpose, and its collection
period was unusually long as a result. None of the polls was within the margin of error to
the eventual result, and ironically the one that came closest, the AskingCanadian-LRT on
the Green poll, was the one whose principal design flaw (question order) was most
obvious on its face.
No set of standards would make polling impermeable to error, even in theory. However,
the panel review saw a number of areas where practices and disclosure could be
significantly improved.
1.1 Sample Source
The theory of polling is based on the premise that if a group of respondents is randomly
selected from the target universe, the laws of statistics will apply and the results will fall
within a calculable Margin of Error. Historically, after the advent of scientific polling,
this was accomplished by randomly selecting households, and later phone numbers.
While imperfect, these approaches did usually achieve a reasonable facsimile of a
random selection of potential respondents from the population being studied.
The simplicity of these approaches has been undermined by costs, the advent of
cellphones, and the development of new modes of polling, notably online. As a result,
samples are being constructed in novel ways, including the acquisition of lists from a
variety of sources, sometimes collected for unrelated purposes. In the case of the Calgary
election, Mainstreet’s publicly released IVR polls were not conducted by Random Digit
Dialling, but by sampling from a “directory”; the LRT on the Green poll was conducted
by drawing on a large bank of email addresses assembled from many sources, including
retail loyalty programs for example, and adjusted to match Census data.

52

 Obviously, these sources of samples may potentially be the sources of error in polling,
and in the case of Mainstreet’s Calgary polls, there is some evidence that this was the
case. Moreover, these sampling techniques are opaque to the media and the public.
The MRIA needs to review and update its sampling and other polling-related standards
for both practice and disclosure to the public. Our panel’s recommendations regarding
sampling-related issues and proper industry practices are as follows:
1. The MRIA should develop a series of statements regarding standards and best
practices relating to proper sampling techniques. These must be based on high
quality and up-to-date academic and professional research. These statements
would be made available to guide researchers and to inform media and the public
on these matters.
2. Polling firms must be required to disclose fully all the sources of their sample that
were used for a particular poll, and the means by which the sample was acquired.
This includes any third-party sources through which sample lists were acquired or
purchased.
3. When results are released to the public, polling firms must make available a
statement regarding the size of the sample frame in numbers, and where
applicable a clear statement regarding the percent of cell phones that are listed in
the sample frame.
4. Where applicable, the firm must state the percent of cell phone users who were
interviewed in the final results of the survey.
5. The firm must provide a link to its website in its media release for individuals
who wish to read the full wording of a questionnaire’s introduction as well as any
screening questions used in the survey.
6. The firm must provide a link to its website in its media release so that individuals
can examine the unweighted and weighted results by totals and key variables
(such as basic demographics), as well as the statistical weighting procedures (i.e.
mathematical calculations) that were applied to the final results.
7. The polling firm must provide information about whether or not callback
procedures were put into place during the fielding of the survey, and how these
were implemented.
8. The firm must indicate whether or not autodialing technology was used when
doing the poll.
9. The firm must provide a link to completion rate information for the poll. This
includes:
a. The total number of calls made, including not-in-service numbers, etc.;

53

 b. The total number of calls that were answered by a potential respondent;
c. The total number of calls that resulted in a completed interview.
10. When a poll is aimed at understanding the electorate and voter choices, a link
must be provided to the results that must include:
a. Results along with the DNKs (do-not-knows) and “will not vote” and
“refused to answers”
b. Results along with the DNKs
c. And as an option, the results with the DNKs excluded
11. When a poll is released to the public, the media release must clearly identify the
name of the individual (including contact information) who was the director of the
poll, and who oversaw the methodology, sampling, and analysis.
1.2 Margin of Error
Considerable progress has been made in recent decades in terms of the release of Margin
of Error for surveys where appropriate, and the publication of this information in the
media. However, MOE captures only sampling error and ignores other sources of error
which were almost certainly at play in the Calgary polling as in other recent polling
failures. Moreover, some polling modes do not allow for the publication of MOE under
current rules.
One recent attempt to calculate actual historical, rather than theoretical, margin of error in
Canadian federal and provincial elections, put the MOE at 5.68%, 19 times out of 2051 –
well beyond what is clearly communicated to the media and public when polls are
released.
1. The MRIA should study whether an historical MOE can be calculated comparing,
for example, polls conducted in the final week of federal, provincial, and possibly
municipal campaigns with election results. These figures could be published on an
industry-wide basis, or by mode, or perhaps by individual company.
2. Until a new industry-wide practice is adopted by the MRIA, research companies
should adopt language in their releases along these lines: “In addition to the
theoretical sampling error described by margin of error, public opinion polls are
subject to many other possible sources of error in published results. These may
add significantly to discrepancies from election results and are often a factor when
results fall outside the stated margin of error.”
3. Journalistic organizations should be encouraged to adopt reporting language along
these lines as part of their journalistic codes.

51

Bryan Breguet, http://www.tooclosetocall.ca/2018/05/can-we-trust-canadian-polls.html
54

 On a separate issue involving Margin of Error, it is not obvious why phone polls (livecaller or IVR) that sample from lists or directories assembled from a variety of sources
rather than Random Digit Dialling (RDD) should be able to be able to claim a margin of
error when many online surveys cannot.
4. The MRIA should consider how its standards for MOE should be adapted to take
into consideration non-RDD phone polls.

2. Communications & Social Media
In our view, the object and justification for publishing polls during an election is to
inform and enlighten the public. From this insight flows an expectation of transparency
from those conducting polls; not only about their methodology, but also about what roles
they are performing in their public pronouncements. It should go without saying that
researchers should act with civility towards others in the public sphere, even when others
have been uncivil to them.
In public and in the media, pollsters may operate in at least three different roles: as social
scientists, cleaving close to the tentative nature of their data and the fallibility of their
methodology; as pundits, drawing on their general knowledge of politics and polling to
insert their subjective opinions on events; or as entrepreneurs, promoting their brands.
Individual pollsters may choose one of these roles or try to negotiate a balance among
them. However, in the modern day when much of the political polling that appears in the
media is done either free of charge or for some degree of cost recovery only, there is a
natural temptation to emphasize punditry and self-confident brand-promotion over
tentative social science.
It is important to understand that the public is not well equipped to parse these roles; and
the media, while perhaps better equipped, is often disinclined to do so, particularly if
there is a pungent quote or a provocative headline on offer.
In our view, researchers should be transparent about their work, its methods, and its
limitations. They should also be transparent about other commercial activities that the
public would reasonably be interested to know in evaluating their polls. It is clear that in
Calgary there was not sufficient transparency about the polls for the candidates, the
media or the public to evaluate them adequately.
Moreover, as in this case when there turns out to be a massive polling error, an
overweening self-confidence greatly exacerbates the public’s understandable sentiment
that it has been sold a bill of goods. A serious-minded and courteous discussion of
methods during the Calgary election might have helped voters to evaluate the polls. What

55

 happened in Calgary was much more likely to lead many of them to refuse ever again to
take polls or those who conduct them seriously.
2.1 Methodological Transparency
As discussed in Recommendation 1.1 above, researchers should adopt more rigorous
transparency in methodology including designating which individual researcher was
responsible for the overall design of the survey, questionnaire wording and design
(including whether the reported questions were part of a larger survey, and where
questions appeared in the survey), how the sample was derived and by whom, fielding
window, mode, response rate and, where appropriate, callbacks, Margin of Error, and
cellphone versus landline responses. The MRIA should develop and enforce standards of
methodological transparency.
2.2 Financial Transparency
Researchers should reveal who has financed a survey, in whole or in part. To the degree
that a survey is self-financed by the researcher, an explanation should be offered as to the
reason for doing so. The MRIA should develop standards of financial disclosure.
2.3 Commercial Transparency
Researchers should disclose whether they, or any affiliated commercial entity, is
otherwise active during an election campaign. For example, whether it is conducting
polling, voter identification or other services not available to public, or providing services
to those who are, and who is financing these other activities. The MRIA should develop
standards of commercial disclosure.
2.4 Transparency of Role
In their interactions with the media and the public, including on social media, researchers
should be clear about when and how the views they express relate to the data they have
produced. When researchers are acting as pundits or entrepreneurs rather than as social
scientists, they should be clear about it.
2.5 Civility in Discourse
Researchers should maintain a civility in their discourse consistent with their status as
social scientists even when critics act discourteously or uncivilly towards them.
2.6 Conduct on Social Media
The informality and speed of social media can be detrimental to clarity and civility.
Researchers should adopt a social media policy for executives and employees, as media
organizations such as the CBC and the New York Times have done. The policy should
forbid researchers from interacting on social media anonymously with respect to
elections. The MRIA should consider drawing up a model social media policy to assist its
members.

56

 3. Media
3.1 Duties in Exclusive Arrangements
With regard to exclusively obtained polls in particular, news organizations have a
responsibility of due diligence. In entering into an agreement, they should insist on high
methodological standards and disclosure of any potential political or commercial
conflicts. When they have concerns about the methods or professionalism of the
researchers, including with regard to public communications, they should raise them not
only with the provider but be transparent with their readers, listeners or viewers.
3.2 Financial Disclosure
In exclusive arrangements, news organizations should disclose the general financial
arrangements. If the poll is not obtained at full commercial rates, explain who else,
including the researcher, is financing the poll and why.
3.3 Reporting Standards
News organizations should adopt as part of their journalistic codes a requirement that
reporting on polls be subject to the routine critical practices of journalists. These should
include an assessment of the track record of the polling company, the funding of the poll,
potential conflicts of interest, a consideration of the methodology, the existence of
conflicting or contrary evidence available either in other polls or through other forms of
reporting. It is important that no exception to these critical practices be made in the case
of exclusively obtained polls.
3.4 Training
Media organizations should ensure that journalists who regularly report on polls have
specific training on how to analyze and report them, particularly at election time.
3.5 MRIA Role
The MRIA should work to assist the media in understanding and reporting on polls. It
should consider, for example:
•
•

Developing a fact sheet on modern methodology and modes, including frequently
asked questions and a checklist that journalists can use in examining and reporting
on polls
Conducting workshops for journalists in cooperation with media outlets or
journalist organizations such as the Canadian Association of Journalists.

57

 •
•
•
•

Offering a webcast or live Q & A session for journalists on the eve of election
campaigns.
Compiling a list of independent experts and academics who are prepared to assist
journalists in assessing polls and their methods on news deadlines.
Developing a module for use in journalism schools.
Working with media organizations to develop internal standards for the reporting
of polls.

4. MRIA
The MRIA is currently the only Canadian industry organization representing the polling
industry. Many of those we interviewed took the view that the MRIA should play the role
of a professional association in addition to its current role as an industry association.
Meanwhile, the leadership of the MRIA told us that they are interested in adopting a
more proactive role with regard to polling industry standards. In the panel’s view,
commissioning the current review of the Calgary election - something it had not done in
the aftermath of other perceived failures of polling in recent years in Canada - was an
important step in this direction. Several leaders of the MRIA told us that they saw this as
an inflection point towards a more active and effective role for the association.
The MRIA’s imprint was not significantly felt during the Calgary election in part because
in its current configuration, it does not have the instruments at hand to play a useful role.
Janet Brown, a prominent Calgary researcher, who is not currently a member of the
MRIA, told the panel that the MRIA should be taking a more active role in identifying
standards of both methodology and public behaviour and that it should work with the
media to identify failures to meet standards in real time. She suggested that in adopting
this role, the MRIA would not only strengthen the industry and assist the public in
evaluating polls but would make membership in the organization more attractive to
researchers like her.
Some of the participants in the Calgary election, including Forum Research and
AskingCanadians, are MRIA members. However, in both these cases they were
undertaking research for non-members who had commissioned the polls and who
presented the polls to the public. Brian Singh, who was working for the Nenshi campaign
and is an MRIA member, took a very public role in attacking Mainstreet’s methods and
motivations. Despite the fact that Mainstreet Research is not a member of the MRIA - the
result of an unrelated earlier dispute, subject to ongoing litigation - Quito Maggi,
Mainstreet’s CEO, told us he felt that the MRIA should have intervened to check Singh’s
criticisms.

58

 In reviewing the compliance of the published polls in Calgary with MRIA standards, the
panel members were struck by the fact that those standards have not completely come to
grips with the issues raised by fast-changing methodology. Moreover, while Mainstreet
Research, which is not a member of the MRIA, did not meet all the disclosure standards
of the MRIA in its polls, neither did the MRIA members polling in the Calgary election.
Although the panel members did not conduct a review of polling outside the Calgary
election, their impression is that many of the MRIA’s strictures are routinely ignored by
members in the publication of polls.
Simply put, the MRIA standards need faster updating and more energetic accountability
and enforcement, including outreach to the media, if they are to play an effective role in
regulating the industry. Interviews with researchers such as Janet Brown led the panel
members to think that an improvement in the polling standards of the MRIA and more
effective enforcement would not only improve the quality of polls appearing in the media
but might add value to MRIA membership and encourage more researchers to join.
Of course, the MRIA does not possess any legal regulatory powers. However, it is the
only body equipped to promulgate and enforce standards in Canada. In doing so at a time
of rapid methodological change, it must be mindful that its role is not to discourage
innovation or protect the commercial dominance of existing players. This is an
exceedingly delicate task and the MRIA should look outside the ranks of the industry to
academic and other experts who do not have commercial interests at stake.
The recommendations below are intended to suggest a path for the MRIA which would
be built around greater transparency in methods by those who publish polls, and a degree
of accountability built around that framework of transparency.
In its recommendations on methodology above (1.1), the panel proposed enhanced
industry standards for public opinion polling results released to the public. The MRIA
needs to ensure that these standards are adhered to in published polls. In particular, we
recommend:
4.1 Updating Standards
MRIA methodological standards should not stifle innovation or competition but should
keep abreast of them. Given the brisk pace of innovation in the industry, the MRIA
should speed up the process of updating its standards and should draw on academic
participation and advice in order to ensure that standards are based on sound theory and
research. Standards should be under continuous review as industry practices develop and
change.
4.2 Checklist
The MRIA should provide on its website a basic downloadable form with a checklist of
the MRIA endorsed standard-related items. All firms conducting a poll for public

59

 consumption should complete each item in the list in the form, thereby providing
information on how a poll was conducted and in accordance to MRIA standards. This
should be required of members and encouraged among non-members in the expectation
that the media and the public would begin to see compliance as an indication of sound
practice.
a. All media releases providing a poll’s results should contain a link to the
completed form so that readers could examine the extent to which a
particular poll was in accordance with MRIA standards.
b. The completed form should be emailed to MRIA for its own website
posting.
4.3 Audit
The MRIA should audit compliance with standards including the completion and
submission of checklist forms. For any firm that is non-compliant to the standards, the
MRIA should write a confidential letter to that firm requesting that non-compliance be
rectified.
Should the MRIA determine that a firm remains non-compliant, the MRIA should move
towards increasingly serious steps, beginning with a public statement regarding the firm’s
non-compliance to the membership and through a notice on its website. The MRIA might
also consider suspension of membership if a firm remains non-compliant.
The MRIA should provide clear contact information for individuals who wish to
complain about a specific poll or company’s activities relating to standards in public
opinion polling. The MRIA should prepare a quarterly report card to its Board regarding
complaints received and any follow-up action that is required.
4.4 Annual Review
The MRIA should implement an annual review of publicly released polls in Canada
conducted by both members and non-members. This should include:
a. Reporting on the information provided by each firm based on the
checklists proposed above.
b. Information with statistics regarding non-compliance, both for firms
reporting on their activities that are not in accordance to the MRIA
standards and those firms who have not submitted information for polls
conducted.
c. Information on historical MOE (see recommendation 1.2 above).
d. A report on public complaints and their disposition.
4.5 Real-Time Public Engagement

60

 The MRIA should respond quickly to media inquiries about a firm’s compliance with
MRIA standards, and its past records of compliance, based in part on the checklists.
The MRIA should develop a capacity either internally or through reliance on outside
expertise to comment on media coverage of polls, particularly with regard to
methodology and media standards of reporting.
It should also develop a list of neutral academic experts who are willing and able to
comment on the methodology of polls on media deadlines where feasible to assist the
media and public in assessing quality and compliance with MRIA standards.
4.6 Implementation of the Panel Review report
The MRIA should create a committee or other internal mechanism to review this panel’s
report and to provide progress reports to the Board on the extent to which the MRIA is
implementing its recommendations.

5. Mainstreet
Mainstreet Research was principally responsible for what was widely considered a
polling fiasco in the Calgary election. Its survey findings did not remotely mirror what
was happening in the election and its final poll fell many times outside its claimed margin
of error. One researcher ran 100,000 simulations of the Calgary election with simulated
samples of 1,500 and failed to get a single discrepancy as large as Mainstreet’s,52
suggesting that Mainstreet’s failure was more than bad luck. Although the two other polls
that received significant media attention - those from the LRT on the Green Foundation
and CMES - also had results that were outside the stated margins of error when compared
to the final electoral outcome, the discrepancies were not as great and from the public’s
point of view, they at least called the winner correctly.
Mainstreet’s methodological failure was compounded by the way its executives chose to
comport themselves in the media and on social media. They adopted a flamboyant selfconfidence about their polls which was not justified by social science and which should
have been tempered by the many contrary signals they had from other polls, oddities in
their sub-samples, conflicting evidence from their own online sample, and so on.
Although they were by no means the only ones to adopt a tone of reckless rudeness that
characterized the public controversy about their polls, they were enthusiastic contributors.
As the campaign progressed, they began dispensing pundit-style political advice that
went well beyond their research and this became inflected with personal feeling.
52

Bryan Breguet, http://www.tooclosetocall.ca/2017/10/how-bad-were-polls-for-calgarymayoral.html
61

 It should be said that among the victims of the controversy in Calgary - which included
first of all the misinformed public, but also the candidates and academic commentators –
was Mainstreet itself. The controversy affected the reputation of the firm which is an
element in its commercial success.
In the aftermath of the election, Mainstreet took a number of steps that the panel review
applauds. After initially dispatching a lawyer’s letter to one individual, it withdrew its
threat of legal action against him and other critics. Encouraged by its media partner in
Calgary, Postmedia, Mainstreet initiated two reports, one an internal methodological
review and the other an external review of its communications during the election. As the
panel members remarked elsewhere in this report, these actions were in the best traditions
of social science. Moreover, Mainstreet has committed publicly to implementing the
recommendations of these reports and while it is not a member of the MRIA, has pledged
to follow the standards of respected international polling organizations.
Finally, Mainstreet executives, Quito Maggi and Joseph Angolano, made themselves
available to the members of this review for an extended interview and subsequently
exchanged views and answered questions the panel members had in a series of exchanges
by correspondence. These direct interactions with the review panel, as well as the two
reviews commissioned by Mainstreet, assisted us greatly in the research and production
of this report.
Nonetheless, the panel remained uncertain at the end of its inquiries to what extent the
composition of the “directory” used by Mainstreet to extract its sample contributed to the
massive failure in Calgary and why. Moreover, it was beyond the mandate of the panel
review to audit the degree to which Mainstreet has met its stated intentions and
incorporated the lessons of the Calgary election into its subsequent practices.
Mainstreet and the MRIA each told the review panel that they are currently involved in
litigation that arose from an unrelated dispute predating the Calgary election. We did not
see this as the business of the current review. However, the panel does believe that the
public, the industry and Mainstreet itself would benefit from reconciliation, not the least
because it would allow the MRIA to certify the soundness of Mainstreet’s methodology.
To that end, we recommend:
5.1 Audit
Mainstreet Research and the MRIA agree on a respected, independent academic auditor
who could examine its current methods, including the representativeness of its sample,
with particular regard to its use of a “directory”, as well its compliance with its public
commitments and the international standards to which it has pledged to adhere.
5.2 Membership

62

 On successful completion of this audit, Mainstreet should re?join the MRIA.

63

Appendix A: Summary of the Polling Results
Table: 2017 Calgary Election & Publicly Released Polls
Polling Firm
Actual Vote

Fielding
Dates
October
16, 2017

Mainstreet Research Fielded
Poll #1 for Postmedia Sept 28
MR Poll#1 results
with DNKs removed
Released
Sept 30th
Mainstreet Research Oct 3-4
Poll #2 for Postmedia
MR Poll#2 results Released
with DNKs removed
Oct 7th
Mainstreet Research Fielded
Poll #3 for Postmedia Oct 10-11
MR Poll#3 results
with DNKs removed
Released
Oct 13th
Pantheon Research Sept 29released by Common Oct 2
Sense Calgary
AskingCanadians did
the poll for LRT on
the
Green
Foundation
AskingCanadians
results with DNKs
removed
Forum
Research
Canadian Municipal
Election
Study
(CMES)
Forum results with
DNKs removed

Oct 7-10
Released
Oct 11

Sept 28Oct
12
Released
Oct 13

Method

Naheed
Nenshi
199,131
51.38%

Bill
Smith
169,587
43.76%

Andre
Chabot
11,946
3.09%

Others

41.7%

6.6%

4.4%

48.7%

7.7%

5.1%

DNK
14.4%
NA

31.0%

47.6%

5.6%

3.1%

12.8%

35.5%

54.5%

6.4%

3.6%

NA

35.5%

47.2%

5.5%

1.9%

9.9%

39.4%

52.4%

6.1%

2.1%

NA

49.72%

5.91%

7.61%

Not
reported

26%

3%

2%

DNK 28%

36%

4%

3%

NA

33.0%

4.1%

0.5%

DNK
12.5%

37.8%

4.7%

.6%

NA

Votes:
387,583
100%
N=1,000 IVR 32.9%
using
Mainstreet’s 38.4%
sample
N=1,500 IVR
using
Maintreet’s
sample
N=1,500 IVR
using
Mainstreet’s
sample

N=4,887 IVR 36.76%
Sample
source not
stated
N=1,004
41%
Online panel
57%
N=843,
49.8%
Online
survey with
RDD
using
IVR
57.0%
recruitment

DNK

5,662
1.46%

64

 Appendix B: Mainstreet Research Questionnaire
Each of the three polls released to the public by Mainstreet Research commenced with
the following questions53:
Q1: How likely are you to vote in the municipal election?

If you are Absolutely Certain To Vote, press 1
If you are Likely To Vote, press 2
If you are Might Vote, press 3
If you are Unlikely To Vote, press 4
If you are Not Eligible To Vote, press 5 [THANK AND TERMINATE]
Q2: What is your gender?
Press 1 for Male
Press 2 for Female
Q3: What is your age?
Press 1 if you are between 18 to 34 years of age
Press 2 if you are between 35 and 49 years of age
Press 3 if you are between 50 to 64 years of age
Press 4 if you are over 65 years of age
Q4: If the election for Mayor of Calgary were held today which candidate would you
support?
(first three responses randomized)
Naheed Nenshi (route to Q5)
Bill Smith (route to Q6)
Andre Chabot (route to Q7)
Someone Else (route to Q7)
Undecided (Route to Q4)
Q5: Which candidate are you leaning towards?
(first three responses randomized)
Naheed Nenshi (route to Q5)
Bill Smith (route to Q6)
Andre Chabot (route to Q7)
Someone Else (route to Q7)
Undecided (route to Q7)

53

The IVR scripts were provided by Joseph Angolano, in a personal email to Christopher
Adams, April 19, 2018.
65

 Appendix C: Correspondence with Postmedia

Dear Mr. Motley,
As you know the Marketing Research and Intelligence Association (MRIA), the national
standards body for public opinion research in Canada, has launched an independent
review into the root causes of the ‘underperforming and conflicting polling results’ in the
October 19, 2017 Calgary election. For more information about the mandate of the
review and the scope of its work, please refer to the Press Releases that were issued by
the MRIA (here and here).
The three signatories below are the three members of the review panel and our plans are
to complete the work by mid-May in order to present our findings to the MRIA Annual
Conference in Vancouver in early June. To inform our work, we would like to meet with
as many key participants in the Calgary election as possible. Accordingly, we will be
travelling to Toronto on April 5th and 6th and then to Calgary on April 17th and 18th to
conduct a series of one-hour interviews.
The purpose of this communication is to ask whether you would be able to arrange for us
to meet with some of the editors and reporters most closely involved in the polling
coverage of the municipal election when we are in Calgary. It would be most appreciated
if you would indicate which date suits your schedule and what time of the day is most
convenient.
Thank you for considering this invitation. We look forward to meeting with you and/or
other Postmedia staff to better understand how the election polling was done, how the
results were interpreted, and the dynamics that surrounded the October election
campaign. With your insights we will be better positioned to make recommendations that
will help MRIA members and others who are involved in the field of public opinion
research, in their future market and survey research endeavours.
With thanks,
Christopher Adams
University of Manitoba
Paul Adams
Carleton University
David Zussman
University of Victoria

66

 Dear Etta,
Thank you for the reminder.
Unfortunately, a legal action citing Postmedia in a separate matter will prevent us from
being able to take part in this review by the MRIA.
All the best to you.
Lorne
LORNE MOTLEY
VICE PRESIDENT EDITORIAL WEST REGION
EDITOR IN CHIEF, CALGARY HERALD AND CALGARY SUN

Dear Mr. Motley,
Thank you for getting back to us.
We understand that unrelated litigation is an issue for you but let us make a second pitch
to meet with editors or reporters of your selection.
First of all, we are happy to fence off any areas of inquiry that you feel may be sensitive - an accommodation we have made for others.
The mandate we have from the MRIA is to investigate not only the technical reasons for
the issues in the Calgary polls, but also communications by the pollsters and coverage in
the media.
In the course of our inquiry, we have already met with Mainstreet, which has given us
their account and understanding of the editorial relationship with Postmedia. We have
reviewed much of the coverage in your newspapers and other news outlets. We hope to
speak with representatives of other media while in Calgary, as well as hear from pollsters,
politicians and academics, including experts in the media.
Inevitably, our report will reflect in part on Postmedia's coverage of the polls. We feel we
would be in a much better position fully and fairly to evaluate that coverage if we had an
opportunity to meet. We still have many slots open on Wednesday April 18 and would be
happy to come to your newsroom if that were more convenient to you.

67

 Thank you,
Christopher Adams
University of Manitoba
Paul Adams
Carleton University
David Zussman
University of Victoria

Dear Etta,
Appreciate you reaching out again. Unfortunately, none of this changes the potential legal
issues which may arise from commenting — particularly as this is an action still under
review by counsel.
All the best to you,
Lorne
LORNE MOTLEY
VICE PRESIDENT EDITORIAL WEST REGION
EDITOR IN CHIEF, CALGARY HERALD AND CALGARY SUN

68

 Appendix D: Interviews Conducted
The review panel would like to thank all those who met with us to provide their insights,
perspectives, and suggestions. In most cases interviews were done in person, with a few
via conference telephone calls. Some were conducted with individuals who preferred to
speak confidentially and thus are not identified here.
Here we list individuals in the chronological order in which we met.
Toronto, April 5 & 6, 2018
Quito Maggi, President and CEO, Mainstreet Research
Joseph Angolano, Vice President, Mainstreet Research
Jason Cherniak, lawyer for Mainstreet Research
Michael McGregor, Assistant Professor, Politics & Public Administration, Ryerson
University / Principal Investigator, Canadian Municipal Election Study
Jack Lucas, Assistant Professor, Political Science, University of Calgary / Researcher,
Canadian Municipal Election Study
Lorne Bozinoff, President and CEO, Forum Research
William Schatten, Vice President, Forum Research
Graeme Page, Interim CEO, MRIA
Mark Wood, Outgoing Board Chair, MRIA
Amy Charles, Incoming Board Chair, MRIA
Dan Jackson, MIRA Staff, Member Development Officer
Calgary, April 17 & 18, 2018
Duane Bratt, Political Scientist, Mount Royal University
Raj Manocha, Executive VP, Delvinia
Mr. Roy Gonsalves, Senior VP, AskingCanadians
Janet Brown, Scout Communications
David Taras, Professor of Communication Studies, Mount Royal University
Brian Singh, President & Founder, zinc tank
Naheed Nenshi, Mayor of Calgary
Zain Velji, Campaign Manager for Nenshi Campaign
Chima Nkemdirim, Chief of Staff for the Mayor of Calgary

69

 Other Dates – Conference Calls
April 26, 2017
Melanee Thomas, Associate Professor of Political Science, University of Calgary
May 4, 2017
Dr. Claire Durand, Professor of Sociology, Université de Montréal

Thank You
We wish to acknowledge the support and help received from the MRIA and its staff, most
particularly, this includes:
•
•
•
•
•
•
•
•
•
•
•
•

Kara Mitchelmore, (former) Chief Executive Officer MRIA
Graeme Page, Interim Chief Executive Officer MRIA
Mark Wood, Outgoing Board Chair
Amy Charles, Incoming Board Chair
Wayne Lawrence, Member Development Officer
Dan Jackson, Member Development Officer
Lee Robinson, Compliance Officer
Erica Klie, Manager, Member Support Services
Lucy Pizunski, Controller
James Popkie, Social Media and Communications Coordinator
Etta Wahab, Executive Assistant/Accounts Payable
Greg Jodouin, Pace Consulting

The panel also thanks those who preferred not to be identified in this report, but who
made themselves available to provide us with valuable contextual information to help us
better understand what transpired during the 2017 election.

70

 71