EXECUTIVE REPORT Analysis of the Effects of Informational Quizzes on Policy Knowledge, Preferences, and Attitude Change in Wisconsin, Pennsylvania, and New York-19 CONFIDENTIAL DOCUMENT PREPARED BY PROF. DARON SHAW FOR THE INDEPENDENT WOMEN’S VOICE/INDEPENDENT WOMEN’S FORUM Overview Between October 29 and November 7, 2016, the Independent Women’s Voice (IWV) and the Independent Women’s Forum (IWF) sponsored mail and phone calls conveying issuebased messaging—in the form of informational quizzes—in three venues: Wisconsin, Pennsylvania, and congressional district 19 in New York. Unlike most efforts of this sort, the IWV/IWF program was conducted within the context of a massive field experiment, 1 which was designed to assess the impact of these informational quizzes on peoples’ policy knowledge, policy preferences, and more general political attitudes. Put simply, some number of those targeted were randomly identified and set aside as a control group, so that we could compare the influence of outreach on those who received it versus those who did not. The following report delineates the results of these experiments. More specifically, it examines the effects of the IWV/IWF messaging on health care and Supreme Court nominations in the battleground states of Pennsylvania and Wisconsin, as well as in the competitive district 19 race in New York. Design and Data The focus of the current study is on ascertaining how people respond to informational quizzes about substantive policy matters. In particular, the IWV and the IWF were interested in whether or not people are influenced by the presentation of information about the Affordable Care Act (2010) in the form of “true or false” quizzes. The quizzes averaged about ten items each, and were delivered via postcards, direct mail, phone calls, or digital outreach during the 2016 general election campaign. The particular parameters of the individual projects are detailed below… 1 Field experiments have a long academic pedigree and are increasingly used in practical politics as a way to isolate causal effects (internal validity) within the context of actual political campaigns (external validity). See Gerber and Green’s “Get Out the Vote” (2004) for a more extensive discussion of the merits of field experiments over lab experiments or survey-based, cross-sectional analyses. Shaw & Company Research [CONFIDENTIAL] 1 Wisconsin Health care messaging was targeted to all registered independents and all registered Republican-leaning females in the state.2 That created a treatment group of 880,980 households (HHs). Approximately 40,000 additional HHs from these targets were set aside as a control group. The messaging proceeded in three phases. In Phase 1, IWF conducted informational quiz calls or sent informational mailers. In Phase 2, IWV offered an informational phone call on the candidates’ positions on the ACA. Finally, in Phase 3 IWV sent an information postcard (to independents only) on the Supreme Court. Pennsylvania Health care messaging was targeted to all registered independents and all registered Republican-leaning females in the state.3 That created a treatment group of 2,062,391 HHs. Again, 40,000 HHs from these targets were set aside as a control group. The messaging proceeded in two phases. In Phase 1, IWF conducted informational quiz calls. In Phase 2, IWV offered an informational phone call on the candidates’ positions on the ACA. New York – 19 The design here was very similar to Pennsylvania. Health care messaging was targeted to all registered independents and all registered Republican-leaning females in the district.4 That created a treatment group of 60,212 HHs. Again, 40,000 HHs from these targets were set aside as a control group. The messaging proceeded in two phases. In Phase 1, IWF conducted informational quiz calls and sent 2 postcards. In Phase 2, IWV offered an informational phone call on the candidates’ positions on the ACA. The Polls Following the election, we conducted two surveys in each state/district. One survey was of people who received the informational treatment, and the other was of people who were in the control group. This created a six-survey data set—treatment and control polls from WI, PA, and NY-19—with 300 respondents in each. The differences between treatment and control groups in each state/district are the focus of this analysis. The margin of error for the polls is +/- 5.57 percentage points. Although those assigned to either the treatment or control groups should be identical in the aggregate, random noise as well as variations in response rates can produce slight “imbalances” between the groups with respect to demographic and political variables. In additional to simple difference of means tests between treatment and control samples, we also estimate multivariate models in which assignment to treatment is one of several potential explanatory factors for various political attitudes; this allows us to assess possible imbalances and (whenever necessary) control for them in our analyses For Wisconsin, “Republican-leaning” was defined as a Republican vote propensity of 0.20 to 1.00. For Pennsylvania, “Republican-leaning” was defined as a Republican vote propensity of 0.50 to 1.00. 4 For New York -19, “Republican-leaning” was defined as a Republican vote propensity of 0.60 to 1.00. 2 3 Shaw & Company Research [CONFIDENTIAL] 2 Memory of the Quizzes Perhaps the most obvious initial measure of the effect of outreach is whether or not people actually remember the contact. 5 Between 41% (Wisconsin) and 17% (Pennsylvania) of those treated explicitly remember the outreach. This is, in fact, fairly impressive given the amount of campaign material forced on these people in the two weeks before the 2016 election in these venues. Phone appears to have been slightly more memorable than mail: more than twice as many people remember the quiz calls compared to mail in PA and NY-19, while about one-third more remembered the quiz call in WI. Percent of treatment group reporting contact Overall By Mail By Phone By Digital 100 80 60 40 41 34 23 20 8 17 6 22 14 2 9 18 0 Wisconsin Pennsylvania 1 NY-19 Quiz Messaging and Policy Knowledge The Affordable Care Act Beyond simple recall of the contact, we are interested in whether or not people learned something about policy specifics after being given these informational quizzes. Towards this end, the post-election surveys asked people about the discrete pieces of information about the Affordable Care Act offered in the quiz-based messaging. The specific objective was to see if those who were exposed to the ACA quizzes were subsequently more informed on the issue of health care. The estimated effects range from substantial (Wisconsin) to modest (Pennsylvania and NY-19). In Wisconsin, where the treatment was most extensive, out of ten ACA questions those in the treatment group got (on average) 6.5 correct. Those in the control group got, on average, 5.3 correct. There is only about a one in one thousand chance that this effect is statistical noise. For Pennsylvania, the differences between treatment and control are 5.7 correct and 5.3 correct. There is about a six in one hundred chance that this effect is a statistical anomaly. Finally, in NY-19 the difference is 5.4 for treatment and 5.3 for control— one cannot rule out that this small difference is due to statistical noise. 5 One thing to bear in mind, though, is that there is substantial research demonstrating that people can be influenced by campaign information even if they do not remember being contacted. This is at the heart of the distinction between memory-based versus on-line information processing models (see Lodge, Steenbergen, and Brau, 1995). Shaw & Company Research [CONFIDENTIAL] 3 Average number of correct responses out of ten information items on the ACA Control Treatment 10 8 6 6.5 5.3 5.3 5.7 5.3 5.4 4 2 0 Wisconsin Pennsylvania New York-19 In short, ACA knowledge seems to be positively affected by these informational quizzes, especially when they are offered several times and in a variety of formats. Supreme Court Nominees In Wisconsin, you will recall that IWV provided informational quizzes not only on the ACA but also on the ideological balance of the Supreme Court and the process for replacing Supreme Court justices. We found that those who received this information were substantially more likely to correctly answer questions about maintaining balance on the Court. More specifically, 71% in the treatment group correctly noted that the next nominee would need to be a conservative to preserve the current composition of four liberals, four conservatives, and one swing vote. There was also a slight improvement on the question of how many votes are needed to pass a cloture vote and end a filibuster in the U.S. Senate, which may seem esoteric but is actually crucial for Supreme Court nominees. Wisconsin Percent of correct responses on Supreme Court questions Control Treatment 100% 71% 80% 60% 53% 40% 20% 7% 9% 0% In order to maintain balance, the next justice would have to be… Shaw & Company Research [CONFIDENTIAL] Number of votes needed to end a filibuster… 4 Quiz Messaging and Issue Salience Improving substantive knowledge is one thing, but does receiving the quiz information increase the sense that the issue is important? Several results from the analysis indicate that the answer is “yes.” Within the treatment group in Wisconsin, 11% said health care was the most important issue for their vote in 2016, while within the control group it was 8%. In Pennsylvania, it was 9% for the treatment and 7% for the control. In NY-19, it was 8% and 6%, respectively. For the Supreme Court, in Wisconsin (the only venue where Supreme Court messaging occurred), there was a one-point difference between treatment and control (7% of those treated had it as their number one issue, compared to 6% in the control group). When asked how important the issue of Obamacare was in the election, members of the treatment group were 5 points more likely to say it was important in Wisconsin. In Pennsylvania, treatment group members were 8 points more likely to say it was important. In NY-19, however, those in the control group were actually more likely to say it was important. How important was the issue of Obamacare for you in this election? 100 90 80 77 72 70 70 62 68 73 60 50 40 30 20 10 0 Wisconsin (T) Wisconsin (C) Pennsylvania (T) Pennsylvania (C) New York-19 (T) New York-19 (C) But when asked about whether they would be more or less likely to support a candidate who signed a pledge to repeal and replace Obamacare, the effect in NY-19 reverses, with 46% of those in the treatment group saying they’d be more likely to support Faso if they knew this about him, compared with 44% of those in the control group giving a similar response. In Wisconsin the effect is stronger, with 62% in the treatment group saying it would make them more likely to support Johnson, compared to 50% more likely in the control group. (This question was not asked in Pennsylvania, since Toomey did not make this pledge.) Shaw & Company Research [CONFIDENTIAL] 5 Were you more/less likely to vote for the candidate when you learned they had signed a pledge to “repeal and replace” Obamacare? (Base colors indicate “much more likely to vote for,” hatched colors indicate “somewhat more likely to vote for.”) 100 80 60 40 20 15 12 10 12 47 38 36 32 Wisconsin (T) Wisconsin (C) NY-19 (T) NY-19 (C) 0 All told, the data clearly suggest that quiz messaging on the ACA increases the proportion of people who think the issue is important. As expected, the magnitude of this effect is conditioned by the force and repetition of the message, as well as political context. Similarly, on the issue of Supreme Court nominees, in Wisconsin those in the treatment group were 6-points more likely than those in the control to say the issue was important to them in the election. In Pennsylvania, where Supreme Court messaging was not explicitly part of the field experiment, there was a 9-point difference between treatment and control. Quiz Messaging and Voting Behavior Beyond potentially affecting individual-level information as well as issue and policy priorities, it is possible that informational messaging might also indirectly influence political behavior. More specifically, it is possible that information about the ACA or about the Supreme Court might make people more likely to support a conservative and/or Republican candidate for office. Indeed, our field experiment from 2016 suggests that informational quiz messaging—when done with maximal force and repetition—can also have residual or ancillary effects on political behavior. Interestingly, those exposed to the “maximum” version of quiz messaging on the ACA in Wisconsin were significantly more likely to say they supported Republican Donald Trump over Democrat Hillary Clinton. Those in the treatment group preferred Trump by 49 points, while those in the treatment group favored Trump by 21 points (a 28-point difference). But the somewhat less robust efforts in Pennsylvania and New York-19—which occurred during more intense state and local campaigns than we saw in Wisconsin—did not produce comparable presidential ballot effects. In fact, support was greater for Trump in the control groups than in the treatment groups. Shaw & Company Research [CONFIDENTIAL] 6 Presidential Ballot Margin Treatment Margin between Trump and Clinton 60 50 49 Control 46 37 40 30 21 20 20 26 10 0 Wisconsin Pennsylvania New York-19 The multivariate models tell much the same story: in Wisconsin exposure to the quizzes increased the probability of supporting Trump by approximately 14 percent, while there was no statistically significant effect on the presidential ballot in either Pennsylvania or New York -19. The story is much the same with respect to voting for U.S. Senate or U.S House. In Wisconsin, those exposed to quiz information preferred Ron Johnson over Russ Feingold by 52 points compared to a 21-point margin within the control group. According to the models, the treatment effect was slightly less than a 16 percent increase in the probability of voting for Johnson. But there is no statistically significant difference between treatment and control groups with respect to support for Toomey over McGinty in Pennsylvania or Faso over Teachout in New York – 19. U.S. Senate/Congress Ballot Margin Treatment Margin between Rep. and Dem. 60 52 46 50 36 40 30 20 Control 21 20 26 10 0 Wisconsin Shaw & Company Research [CONFIDENTIAL] Pennsylvania New York-19 7 Group Differences Given the composition of study’s target groups, it should be clear that we are particularly interested in how certain kinds of people respond to being presented with issue and policy information. Most obviously, we want to know if those who do not affiliate with either political party or if women are relatively more affected by the informational quizzes. Each is analyzed, in turn, below… Independents Overall, the quiz information treatment produced slightly greater effects among independents than among Republicans. The differences occur across the board, but are perhaps best illustrated by looking at self-reported voting data. The table below shows the results—for independents only—from our multivariate models of presidential and congressional voting across the three states/districts. We see that the quiz effects are about 2 points greater in Wisconsin among independents (16% and 18% increases in Republican vote likelihood, compared to 14% and 16% overall). But whereas there were no other vote effects amongst all respondents elsewhere, amongst independents we also see that the quizzes produced a 4 percent effect in the Toomey-McGinty U.S. Senate race in Pennsylvania. There is only a six in one hundred chance that this effect is due to random noise. Estimated Effect of Quiz Information Treatment on Independents % increase in probability of voting for Rep. candidate Presidential Vote Senate/Congress Vote 25.0% 20.0% 15.9% 17.8% 15.0% 10.0% 3.8% 5.0% 0.0% 0.0% Wisconsin Pennsylvania 0.0% 0.0% New York-19 The take-away is that independents appear to be somewhat more likely to use policy information to inform their vote choice. Women Overall, the quiz information had a slightly greater impact on women than on men. Again, the self-reported voting data are perhaps most instructive here. In Wisconsin, our models tell us that exposure to the treatment made men 11 points more likely to support Trump over Clinton; the same exposure made women 17 points more likely to support Trump. In the U.S. Senate race, by contrast, gender differences were statistically insignificant. In Pennsylvania and New York – 19, women were slightly more likely than men to prefer the Republican candidate (for president or for U.S. Senate/House) when exposed to the quiz messages. But the differences, while consistent, were not statistically significant. Shaw & Company Research [CONFIDENTIAL] 8 Moreover, even for women the treatment effects were indistinguishable from zero in Pennsylvania and New York - 19. State/District Gender Wisconsin Men Coefficient estimating effect of treatment on Trump vote Coefficient estimating effect of treatment on Rep. vote +0.111* (0.049) +0.174*** (0.051) -0.136** (0.049) +0.043 (0.048) -0.075 (0.051) +0.009 (0.050) +0.167*** (0.052) +0.153*** (0.050) -0.107** (0.050) +0.007 (0.047) -0.103 (0.055) -0035 (0.048) Women Pennsylvania Men Women New York – 19 Men Women Notes: Cell entries are regression coefficients. Parenthetical numbers are the standard errors associated with the estimates. ***statistically significant at the 0.001 level; **statistically significant at the 0.01 level; *statistically significant at the 0.05 level. In sum, women do appear to be a bit more responsive to the sorts of informational messaging under scrutiny here. This “responsiveness,” however, is only significant when (a) the treatment is maximal (as in Wisconsin), and (b) the Republican position or candidate of interest is, ceteris paribus, less appealing to women. Conclusion The results of the Wisconsin, Pennsylvania, and New York–19 field experiments indicate that even relatively modest quiz messaging can have an impact on people, especially with respect to their knowledge of issue and policy arguments. However, to persuade people about the salience of the issue, or to move broader attitudes on the issue, it seems that multiple “touches” and different modes of outreach are necessary, as we saw in the Wisconsin test. The more difficult question, of course, is gauging what all of this means for the election. The projects conducted in Pennsylvania and New York – 19 did not appear to have statistically significant effects on the elections for U.S. President, Senate, or House. In Wisconsin, however, the effects were both statistically and substantively significant. A little back of the envelope estimation is useful here: There were 3,619,996 registered voters in Wisconsin. We had a target universe of 880,980 households (recall that we targeted independents and Republican-leaning women). Let us assume that there are roughly 1.75 target voters per household (given our gender focus, it seems likely that the usual conversion rate of 2.0 or 2.2 voters per household is too high, hence our downward adjustment). This yields 1.54 million registered voters, or 43 percent of the total registered voter population. The estimates from our models indicate that Trump received 1.28 million votes (83%) from this target population. If the quiz messaging had not occurred, estimates from the control group (40,000 households) indicate that Trump would have received 1.06 million votes (69%). In other words, had the IWF/IWV quiz information messaging not occurred, Trump would have received 215,840 fewer votes. Given that his ultimate margin in Wisconsin was 30,000 or so, this outreach clearly had an impact. Even if the control group had a baseline Republican vote of 79% (10 points higher than we Shaw & Company Research [CONFIDENTIAL] 9 estimate here), the total vote loss for Trump had messaging not occurred would have been 61,669—again, well over his margin of victory. For Johnson, a similar back-of-the-envelope calculation suggests he would have received 246,674 fewer votes with no quiz information messaging program. There is also, as mentioned earlier, some evidence that these effects slightly greater among independents. In Wisconsin, the added boost is 1-2 points, while in the U.S. Senate race in Pennsylvania it is closer to 4 points. Further Analyses and Final Thoughts A number of questions remain with respect to how (and how much) informational issue messaging influences political behavior in competitive elections. For example, one possibility not explored here is that the outreach might have produced a turnout effect. Put another way, perhaps lower propensity individuals who received the quiz information messaging were more likely to vote than those in the control group. This possibility makes sense given the increased knowledge we see from these people on policy items in the post-election survey (the relationship between knowledge and voting is well-documented). Unfortunately, very few people reported not voting in the post-election surveys. This could mean that both treatment and control groups experienced comparable turnout rates. Or it could mean that there is a social desirability bias when it comes to turnout that makes discerning differences problematic. In either event, we will validate individual-level selfreported turnout estimates with actual data from the voter files as soon as the states update their files with 2016 results. At that time, we can also estimate differences between our treatment and control groups. Another possibility that we did examine is whether late deciders were relatively more likely to respond to the treatment. In other words, if we isolate those who reported deciding “after the debates” do we see a greater difference between treatment and control groups? The answer is “not really,” although this null finding may be a function of the fact that approximately two-thirds of respondents say they decided prior to the debates. It is, we think, interesting that treatment and control differences in Wisconsin are so great given the large number of people who claim to have made up their minds very early on. This leads to a related observation that we would like to emphasize. The experiments we conducted late in the 2016 general election campaign suggest that quiz messages are more likely to be effective in a relatively “quiet” environment—the more cacophonous campaign environments of Pennsylvania and New York -19 proved tougher sledding when it comes to reaching voters. Practically, the implication is that outreach ought to be done early on, perhaps in late August or early September, before the volume and partisanship increase and drown out substantive, factual information. Shaw & Company Research [CONFIDENTIAL] 10 Methodology From November 9-12, 2016, Sentis Research conducted six telephone surveys: one each for treatment and control groups in New York-19, Pennsylvania, and Wisconsin. The surveys utilized a list-based, stratified probability sample design, with respondents being randomly selected at the level of the household. On average, respondents completed the interview in 12 minutes. For each survey, approximately 7,200 records were drawn to yield 300 completed interviews. A total of 300 respondents were obtained for each survey, with an associated margin of error of +/- 5.57 percentage points. Estimates of opinion for specific sub-groups have a larger margin of error. Shaw & Company Research [CONFIDENTIAL] 11 Appendix IWV/IWF Outreach in Wisconsin, Pennsylvania, and New York - 19 Date Name Where Paid for by 10/29/16 10/29/16 Wisconsin ACA Quiz Call #1: Eric Novack New York CD 19 ACA Quiz Call #1: Eric Novack Pennsylvania ACA Quiz Call #1: Eric Novack Wisconsin ACA Quiz Call #2: Eric Novack Wisconsin NY-19 IWF IWF Pennsylvania Wisconsin IWF IWF New York CD 19 ACA Quiz Call #2: Eric Novack Pennsylvania ACA Quiz Call #2: Eric Novack Wisconsin ACA Generic Mailer #1 NY-19 IWF Pennsylvania Wisconsin IWF IWF Wisconsin ACA Quiz Call #3: Eric Novack New York CD 19 ACA Quiz Call #3: Eric Novack Pennsylvania ACA Quiz Call #3: Eric Novack Wisconsin ACA Generic Mailer #2 Wisconsin SCOTUS Connection Postcard Mailer #1 Wisconsin ACA Quiz Call #4: Eric Novack New York CD 19 ACA Quiz Call #4: Eric Novack Pennsylvania ACA Quiz Call #4: Eric Novack Wisconsin ACA Connection Call #1: Eric Novack New York CD 19 ACA Connection Call #1: Eric Novack Pennsylvania ACA Connection Call #1: Eric Novack Wisconsin NY-19 IWF IWF Pennsylvania Wisconsin Wisconsin IWF IWF IWV Wisconsin NY-19 IWF IWF Pennsylvania Wisconsin IWF IWV NY-19 IWV Pennsylvania IWV 10/29/16 10/31/16 10/31/16 10/31/16 10/31/16 11/2/16 11/2/16 11/2/16 11/2/16 11/3/16 11/5/16 11/5/16 11/5/16 11/7/16 11/7/16 11/7/16 Shaw & Company Research [CONFIDENTIAL] 12 Biography Daron R. Shaw University Distinguished Teaching Professor & Frank C. Erwin, Jr. Chair of State Politics University of Texas at Austin Professor Shaw received his B.A. and Ph.D. degrees from UCLA before joining the faculty at the University of Texas at Austin in the fall of 1994. His most recent book is “Unconventional Wisdom” (Oxford University Press), which examines across time survey data to inform the popular conversation about voting and elections in the United States. In 2006, he published “The Race to 270” (University of Chicago Press) which analyzes the effects of TV advertising and candidate visits on the 2000 and 2004 presidential elections. In addition, Professor Shaw has published articles in the American Political Science Review, American Journal of Political Science, The Journal of Politics, Political Communication, The British Journal of Political Science, Public Opinion Quarterly, Political Behavior, Political Research Quarterly, Presidential Studies Quarterly, PS: Political Science, Party Politics, Electoral Studies, The Journal of Political Marketing, and American Politics Research. Before accepting a position at Texas, Professor Shaw worked as a survey research analyst in several campaigns, including a stint as senior national data analyst for the 1992 Bush-Quayle campaign. In 1999-2000, he served as director of election studies for the Bush-Cheney campaign. In 2004, he served as a consultant for the Bush-Cheney campaign and the Republican National Committee. Professor Shaw is currently a member of the board of overseers for the National Election Study, a member of the Fox News Decision Team, codirector of the Fox News Poll, the PI for the UT Government Department/Texas Tribune survey, the director of the Texas Lyceum Poll, and a member of the advisory board for the Annette Strauss Institute. In 2013 he served as one of the lead academic advisors for the Presidential Commission on Election Administration. From 2003-2009 he served as a presidential appointee to the National Historical Publications and Records Commission. He has also served as a research fellow for the Hoover Institution, and as a consultant for the Tomas Rivera Policy Institute and the Texas Poll. Professor Shaw is the founder and director of Shaw & Company Research. Professor Shaw teaches American Government, Campaigns and Elections, Political Parties, Public Opinion and Voting Behavior, and Applied Survey Research. Shaw & Company Research [CONFIDENTIAL] 13