PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

4/26/201611:20 PM

CHILLING EFFECTS:
ONLINE SURVEILLANCE AND WIKIPEDIA USE
Jonathon W. Penney*

ABSTRACT
This article discusses the results of the first empirical legal study providing evidence of
regulatory “chilling effects” of Wikipedia users associated with online government surveillance.
The study explores how traffic to Wikipedia articles on topics that raise privacy concerns for
Wikipedia users decreased after the widespread publicity about NSA/PRISM surveillance
revelations in June 2013. Using an interdisciplinary research design, the study tests the
hypothesis, based on chilling effects theory, that traffic to privacy-sensitive Wikipedia articles
reduced after the mass surveillance revelations. The Article finds not only a statistically
significant immediate decline in traffic for these Wikipedia articles after June 2013, but also a
change in the overall secular trend in the view count traffic, suggesting not only immediate but
also long-term chilling effects resulting from the NSA/PRISM online surveillance revelations.
These, and other results from the case study, not only offer compelling evidence for chilling
effects associated with online surveillance, but also offer important insights about how we
should understand such chilling effects and their scope, including how they interact with other
dramatic or significant events (like war and conflict) and their broader implications for privacy,
U.S. constitutional litigation, and the health of democratic society. This study is among the
first to demonstrate— using either Wikipedia data or web traffic data more generally— how
government surveillance and similar actions impact online activities, including access to
information and knowledge online.

I.

INTRODUCTION

On March 10, 2015, the American Civil Liberties Union, on behalf of the
Wikimedia Foundation and eight other organizations, filed a lawsuit against the
United States Department of Justice and the National Security Agency (NSA)
challenging the constitutionality of NSA online surveillance:
This lawsuit challenges the suspicionless seizure and
searching of internet traffic by the National Security Agency
(“NSA”) on U.S. soil. The NSA conducts this surveillance,
called “Upstream” surveillance, by tapping directly into the
internet backbone inside the United States—the network of
high-capacity cables, switches, and routers that today carry vast
numbers of Americans’ communications with each other and
with the rest of the world. In the course of this surveillance,

Electronic copy available at: http://ssrn.com/abstract=2769645

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __
the NSA is seizing Americans’ communications en masse while
they are in transit, and it is searching the contents of
substantially all international text-based communications—and
many domestic communications as well—for tens of
thousands of search terms. The surveillance exceeds the scope
of the authority that Congress provided in the FISA
Amendments Act of 2008 (“FAA”) and violates the First and
Fourth Amendments.1

An Op-Ed published the same day in The New York Times, co-authored by
Wikipedia Founder Jimmy Wales and Wikimedia Foundation’s Executive
Director Lila Tretikov, explained the lawsuit was necessary because “pervasive
surveillance” caused “a chilling effect” that stifled the “freedom of expression”
and “free exchange” of ideas on Wikipedia,2 the collaborative online
encyclopedia that is global in both content and scope—it contains over 30
million articles available in over 200 languages and is among the ten most
visited websites globally.3 However, like previous constitutional challenges to
* The author would like to thank Victoria Nash, Urs Gasser, Joss Wright, Ron Deibert,
Paul Kelly, Ryan Budish, Jamie Baxter, Vern Paxon, Kendra Albert, Andy Sellars, Molly
Sauter, J. Nate Mathias, Yana Welinder, Michelle Paulson, Geoff Brigham, David Abrams,
Bernie Black, Mordechai Kremnizer, Adam Chilton, Adi Leibovitch, Kate Litvak, James W.
Coleman, Aurelie Ouss, Ben Perryman, Dahlia Othman, Elissa Berwick, Ari Waldman,
Meredith Whittaker, Daniel Villar-Onrubia, Wenming Xu, Yehonatan Givati, Jeff Knockel, Jed
Crandall, Phillipa Gill, Masashi Crete-Nishihata, and Derek Bambauer for comments, advice,
and/or suggestions on previous drafts or talks based thereon. The author would also like to
thank participants for his talks at the 2015 Internet Law Work in Progress Conference, Santa
Clara Law School, Santa Clara University, March 2015; the Berkman Center for Internet &
Society, Harvard University, May 2015; the 2015 Free and Open Communications on the
Internet (FOCI) workshop, USENIX Security Symposium, Washington, D.C., August 2015;
and the Inaugural Society for Empirical-Legal Studies (SELS) Global Workshop for Junior
Empirical Legal Scholars, Hebrew University, Jerusalem, Israel, December 2015.
1 Complaint for Declaratory and Injunctive Relief at 1, Wikimedia Found. v. NSA,
No. 1:15-cv-00662-RDB, 2015 WL 1033734, (D. Md. Mar. 10, 2015),
https://www.aclu.org/files/assets/wikimedia_v2c_nsa_-_complaint.pdf. For a definition of
mass surveillance, see Ben Beaumont, Easy Guide to Mass Surveillance, AMNESTY
INTERNATIONAL
(Mar.
18,
2015),
https://www.amnesty.org/en/latest/campaigns/2015/03/easy-guide-to-mass-surveillance/.
2 Jimmy Wales & Lila Tretikov, Opinion,Stop Spying on Wikipedia Users, N.Y.
TIMES(Mar. 10, 2015), http://www.nytimes.com/2015/03/10/opinion/stop-spying-onwikipedia-users.html
3 See Judit Bar-Ilan & Noa Aharony, Twelve Years of Wikipedia Research, PROCEEDINGS
OF THE 5TH ACM CONFERENCE ON WEB SCIENCE 243, 243 (2014); David J. McIver & John
S. Brownstein, Wikipedia Usage Estimates Prevalence of Influenza-Like Illness in the United States in
Near Real-Time, 10:4 PLOS COMPUTER BIOLOLOGY 1, 1 (2014); Stefanie Hilles, To Use or Not
to Use? The Credibility of Wikipedia, 10:3 PUBLIC SERVICE QUARTERLY 245 (2014).Wikipedia
provides a wealth of information about its number of articles, editors, page views, etc.
SeeWikimedia Statistics, http://stats.wikimedia.org/#fragment-14 (“The Analytics team

Electronic copy available at: http://ssrn.com/abstract=2769645

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

3

NSA surveillance,4the lawsuit was not heard on the merits but dismissed in
October for lack of standing.5 Wikimedia Foundation, the lead complainant,
intends to appeal.6
The idea that government surveillance is harmful to free expression and
association is not new, nor is skepticism about its empirical and legal basis. In
the 1972 Supreme Court case Laird v. Tatum, for example, the complainants
argued that broad government surveillance and data gathering
unconstitutionally chilled the their rights.7 The Court rejected the claim due to
lack of standing, finding that the surveillance did not constitute an “objective
harm or a threat of specific future harm.”8 The decision reflected a deep
skepticism about both the potential chilling effects and attendant harms of
surveillance.9 Such “judicial skepticism”10 has persisted over the decades. In a
2013 case, Clapper v. Amnesty International,11the Court cited to Laird to dismiss a
empowers and supports data informed decision making across the Foundation and the
Community.”).
4 See e.g., Clapper v. Amnesty Int‘l, 568 U.S. ___, ___, 133 S. Ct. 1138, 1152 (2013)
(dismissing a constitutional challenge to NSA surveillance practices for lack of standing).
5 Wikimedia Foundation, et al. v. NSA, et al., Case 1:15-cv-00662-RDB, slip opinion,
2015 WL 6460364 (D. Md. Oct. 23, 2015).
6 Michelle Paulson and Geoff Brigham, District Court Grants Government’s Motion to
Dismiss Wikimedia v. N.S.A., Appeal Expected, WIKIMEDIA BLOG(Oct. 23, 2015),
http://blog.wikimedia.org/2015/10/23/wikimedia-v-nsa-lawsuit-dismissal/ (“Unfortunately,
the court did not actually rule on whether the NSA’s upstream surveillance is legal or illegal.
Judge T.S. Ellis III, the presiding judge, dismissed the case on standing grounds... We
respectfully disagree with the Court’s decision to dismiss. There is no question that Upstream
surveillance captures the communications of both the user community and the Wikimedia
Foundation itself. We believe that our claims have merit. In consultation with our lawyers at
the ACLU, we will review the decision and expect to appeal to the Fourth Circuit Court of
Appeals.”).
7 Laird v. Tatum, 408 U.S. 1, 13–14 (1972) (“[The respondents’] claim, simply stated, is
that they disagree with the judgments made by the Executive Branch with respect to the type
and amount of information the Army needs, and that the very existence of the Army's datagathering system produces a constitutionally impermissible chilling effect upon the exercise of
their First Amendment rights. That alleged ‘chilling’ effect may perhaps be seen as arising
from respondents' very perception of the system as inappropriate to the Army's role under our
form of government, or as arising from respondents' beliefs that it is inherently dangerous for
the military to be concerned with activities in the civilian sector, or as arising from
respondents' less generalized yet speculative apprehensiveness that the Army may at some
future date misuse the information in some way that would cause direct harm to respondents...
Allegations of a subjective ‘chill’ are not are not an adequate substitute for a claim of specific
present objective harm or a threat of specific future harm.”). Chilling effects theory is
discussed in more detail in Part II.A of this Article.
8 Id. at 15.
9 See Margot E. Kaminski & Shane Witnov, The Conforming Effect: First Amendment
Implications of Surveillance, Beyond Chilling Speech, 49 U. RICH. L. REV. 465, 480 (2015).
10 Id. at 482.
11 Clapper, supra note 4, at 1152.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

4

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

challenge to the legality of NSA surveillance authorized by the Foreign
Intelligence Surveillance Act (FISA), and noted that chilling effects fears were
“too speculative.”
Skepticism about chilling effects is not confined to courts. Legal
commentators have long questioned the existence or scope (beyond mere
triviality) of surveillance related chilling effects harms, or expressed skepticism
as to whether the premises of chilling effects theory could be empirically
substantiated. Even Frederick Schauer, who offered an early classic statement
of chilling effects theory and doctrine, admitted in 1978 that its empirical
assumptions about human behavior were “most likely unprovable.”12 Nearly a
decade after Schauer, Vincent Blasi observed that the notion of “chilling
effects” on supposed “fearful and overly risk-averse” speakers was “oftcriticized” and based on “crude behavioral speculation.”13 More recently,
Leslie Kendrick, after surveying both literature and case law, emphasized the
theory’s “weak” and “flimsy” empirical basis and concluded additional
research was required for the “unsubstantiated empirical judgments” of
chilling effects claims.14 Also recently, Margot Kaminski and Shane Witnov,
while acknowledging certain social science studies that corroborate forms of
chilling effects, nevertheless called for more empirical work on surveillance
and its impact in a “number of critical areas,” including the existence,
magnitude, and persistence of surveillance related chilling effects.15
Privacy theorists, security researchers, and social scientists have also
expressed skepticism about the possibility of large scale chilling effects caused
by online surveillance.16 One reason for such skepticism is increasing public
acceptance of, or desensitization to, privacy and surveillance concerns,

12 Frederick Schauer, Fear, Risk, and the First Amendment: Unraveling the “Chilling Effect”, 58
B.U. L. REV. 685, 730 (1978).
13 Vincent Blasi, The Pathological Perspective and the First Amendment, 85 COLUM. L. REV.
449, 482 (1985).
14 Leslie Kendrick, Speech, Intent, and the Chilling Effect, 54 WM. MARY L. REV. 1633, 1657
(2013).
15 Kaminski &Witnov, supra note 9, at 517(calling for further research on the “types of
surveillance and surveillance cues that cause chilling effects”, as well as the strength and
persistence of such chilling effects).
16 See e.g., Daniel Solove, The First Amendment as Criminal Procedure, 82 N.Y.U. L. REV. 112,
155 (2007)(“Determining the existence of a chilling effect is complicated by the difficulty of
defining and identifying deterrence. It is hard to measure the deterrence caused by a chilling
effect because it is impossible to determine with certainty what people would have said or
done in the absence of the government activity. Often, the primary evidence will be a person's
own assertions that she was chilled, but merely accepting such assertions at face value would
allow anyone claiming a chilling effect to establish one. At the same time, demanding empirical
evidence of deterrence is impractical because it will often be impossible to produce.”).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

5

particularly in new technological contexts.17 Indeed, some research in the field
suggests that any chilling effects would, at the very most, be temporary or
ephemeral, as online users have shifted quickly in response to shifting norms.18
So, the empirical basis for chilling effects theory, and its different
dimensions, remain controversial. Part of the challenge, as privacy scholars like
Kendrick,19 Daniel Solove,20 and Neil Richards21 have noted, is the often
17 See David Lyon, Surveillance, Snowden, and Big Data: Capacities, Consequences, Critique, 1
BIG DATA & SOC’Y 1, 51 (2014) (noting that the constant “ratcheting up” of government
surveillance in recent times is not just a product of the growth of new technologies, but also
broader cultural trends accommodating increasing amounts of societal surveillance); BRUCE
SCHNEIER, DATA AND GOLIATH: THE HIDDEN BATTLES TO CAPTURE YOUR DATA AND
CONTROL YOURWORLD95-99 CH. 6 (2015) (Schneier, a leading information security expert,
speaks of how surveillance leads to “conformity” (95-99) and, in Chapter 6, generally discusses
the need to change lax and accepting public attitudes about increasing surveillance and its
harms); Sandro Nickel, The Double-Edged Effects of Social Media Terror Communication: Interconnection
and Independence vs. Surveillance and Human Rights Calamities, in NEW OPPORTUNITIES AND
IMPASSES: THEORIZING AND EXPERIENCING POLITICS 255, 263(Zeynep Guler, ed., 2014)
(“The majority of the population will most probably not fall into self-censoring behavior, a
reason for this possibly being the desensitization concerning privacy in general, at least coconstituted by the very digital experiences of the past decade(s)...”).
18 Bernhard Debatin & Jennette P. Lovejoy, Facebook and Online Privacy: Attitudes,
Behaviors, and Unintended Consequences, 15:1 J. COMPUTER-MEDIATED COMM. 83 (2009),
http://dx.doi.org/10.1111/j.1083-6101.2009.01494.x (documenting Facebook users’ “lax”
attitudes about privacy concerns). For research or works suggesting online chilling effects
would be temporary or ephemeral, see Laura Bernescu, When is a Hack not a Hack: Addressing
the CFAA's Applicability to the Internet Service Context, U. CHI. LEGAL F. 633 (2013)(arguing that
users will quickly adopt to changes in the regulatory environment in relation to the Computer
Fraud and Abuse Act, rendering any “chilling effects” temporary); Chris Rose, The Security
Implications of Ubiquitous Social Media, 15:1INT’L J. MGMT & INF. SYS. 35, 37 (2011) (noting that
increased comfort with using the Internet has led many consumers to conform to new norms,
particularly on privacy). See also Alessandro Acquisti, Leslie K. John & George Loewenstein,
What is Privacy Worth?, 42 J. LEGAL STUD. 249, [pin cite] (2013),
https://www.cmu.edu/dietrich/sds/docs/loewenstein/WhatPrivacyWorth.pdf(advancing
explanations for the disconnect between privacy attitudes and the lax or loose approach to
privacy in practice).
19 Kendrick, supra note 14, at 1638 (“But there are reasons to doubt the chilling effect
account. A claim of a chilling effect necessarily rests upon suppositions about the deterrent
effects of law. These suppositions rest in turn upon predictions about the behavior of speakers
under counterfactual conditions. Meanwhile, the selection of a remedy for chilling—such as an
intent requirement—rests on similar predictions about the remedy’s speech-protective effects.
In short, both the detection of a problem and the imposition of a remedy involve intractable
empirical difficulties.”).
20 Solove, supra note 16, at 155 (“Often, the primary evidence will be a person's own
assertions that she was chilled, but merely accepting such assertions at face value would allow
anyone claiming a chilling effect to establish one. At the same time, demanding empirical
evidence of deterrence is impractical because it will often be impossible to produce.”).
21 Neil Richards, The Dangers of Surveillance, 126 HARV. L. REV. 1934, 1964 (2013)(“This is
not to say that individual determinations of the chilling of intellectual activities will always be
easy. Determining whether a chill to intellectual privacy is substantial would certainly present
difficult cases at the margins.”).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

6

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

“intractable empirical difficulties” in designing research to demonstrate or
measure chilling effects.22 Showing the impact and harms of surveillance
involves dealing with counterfactuals or proving a negative—self-censorship.
As such, it is “difficult to establish either the presence or the absence of a
chilling effect, let alone to measure the extent of such an effect”.23 With the
absence of sound empirical research to substantiate chilling effects,
compounded by the methodological challenges for designing and carrying out
such research, it is unsurprising skepticism about the theory persists.
With the revelation of widespread internet surveillance by the United
States and other Western governments (thanks to the leaks and disclosures of
Edward Snowden), the need for empirical and theoretical study has taken on
even greater urgency. This is particularly true because of the range of lawsuits
filed by companies, citizen groups, and organizations to challenge government
surveillance and related laws,24 but beyond the legal arena, it is crucial for
understanding the potential harms of such surveillance to activities online.
The empirical case study discussed in this Article attempts to address this
research void. Building on a recent study of Google search traffic and Internet
surveillance,25 this Article discusses the first original empirical study of the
impact such surveillance has had on Wikipedia use. Consistent with the recent
growth of empirical approaches in legal research26, the study uses an empirical

22 Kendrick, supra note 14, at 1675.
23 Id. at 1638.
24 For a review of the broad range of constitutional litigation arising since the Snowden
leaks, see generally: Edward C. Liu, Andrew Nolan, & Richard M. Thompson, CONG.
RESEARCH SERV., R43459, OVERVIEW OF CONSTITUTIONAL CHALLENGES TO NSA
COLLECTION ACTIVITIES AND RECENT DEVELOPMENTS7-5700, 1 (2014)(“Beginning in the
summer of 2013, media reports of foreign intelligence activities conducted by the National
Security Agency (NSA) have been published and are apparently based on unauthorized
disclosures of classified information by Edward Snowden, a former NSA contractor... As the
public’s awareness of these programs has grown, Members of Congress and the public have
increasingly voiced concerns about the constitutionality of these programs. This report
provides a description of these two programs and the various constitutional challenges that
have arisen in judicial forums with respect to each.”).
25 Alex Marthews & Catherine Tucker, Government Surveillance and Internet Search Behavior,
(MIT
Sloane
Working
Paper
No.
14380,
Apr.
29,
2015),
https://mitsloan.mit.edu/shared/ods/documents/Tucker_WP_2015_Government.pdf&PubI
D=14380
26 For discussion of the empirical and experimental turn in legal research, see Daniel E.
Ho & Larry Kramer, Introduction: The Empirical Revolution in Law, 65 STAN. L. REV. 1195 (2013).
See generally Gregory Shaffer & Tom Ginsburg, The Empirical Turn In International Legal
Scholarship, 106 AM. J. INT’L L. 1 (2012) (also discussing the empirical “turn” in international
law research). See also Adam Chilton & Dustin Tingley, Why the Study of International Law Needs
Experiments, 52 COLUMB. J. TRANS. L. 173, 180, 187-190 (2013) (discussing the “growth” of
experimental and quasi-experimental methods in legal research).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

7

interrupted time series (ITS) design27 to determine whether traffic for articles
that may raise privacy concerns for Wikipedia users decreased after the
widespread publicity about NSA online surveillance activities. In short, this
case study asks: Did Wikipedia traffic for articles on privacy-sensitive topics
decrease after the “exogenous shock” of widespread publicity surrounding the
surveillance programs in June2013?
A hypothesis based on chilling effects theory would hold that Internet
users will be less likely to view or access such privacy-sensitive Wikipedia
articles after the revelations. Ultimately, this case study provides results
consistent with surveillance related chilling effects, among other findings. The
context of the study is also important. Wikipedia was chosen as the focus of
this case study for a number reasons (discussed in more detail in Part III), but
most importantly because any chilling effect on Wikipedia users has farreaching implications. The site, which is growing both in popularity and scope,
serves as an essential source of information and knowledge online, and
functions as an important public tool to complement the democratic process
in promoting collective understanding, decision-making, and deliberation.28
Part II of the Article provides additional context related to chilling
effects research and the impact of the Edward Snowden disclosures. Part III
sets out and justifies the case study’s methodology and research design,
including its focus on Wikipedia. Part IV discusses the results of the study:
Consistent with chilling effects theory, (1) Wikipedia traffic to privacy-sensitive
articles showed a statistically significant reduction after June 2013, and (2)
there was a long lasting change in the overall secular trend in traffic to such
articles. The study’s implications are discussed in Part V. Part VI concludes
and considers possible directions for future research.
27 For discussion of interrupted time series research design, see DONALD T. CAMPBELL,
JULIAN C. STANLEY, & NATHANIEL L. GAGE, EXPERIMENTAL AND QUASI-EXPERIMENTAL
DESIGNS FOR RESEARCH 37-43 (1966) (discussing the components of time series designs and
their methodological advantages and limitations...); Melvin M. Mark, Charles S. Reichardt, &
Lawrence J. Sanna, Time-Series Designs and Analyses in HANDBOOK OF APPLIED MULTIVARIATE
STATISTICS AND MATHEMATICAL MODELING 354-355 (2000) (discussing the use of time series
designs to assess the impact of interventions). See also Carlotta Ching Ting Fok, David Henry,
& James Allen, Research Designs for Intervention Research with Small Samples II: Stepped Wedge and
Interrupted Time-Series Designs, PREVENTATIVE SCIENCE 1, 4 (2015) (published online May 29,
2015): http://link.springer.com/article/10.1007/s11121-015-0569-4 (offers some suggestions
to strengthen the methodological dimensions of ITS designs to study the impact of health
interventions); A.K. Wagner, F. Zhang, & D. Ross-Degnan, Segmented Regression Analysis of an
Interrupted Time Series in Medication Use Research, 27 J. CLINICAL PHARMACY AND THERAPEUTICS
299 (2002) (a discussion advantages of using of segmented regression analysis, along with ITS
design, in the context of health research); Robert B. Penfold, Fang Zhang, Use of Interrupted
Time Series Analysis in Evaluating Health Care Quality Improvements, 13:6 ACADEMIC PEDIATRICS
S38 (2013) (discussing the advantages and limitations of employing time series analysis to
understand and explore the impact of health policy changes).
28 See infra notes 123-129 and accompanying text.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

4/26/201611:20 PM

8

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

II.

CHILLING EFFECTS THEORY AND RESEARCH AFTER
SNOWDEN

This section provides a more in depth discussion of chilling effects
theory, including leading accounts of its dimensions and assumptions and an
overview of related studies. From there, the Snowden leaks are discussed and
re-framed as presenting a research opportunity to study chilling effects theory.
This idea is developed in the latter parts of the section, with a research
design— employed by the case study discussed in this article— centered on
Snowden revelations about the NSA’s PRISM surveillance program, widely
covered in June 2013, as a reference point for study.
A.

CHILLING EFFECTS THEORY

The idea that government laws or actions might chill peoples’ free
activities gained its most prominent early expression in U.S. during the Cold
War. The “chilling effects doctrine,” a legal doctrine in First Amendment
jurisprudence, took shape in a series of cases decided in the 1950s and 60s that
dealt with anti-communist state measures; essentially, the doctrine encouraged
courts to treat rules or government actions that “might deter” the free exercise
of First Amendment rights “with suspicion.”29
But underlying this legal doctrine was a deeper theory with empirical
assumptions about behavior in relation to government acts—that certain state
acts may chill or deter people from exercising their freedoms or engaging in
legal activities. This theory of “chilling effects” received its first comprehensive
exploration in Schauer’s Fear, Risk, and the First Amendment: Unraveling the
“Chilling Effects Doctrine,30 described as the “definitive treatment” of the theory,
Schauer conceived of chilling effects as primarily resulting from peoples’ fear
of prosecution or legal sanction and the uncertainties of the legal
process.31Here, government surveillance may chill or deter people from
29 Richards, supra note 21, at 1949-1950. For discussion of the origins of the doctrine,
see generally Schauer, supra note 12 (discussing the origins, mechanisms, and foundations for
chilling effects doctrine). For the early cases recognizing the chilling effects doctrine, see for
example Wieman v. Updegraff 344 U.S. 183 (1952); Dombrowski v. Pfister 380 U.S. 479
(1965).
30 Schauer, supra note 12 (a comprehensive discussion of the origins, mechanics, and
foundations for chilling effects doctrine).
31 Schauer, supra note 12, at 687-689. For a discussion of Schauer as the “definitive
treatment,” see Julie Cohen, A Right to Read Anonymously: A Closer Look at 'Copyright Management'
in Cyberspace, 28 CONN. L. REV. 981, n.117 (1996) (suggesting Schauer’s work was the
“definitive treatment”). For applications of chilling effects theory to online contexts, scholars
at the Berkman Center for Internet & Society at Harvard University have been particularly

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

9

engaging in certain legal (or even desirable) online activities because they fear
legal punishment or criminal sanction, and do not trust the legal system to
protect their innocence. Daniel Solove’s work has likewise broadened chilling
effects theory by theorizing and exploring modern surveillance and data
gathering, explaining how such practices can create a kind of regulatory
“environmental pollution” that encourages chilling effects and selfcensorship.32 While Solove’s approach does not discount the sorts of chilling
effects Schauer targets, he focuses primarily on how government surveillance
of online activities creates a broader atmosphere of conformity and selfcensorship. Solove, by contrast, is concerned with the way regulatory
actions—particularly information gathering and surveillance—enhance the risk
that a person may suffer harms in the future (e.g., gathering information about
a person’s activities may increase the risk they are later “victimized” by identity
theft or fraud).33 On this account, people are chilled not because they fear
actual punishment for engaging in certain online activities (as Schauer
theorizes), but to avoid risks of other kinds of future dignity, reputational, or
monetary harms, such as the stigma of being labeled or tracked by state actors
as non-conformists, deviants, or criminals, or the broader concern that
information gathered about such activities may be leaked or disclosed publicly
leading to embarrassment or obtained by third parties and used for nefarious
purposes.34 Such risks and considerations create a societal context that
encourages self-censorship.35 Both of these accounts will be useful to
understand and theorize any surveillance related chilling effects observed.36
prolific. See, e.g., JONATHAN ZITTRAIN, THE FUTURE OF THE INTERNET—AND HOW TO STOP
IT 116, 216 (2008) (exploring the potential chilling effects of perfect enforcement of legal
norms by technology measures and the chilling effects caused by citizen surveillance via the
proliferation of ‘tethered appliances’ (like smartphones) with surveillance capacity); Yochai
Benkler, Through the Looking Glass: Alice and the Constitutional Foundations of the Public Domain, 66
LAW & CONTEMP. PROBS. 173, 216-218 (2003) (arguing that the NET Act and Digital
Millennium Copyright Act expand protections for certain legal rights online in such a way that
will chill expression); Wendy Seltzer, Free Speech Unmoored in Copyright’s Safe Harbor: Chilling
Effects of the DMCA on the First Amendment, 24 HARV. J. L. & TECH. 171 (2010) (analyzing
chilling effects and the DMCA).
32 Daniel Solove, A Taxonomy of Privacy, 154 U. PENN. L. REV. 477, 488 (2006).
33 Id. at 487.
34 Id. at 496 (discussing the example of how information obtained by surveillance was
used to discredit and blackmail Martin Luther King, Jr.).
35 Id. at 495.
36 The findings in the empirical legal case study discussed in this article may also provide
insights, in turn, for these theories as well. For example, Schauer theorizes chilling effects
primarily as a product of individual concerns for actual legal punishment or prosecution in an
uncertain legal system, while Solove’s account captures broader risks and concerns that may
also chill— where online users may not actually fear prosecution, but prefer not to have
governments looking over their shoulder or tracking and compiling data about their online
activities (even if legal). Both such approaches will likely explain or account for instances of

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

10

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

Part of the broader picture is the impact of covert surveillance where people
are either unaware of surveillance or are only aware of the general possibility of
it.37 The Snowden leaks and disclosures, which rendered previously covert
surveillance public, have placed concerns like those Solove explores in urgent
and concrete terms,38 and a range of public opinion polls and survey-based
studies have been conducted to study the effects of the disclosures. Studies by
PEN America39 and Pew Research Center40 provide some empirical foundation
for the claim that surveillance has a chilling effect on peoples’ activities online,
but these survey-based studies have important limitations.
Social science research has long illustrated that self-reported or expressed
concerns about privacy do not necessarily reflect people’s actual behavior
online,41 a phenomenon sometimes referred to as the “privacy paradox.”42 The
surveillance-related chilling effects, but one explanation may prove more common than the
other.
37 Id. at 494-496 (relating such surveillance to Jeremy Bentham’s idea for 19th century
“Panopticon” prison design, or the “Panopticon effect”. Solove struggles with the concept of
covert surveillance in his discussion of its harms and effects).
38 Christopher Slobogin, Standing and Covert Surveillance, 41 PEPP. L. REV. 517, 520 (2014)
(noting that as a result of Snowden’s disclosures the U.S. federal government has been forced
to acknowledge previously covert surveillance practices).
39 The FDR Grp. & Pen Am. Ctr., Chilling Effects: NSA Surveillance Drives U.S. Writers to
Self-Censor,
PEN
at3-4
(Nov.
12,
2013),http://www.pen.org/sites/default
/files/Chilling%20Effects_PEN%20American.pdf (noting that 28% of the writers surveyed
had “curtailed or avoided” certain online activities due to “fear of surveillance”); The FDR
Grp. & Pen Am. Ctr., Global Chilling: The Impact of Mass Surveillance on International Writers,PEN
(Jan. 5 2015), http://www.pen.org/sites/default/files/globalchilling_2015.pdf (international
similarly engaging in forms of self-censorship).
40 Keith N. Hampton et al., Social Media and the ‘Spiral of Silence, PEW RES. CTR., 4 (Aug.
28,
2014),
http://www.pewinternet.org/files/2014/08/PI_Social-networks-anddebate_082614.pdf (finding, for example, 86% of respondents less willing to discuss NSA
surveillance revelations online, than off); Lee Rainie et al., Americans’ Privacy Strategies PostSnowden,
PEW
RES.
INTERNET
PROJECT,
4
(Mar.
16,
2015),
http://www.pewinternet.org/files/2015/03/PI_AmericansPrivacyStrategies_0316151.pdf
(noting that 25% of those aware of surveillance have “changed the patterns” of their use of
“technological platforms”); Lee Rainie et al., Anonymity, Privacy, and Security Online, PEW
RESEARCH CTR. (Sept. 5, 2013), http://www.pewinternet.org/2013/09/05/anonymityprivacy-and-security-online/.
41 See generally Spyros Kokolakis, Privacy Attitudes and Privacy Behaviour: A Review of Current
Research on the Privacy Paradox Phenomenon, COMPUTERS & SOCIETY 1 (2015),
http://www.sciencedirect.com/science/article/pii/S0167404815001017#bib0215 (providing a
comprehensive explanation and review of “information privacy paradox” literature). See also
Alessandro Acquisti & Ralph Gross, Imagined Communities: Awareness, Information Sharing, and
Privacy on the Facebook, in PROC.6TH WORKSHOP ON PRIVACY ENHANCING TECH.(2006)
(finding that Facebook user attitudes concerning privacy differed from their actual behavior
and privacy practices on the platform); J. Alessandro Acquisti, Privacy in Electronic Commerce and
the Economics of Immediate Gratification, PROC.5TH ACM CONF. ON ELECTRONIC COM. (2004),
https://www.heinz.cmu.edu/~acquisti/papers/privacy-gratification.pdf; Acquisti et al., supra
note 18 (advancing, among other things, explanations for the disconnect between privacy

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

11

reasons for this paradox remain contested. Some attribute this disconnect
between privacy concerns and actual behavior to uninformed decisions, while
others point to faulty research and survey design. But few disagree that
compared to how they actually act, people tend to exaggerate privacy concerns,
leading to biased or inaccurate results in research that relies primarily on selfreported privacy behaviors.43 In short, though these survey-based studies
provide some helpful empirical foundation for chilling effects, more work
needs to be done to uncover chilling effects in practice.
B.

POST-SNOWDEN: NEW URGENCY, NEW OPPORTUNITIES FOR
RESEARCH

The Snowden disclosures about NSA surveillance provide new
opportunities for chilling effects research. On June 6, 2013, stories in The
Guardian and The Washington Post detailed previously undisclosed information
and leaked classified documents about the surveillance practices of the United
States and other Western governments.44 The leaked documents also suggested
attitudes and the lax or loose approach to privacy in practice); Bettina Berendt, Oliver
Günther, & Sarah Spiekermann, Privacy in E-commerce: Stated References vs. Actual Behavior, 48:4
COMM.ACM
101,
104
(2005),
http://www.wiwi.huberlin.de/professuren/quantitativ/wi/personen/hl/downloads/BGS.pdf (finding Web users
disclosure practices online were inconsistent with “stated privacy preferences”); danah boyd &
Nicole Ellison, Social Network Sites: Definition, History, and Scholarship, 13:1 J. COMPUTERMEDIATED COMM. 210, 222 (2007), http://onlinelibrary.wiley.com/doi/10.1111/j.10836101.2007.00393.x/epdf (noting literature on the privacy paradox); Jim Harper & Solveig
Singleton, With a Grain of Salt: What Consumer Privacy Surveys Don't Tell Us, COMPETITIVE
ENTERPRISE INST. (2001), http://www.cei.org/PDFs/with_a_grain_of_salt.pdf.
42 Susan Barnes, A Privacy Paradox: Social Networking in the United States, 11:3 FIRST
MONDAY (2006), http://firstmonday.org/article/view/1394/1312.
43 See works cited supra note 41 on “privacy paradox” more generally. Alessandro
Acquisti, for example, has argued that the difference can be explained by the fact that people’s
privacy decisions are irrational and based on flawed or incomplete information. See generally
Acquisti et al., supra note 18 (advancing, among other things, explanations for this disconnect
between privacy attitudes and the lax or loose approach to privacy in practice). See also Harper
and Single, supra note 41, who argue that survey designs have been flawed, leading to
exaggerated self-reported concerns.
44 The June 2013 Snowden leaks centered in large part on “PRISM,” a secret mass
electronic surveillance program operated by the NSA, but they also revealed equivalent
programs operated by the United Kingdom and other countries. The original June 6, 2013
stories detailed NSA collection of phone records and the PRISM surveillance program: Barton
Gellman & Laura Poitras, U.S., British Intelligence Mining Data from Nine U.S. Internet Companies in
Broad
Secret
Program,
WASH.
POST
(June
6,
2013),
http://www.washingtonpost.com/investigations/us-intelligence-mining-data-from-nine-usinternet-companies-in-broad-secret-program/2013/06/06/3a0c0da8-cebf-11e2-8845d970ccb04497_story.html; Glenn Greenwald, NSA Collecting Phone Records of Millions of Verizon
(June
6,
2013),
Customers
Daily,
GUARDIAN
http://www.theguardian.com/world/2013/jun/06/nsa-phone-records-verizon-court-order.
For discussion and analysis of subsequent news stories and revelations about other
surveillance practices by the U.S. and other governments, see David Lyon, Surveillance, Snowden,

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

12

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

a range of major technology companies were involved with the PRISM
program.45 The revelations about PRISM were followed by stories in June and
subsequent months covering a vast array of government surveillance practices
and operations, including the monitoring of phone records, e-mails, online
chats, and browser histories.46 The revelations caused a “media and political
storm,” receiving widespread coverage both in traditional and new media
outlets, and sparking a “heated international debate” in the United States,
Europe, Russia, and beyond.47
Governments cited the “War on Terror” to defend the surveillance
programs and this justification was reflected in media coverage of the
Snowden revelations, particularly by “legacy” news media.48 The Snowden
leaks and coverage, as media scholar Vian Bakir notes, highlighted the
previously limited public awareness about government surveillance activities
while also augmenting that awareness.49 Indeed, at least in the United States,
the widespread media coverage has led to greater awareness and concern
among the general public about government surveillance activities and antiterrorism efforts more generally. A Pew study in 2014 found that 77% of U.S.
adults had heard something about “the government collecting information
about telephone calls, e-mails, and other online communications” as part of
and Big Data:Capacities, Consequences, Critique, 1 BIG DATA & SOC’Y 1, 2 (2014); Marthews &
Tucker, supra note 25, at 5. See also Amy Wu et al., “Whistleblower or Leaker?” Examining the
Portrayal and Characterization of Edward Snowden in USA, UK, and HK Posts, in NEW MEDIA,
KNOWLEDGE PRAC. & MULTILITERACIES (Will W.K. Ma et al., eds.,
2014),
http://link.springer.com/chapter/10.1007/978-981-287-209-8_6; Vian Bakir, Agenda Building,
and Intelligence Agencies: A Systematic Review of the Field from the Discipline of Journalism, Media, and
Communications,
20:2
INT’L
J.
PRESS/POL.
131
(2015),
http://hij.sagepub.com/content/20/2/131.abstract; Keir Giles & Kim Hartmann, Sociopolitical Effects of Active Cyber Defence Measures, 6TH INT’L CONF. ON CYBER CONFLICT (CYCON
2014)
(2014),http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6916393&sor
tType%3Dasc_p_Sequence%26filter%3DAND%28p_IS_Number%3A6916383%29; Jie Qin,
Hero on Twitter, Traitor on News How Social Media and Legacy News Frame Snowden, 20:2 INT’L J.
PRESS/POL. 166 (2015), http://hij.sagepub.com/content/20/2/166.abstract.
45 Bakir, supra note 44,at 132; Lyon, supra note 44, at 2-3; Marthews & Tucker, supra
note 25, at 5-6; Qin, supra note 44,at 171.
46 Lyon, supra note 44, at 2-3; Marthews & Tucker, supra note 25, at 5-6.
47 Bakir, supra note 44, at 132; Giles & Hartmann, supra note 44, at 24; Marthews &
Tucker, supra note 25; Qin, supra note 44,at 166.
48 Bakir, supra note 44, at 133 (“Governments insist that their methods are legal, if
secret, and necessary to fight the War on Terror and organized crime”); Marthews & Tucker,
supra note 25, at 2-6; Nickel, supra note 18, at 255 (“The mentioned surveillance programs have
always – if ‘revealed’ or publicly debated from start with – been justified by their assumed
worth in preventing terror attacks, e.g. by Obama after the NSA’s PRISM program was
revealed.”); Qin, supra note 44, at 178 (finding that a predominant “framing” in traditional
news media coverage of the Snowden surveillance disclosures focused on national security
terrorism, along with international relations).
49 Bakir, supra note 44, at 133.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

13

“efforts to monitor terrorist activity” (with 43% hearing “a lot” and 44%
hearing “a little”); another 80% agreed or strongly agreed that “Americans
should be concerned” about government surveillance.50 This increased
awareness of online government surveillance—and the focal point provided by
the June 2013 revelations—presents a unique opportunity for research.
C.

PRISM/NSA REVELATIONS: A REFERENCE POINT FOR STUDY

The NSA/PRISM surveillance revelations in June 2013 (hereinafter “June
2013 revelations”) and widespread surrounding publicity constituted a kind of
“exogenous shock”—an intervening “focusing event”—that provides a helpful
reference point for study.51 In policy research, the “most prominent” empirical
studies of policy change often focus on the impact of certain such “triggering”
events, typically involving significant “unplanned jolts” or “shocks” like
natural disasters or major economic changes.52 But studies have even shown
that changes in the “tone of media coverage” can create exogenous pressures
that lead to important policy and behavioral changes.53 To understand the
impact of these focusing events, an event is taken as a reference point for
study and observable data before and after the date that the event took place is
compared.54
A recent study on Google search traffic by MIT researchers Alex
Marthews and Catherine Tucker used such a framework to provide an
important contribution to chilling effect research. Their innovative research
design treated the June 2013 revelations about government surveillance as an
50 Mary Madden, Public Perceptions of Privacy and Security in the Post-Snowden Era, PEW RES.
INTERNET
PROJECT
at
2-3
(Nov.
12,
2014),
http://www.pewinternet.org/files/2014/11/PI_PublicPerceptionsofPrivacy_111214.pdf.
51 Graeme Boushey, Punctuated Equilibrium Theory and the Diffusion of Innovations, 40 POL’Y
STUD. J. 127, 130 (2012).
52 William Lowry, Potential Focusing Projects and Policy Change, 34:3 POL’Y STUD. J. 313, 313315 (2006) (discussing research analyzing the impact on policy (and other social and political
factors) caused by focusing or intervening events). See also Boushey, id. at 130 (discussing the
use of focusing events as reference points for policy change studies). See generally FRANK R.
BAUMGARTNER & BRYAN D. JONES, AGENDAS AND INSTABILITY IN AMERICAN POLITICS
(2010) (arguing that dramatic policy shifts can, in part, be attributed to important
triggering/focusing events); THOMAS A. BIRKLAND, AFTER DISASTER: AGENDA SETTING,
PUBLIC POLICY, AND FOCUSING EVENTS 30-35 (1997) (setting out a framework for studying
the impact of “focusing events” on policy changes, including the important role, and impact,
of news coverage of such focusing events); PAUL A. SABATIER & HANK C. JENKINS-SMITH,
THE ADVOCACY COALITION FRAMEWORK: ASSESSMENT, REVISIONS, AND IMPLICATIONS FOR
SCHOLARS AND PRACTITIONERS (1993) (examining, more generally, the role and impact of
external events on policy shifts).
53 See generally BAUMGARTNER & JONES, id. at 39 (generally asserting that dramatic policy
shifts can, in part, be attributed to important intervening/triggering/focusing events).
54 Boushey, supra note 51, at 130 (discussing investigating focusing events, and their
impact, can help understand policy shifts and other changes over time).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

14

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

exogenous focusing event, and tracked the relative number of searches for
certain privacy-sensitive search terms before and after June 2013.55 Marthews
and Tucker found a statistically significant 5% reduction in Google searchers
for certain privacy-sensitive search terms after June 2013.56 Their study not
only provides evidence of chilling effects, but also offers a research design that
may be employed to study chilling effects in other online contexts.
However, the study had its limitations. First, the dataset in the Google
study only included search term data though mid-December 2013. Without
more recent data, it is unclear whether the effects tracked in the study had a
permanent, or at least longer term, impact. Second, the authors obtained their
data from Google Trends, which provides Google search data in “normalized”
or adjusted format.57 The search data is normalized in two ways. First, the data
represents only a percentage of total Google searches for any given term.58
Second, Google then “adjusts” the search data to render comparisons across
regions more easily; these results are further “scaled to a range of 0 to 100”.59
This, the authors admitted, meant it was “harder to make projections” based
on the findings of the study— such as resulting “economic outcomes”60— as
without raw and unadjusted search data, it is difficult to measure, on a more
granular and realistic level, how peoples’ Google search activities were truly
impacted and the resulting economic impact (e.g., Google earns revenue
through search-related ads, so a reduction in Google search inquiries may
affect revenue and profit). A third limitation of the study was the lack of a
genuine control group. The study examined trends before and after June 2013,
but the there was no opportunity to control the PRISM/NSA revelations like a
true experimental intervention.61 A fourth limitation was the possibility that
users were still searching for the same search terms but simply using an

55 Marthews & Tucker, supra note 25, at 5-9.
56 Id. at 3.
57 Marthews & Tucker, supra note 25, at 8.
58
Google
Trends
Help,
Where
Trends
Data
Comes
From,
https://support.google.com/trends/answer/4355213?hl=en&ref_topic=4365599 (“Google
Trends analyzes a percentage of Google web searches to figure out how many searches were
done over a certain period of time. For example, if you search for tea in Scotland in March of
2007, Trends analyzes a percentage of all searches for tea within the same time and location
parameters.”).
59
Google
Trends
Help,
How
Trends
Data
is
Adjusted,
https://support.google.com/trends/answer/4365533?hl=en&ref_topic=4365599 (“Google
Trends adjusts search data to make comparisons between terms easier. Otherwise, places with
the most search volume would always be ranked highest. To do this, each data point is divided
by the total searches of the geography and time range it represents, to compare relative
popularity. The resulting numbers are then scaled to a range of 0 to 100.”).
60 Marthews & Tucker, supra note 25, at 8.
61 The authors note that they only have “quasi” controls: Id. at 6.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

15

alternative search engine to Google (presumably one not expressly linked to
the NSA’s PRISM program).62
Despite these limits, however, the Marthews and Tucker did provide
evidence of chilling effects in a concrete online context—search, though their
findings are not unassailable. Sören Preibusch’s more recent study employed
Marthews and Tucker’s design, primarily examining Bing search term trends
and Tor usage data (as a proxy for users engaging in “privacy-enhancing”
activities as Tor is browser designed to protect privacy and anonymity online)
pre and post Snowden.63 He found that while users’ behavior did change
immediately after the June 2013 revelations, those privacy behaviors “faded
quickly”.64 Like Marthews and Tucker, Preibusch acknowledged important
limitations— for example, his noted his use of Bing data likely “biased” his
results and that his selection of data sources was “partly pragmatic” in this
sense.65 And also like Marthews and Tucker, his data set was temporally
limited— only extending from May 2013 to January 2014.66 In a slightly
different but related 2013 study, Yoan Hermstrüwer and Stephan Dickert
found little evidence of significant chilling effects associated with privacy and
reputational risks of embarrassing online disclosures, leading them to conclude
that “dystopian” concerns often expressed by privacy scholars, about chilling
effects and the conforming impact surveillance, to be overstated.67 Again, the
researchers acknowledged a number of important “caveats” to their findings,
most notably, were likely biased due to self-selection by participants who had
already bound themselves to conforming behavior through informed study to
be involved.68

62 Id. at 6-8.
63 Sören Preibusch, Privacy Behaviors After Snowden, 58:5 COMMUNICATIONS OF THE ACM
48-52 (MAY 2015).
64 Id. at 48, 55.
65 Id. at 55 (“My analysis of Web search behavior through Microsoft’s Bing search
engine may have introduced a bias impossible to quantify, should it exist.”).
66 Id. at 48.
67 Yoan Hermstrüwer & Stephan Dickert, Tearing the Veil of Privacy Law: An Experiment on
Chilling Effects and the Right to be Forgotten 22-23 (Preprints of the Max Planck Institute for
Research
on
Collective
Goods,
No.
2013/15,
2013),
http://www.econstor.eu/bitstream/10419/84983/1/757205445.pdf (an experimental study
on chilling effects finding that risks of “networked publicity” (exposure online of users’
embarrassing activities) did not affect users’ “privacy valuations” nor “dampen” either
“behavioral idiosyncrasies” or the “panoply of different behaviors” involved in the study; they
conclude that concerns about surveillance related chilling and conforming effects may be
“overstated”).
68 Id. at 25 (“A second critique may be that, in our setting, networked publicity is a
function of an endogenous choice, making causal inferences about the factors driving social
norm compliance more difficult. Individuals may have self-selected into networked publicity
because of their stronger inclination to comply with social norms.”).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

16

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

The study in this Article builds on the Marthews and Tucker pre/post
Snowden design, but also aims to address some of the aforementioned
limitations in these related studies. For example, the dataset employed will
include data that starts earlier (January 2012) and extends later (August 2014),
which is far more recent than data employed by Marthews and Tucker or
Preibusch. Furthermore, the Wikipedia article traffic data employed is raw and
unadjusted, providing a more accurate and granular understanding of any
observed changes in data trends. To document how government surveillance
has affected user behavior online, the case study’s interrupted time series
research design approaches the June 2013 revelations as the interrupting
“exogenous shock” or “focusing event,” and examines whether Wikipedia
article traffic for certain topics (that reasonably raise privacy concerns for
Internet users) decreased following those revelations.
D.

HYPOTHESIS

This case study asks: Does the Wikipedia article traffic, for the privacy
concerning topics tracked, decrease after the “exogenous shock” of
widespread publicity surrounding the June 2013 revelations? A hypothesis
based on chilling effects theory may be stated this way: Due to chilling effects
caused by increased awareness of government surveillance online, Internet
users will be less likely to view Wikipedia articles on topics that raise privacyrelated concerns. In ultimately providing compelling evidence suggesting a
NSA/PRISM surveillance related chilling effect, this study is among the first to
do so using web traffic data (instead of survey responses or search) and the
first to evidence the impact of surveillance chill not only on Wikipedia users
but how people seek, and access, information and knowledge online more
generally. The next section sets out this case study’s research design and
methodology, including its focus on Wikipedia.
III.

METHOD AND DESIGN

A.

WHY WIKIPEDIA TRAFFIC?

This case study focuses on English Wikipedia (i.e. articles with content in
the English language) and traffic to specific Wikipedia articles as a means of
exploring chilling effects online. Why Wikipedia? First, despite some
skepticism as to its accuracy, Wikipedia is an influential resource for
information and knowledge online. Over 50% of Internet users use Wikipedia
as a source of information.69 Over a third of Americans visit Wikipedia
69 Lee Rainie et al., Wikipedia, Past and Present, PEW INTERNET & AM. LIFE PROJECT
SURV.
(Jan.
13,
2013),
http://www.pewinternet.org/files/oldmedia/Files/Reports/2011/PIP_Wikipedia.pdf.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

17

annually making it one of the top ten most popular sites on the Internet, and
in a study of college students, researchers Alison Head and Michael Eisenberg
found 52% used Wikipedia “frequently.”70 In other words, if government
surveillance is impacting or “chilling” users from accessing Wikipedia, then
there are implications far beyond Wikipedia’s function as an online
encyclopedia. Indeed, researchers have used Wikipedia for a broad range of
research, relating to both online and offline concerns, including theorizing and
understanding peer-production,71 mapping online knowledge and patterns of
local knowledge production,72 and investigating the subtle ways that popular
information platforms like Wikipedia influence far more than just students or
researchers seeking knowledge online.73 These works all illustrate Wikipedia’s
importance beyond being a basic source of information, so a chilling effect on
Wikipedia users would also threaten or negatively impact these other
important uses and contributions of the site— if people were “chilled” or
deterred en masse from using Wikipedia over time, it could no longer be used
as an important focal point for such research.
Second, there is existing research suggesting media coverage can impact
Wikipedia use. Research has shown not only that Wikipedia is illustrated and
explored how media coverage and “breaking news events” impact Wikipedia
editors and other collaborations on article content.74 If Wikipedia editors and
70 Alison J. Head & Michael B. Eisenberg, How Today’s College Students Use Wikipedia For
Course-Related
Research,
15:3
FIRST
MONDAY
(2010),
http://firstmonday.org/article/view/2830/2476; Hilles, supra note 3, at 247.
71 See, e.g., YOCHAI BENKLER, THE WEALTH OF NETWORKS: HOW SOCIAL
PRODUCTION TRANSFORMS 70-74, 101-123, 287-294 (2006) (discussing Wikipedia in relation
to a range of topics, including “networked information economy,” the “economics of social
production,” and the nature of Internet culture); Yann Algan, Yochai Benkler, Mayo Fuster
Morell, & Jérôme Hergueux, Cooperation in a Peer Production Economy Experimental Evidence from
Wikipedia,
WORKSHOP
ON
INFO.
SYS.
&
ECON.(2013),
https://www.parisschoolofeconomics.eu/IMG/pdf/hergueux_paper-2.pdf (using Wikipedia
to study the social foundations of peer contributions and production).
72 See, e.g., Mark Graham, Bernie Hogan, & Ralph K. Straumann, Uneven Geographies of
User-Generated Information: Patterns of Increasing Informational Poverty, 104:4 ANNALS ASS’N AM.
GEOGRAPHERS 746 (2014) (using Wikipedia to map patterns of global knowledge and
information production).
73 See, e.g., Mark Graham, Internet Geographies: Data Shadows and Digital Divisions of Labour,
in SOCIETY AND THE INTERNET: HOW NETWORKS OF INFORMATION AND COMMUNICATION
ARE CHANGING OUR LIVES 99-116 (Mark Graham & William H. Dutton, eds., 2014) (using
Wikipedia to understand a “digital division” of labor in global information production); Mark
Graham, The Wikimedia Foundation and the Self-Governing Wikipedia Community: A Dynamic
Relationship Under Constant Negotiation, in CRITICAL POINT OF VIEW: A WIKIPEDIA READER
(Geert Lovink & Nathaniel Tkacz, eds., 2011) (discussing elements of Wikipedia’s governance
structure, including how it is vulnerable to “chilling effects” and other regulatory problems).
74 Brian Keegan, A History of Newswork on Wikipedia, PROCEEDINGS OF THE 9TH
INTERNATIONAL SYMPOSIUM ON OPEN COLLABORATION, ACM: NEW YORK (2013) (noting,
among other things, how Wikipedia becomes a focal point for information seekers during

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

18

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

contributors respond to media coverage and significant news events, and
impact the site and its users, it is reasonable to predict that the surveillancerelated revelations may likewise have an impact or chilling effect as on
Wikipedia users). In short, there is an existing empirical foundation that media
coverage of an important story like government surveillance could impact
Wikipedia and its users.
There are also methodological reasons for this case study’s focus on
Wikipedia. First, unlike Google Trends, Wikimedia Foundation provides a
wealth of data on key elements of its site, including article traffic data, which
can provide a more accurate picture as to any impact or chilling effects
identified.75 Second, Wikipedia, a “unique, online, collaborative
encyclopedia,”76 has over 500 million visitors per month, and its collaborative
and peer-produced content is growing at a rate of 17,800 articles per day (as of
May 2014, English Wikipedia content includes over 4.6 million articles).77 In
other words, Wikipedia is both a massively popular medium, but one that is
also growing in content and scope. As such, any observed chilling effect
would implicate a large number of Internet users (accessing Wikipedia) doing
something wholly legal—accessing information and knowledge in an
encyclopedia—and, arguably, such chilled or reduced use would run counter to
these Wikipedia use and content trends.
Finally, the public policy impact of any observed Wikipedia chilling effects
is also a consideration.
Investigating “chilling effect” claims related to
Wikipedia use has recently become a matter of important public interest, in
light of the Wikimedia Foundation lawsuit alleging NSA surveillance has had a

breaking news events, both as a source of information and to understand the event and share
information about it); Brian Keegan, Emergent Social Roles in Wikipedia’s Breaking News
Collaborations, in ROLES, TRUST, AND REPUTATION IN SOCIAL MEDIA KNOWLEDGE MARKETS
57-79 (Elisa Bertino, Sorin Adam Matei, eds., 2015) (reviewing literature exploring how
Wikipedia covers news events, and the impact those events have on Wikipedia collaborative
infrastructure and networks. This article also provides a brief overview of research examining
the nature and structure of Wikipedia editor networks more generally); Brian C. Keegan,
Darren Gergle , & Noshir Contractor, Hot Off the Wiki: Structures and Dynamics of Wikipedia’s
Coverage of Breaking News Events, 57(2) AMERICAN BEHAVIORAL SCIENTIST 595 (2013),
http://abs.sagepub.com/content/57/5/595 (analyzes Wikipedia revision history data to
explore the nature and structure of Wikipedia collaborative efforts in relation to breaking news
events).
75 Wikipedia provides a wealth of information about its number of articles, editors, page
views,
etc,
see
Growth
Per
Wikipedia
Wiki,
https://stats.wikimedia.org/wikimedia/animations/growth/AnimationProjectsGrowthWp.ht
ml(last visited Nov. 17, 2015).
76 See Bar-Ilan & Aharony, supra note 3, at 243.
77 McIver & Brownstein, supra note 3, at 1; Hilles, supra note 3, at 245.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

19

chilling effect on Wikipedia and its users.78 This case study will test and explore
these claims.
B.

RESEARCH DESIGN AND DATA SELECTION

This case study uses an interrupted time series (ITS) design.79 An ITS
design uses a time series, which is a series of measurements or observations
over time that is “interrupted” by some intervention or exogenous event. Such
intervention divides the time series into two segments, resulting in
measurements of time series before and after the intervening event. By
“comparing” patterns in the time series data before and after the interruption,
the study can assess the impact of an interrupting intervention or focusing
event.80 Specifically, this study will compare patterns in the data before and
after the June 2013 revelations.
Furthermore, this study combines segmented regression analysis with its
ITS design. Such combination offers a powerful means for exploring the
effects of interventions, events, or policy changes as long as there is a “clearly
identified time point of intervention.”81 In this case, the identified time point
of intervention is June 2013 surveillance revelations. Segmented regression is
useful because it allows one to compare changes in data levels and trends (like
a reduction in views of Wikipedia articles over time) before and after an
intervening event, while helping to isolate the impact of that event by
controlling for other factors and variables.
Because of its capacity to visualize observed and analyzed data in a
compelling way, ITS design has been applied to a range of fields82 and is

78 See Wales & Tretikov, supra note 3. See generally Complaint, supra note 1 (Wikimedia
Foundation is the lead complainant and, for the record, the only complainant in the lawsuit
that provides accessible data that can be analyzed for the purposes of this study).
79 See supra note 27 (see works cited on ITS design).
80 CAMPBELL & STANLEY, supra note 27, at 37; Fok, Henry, & Allen, supra note 27, at 4;
Mark & Reichardt, supra note 27, at 354-355; Penfold & Zhang, supra note 27, at S39S40;Wagner et al., supra note 27, at 299.
81 Fok, Henry, & Allen, supra note 27, at 10 (“The [ITS design] is especially useful when
there is a clearly identified time point of intervention or policy change.”) [Fix pin cite; no pg 10
for this article]; Mark & Reichardt, supra note 27. at 354-355, 383 (“[I]nterrupted time-series
designs can be among the most credible quasi-experimental designs.”); Penfold & Zhang, supra
note 27, at S38 (stating that ITS design is among the “strongest” where randomized and
controlled experiments are not possible); Wagner et al., supra note 27, at 299 (“Interrupted
time series [with segmented regression analysis] is the strongest, quasi-experimental design to
evaluate longitudinal effects of such time-delimited interventions.”).
82 Mylene Lagarde, How to do (or not to do)… Assessing the Impact of a Policy Change with
Routine Longitudinal Data, 27 HEALTH POL’Y AND PLAN. 76, 76 (2011) (describing how “quasiexperimental” ITS designs, employing segmented regression, has been used in “various fields”
including environmental studies, “financial economics,” and “health policies”),
http://heapol.oxfordjournals.org/content/27/1/76.full.pdf.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

20

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

particularly popular among policy researchers.83 It has also been used to
explore the effects of laws, policing (including surveillance by law
enforcement), and other regulatory actions.84
This case study uses data on English language Wikipedia article view
counts from the online service stats.grok.se, a portal maintained by a
Wikimedia Foundation member. This portal provides access to a range of
Wikipedia analytics, stats, and data.85 In particular, the portal aggregates
Wikipedia article view data on a daily and monthly basis.86 This data at
stats.grok.se has been used in a range of research, including studies involving
market trends, health information access, and social-political change.87
83 See, e.g., Benjamin French & Patrick J. Heagerty, Analysis of Longitudinal Data to Evaluate
a Policy Change, 27:24 STAT. MEDICINE 5005 (2008) (surveying different research designs and
methods in policy change research); Lagarde, supra note 82, at 76 (describing in detail an ITS
design that employs segmented regression as a simple but robust method to study policy
impact and change); Wagner et al., supra note 27.
84 See, e.g., Samuel Cameron, The Economics of Crime Deterrence: A Survey of Theory and
Evidence, 41:2 KYKLOS 301, 314 (1988) (noting economists had begun using ITS designs, which
then had long been used by criminologists); Daniel S. Nagin, Criminal Deterrence Research at the
Outset of the Twenty-First Century, 23 CRIME AND JUST. 1, 8-12 (1998) (discussing a range of laws
and police operations that have been using ITS designs); Lynn W. Phillips & Bobby J. Calder,
Evaluating Consumer Protection Laws: II. Promising Methods, 14:1 J. CONSUMER AFFAIRS 9 (1980)
(surveying literature on methods/research designs used to study consumer protection laws,
including ITS). For some more recent examples, see Carl Bonander, Finn Nilson, & Ragnar
Andersson, The Effect of the Swedish Bicycle Helmet Law for Children: An Interrupted Time Series Study,
51 J. SAFETY RES. 15 (2014) (used ITS design to explore the impact of a bicycle helmet law by
examining inpatient data on injured cyclists before and after the law was enacted); Becky
Briesacher, et al., A Critical Review of Methods to Evaluate the Impact of FDA Regulatory Actions, 22:9
PHARMACOEPIDEMIOLOGY AND DRUG SAFETY 986 (2013) (reviews a range of ITS design
studies examining the impact of FDA regulatory actions often by exploring health data before
and after the FDA action); Benjamin David Décary Hétu, Police Operations 3.0: On the Impact and
Policy Implications of Police Operations on the Warez Scene, 6:3 POL’Y & INTERNET 315 (2014)
(exploring the impact of police operations and crackdown on the “warez” (online piracy) scene
with an ITS design that examined data on the output of different warez communities before
and after five different police operations); Jeffrey T. Ward, Matt R. Nobles, Lonn LanzaKaduce, Lora M. Levett, & Rob Tillyer, Caught in Their Own Speed Trap: The Intersection of Speed
Enforcement Policy, Police Legitimacy, and Decision Acceptance, 14:3 POLICE QUARTERLY 251
(2011)(using an ITS design to study the impact certain policy changes have on public opinion
concerning the legitimacy police action, specifically, comparing speeding citation contestation
rates before and after the introduction of an advertising campaign labeling the intervention city
a “speed trap”).
85 Stats.grok.se is maintained by Domas Mituzas, a Wikipedia developer, past Board of
Trustee on the Wikimedia Foundation, and present member of its Advisory Board, see
Frequent Questions, GROK. http://stats.grok.se/about (last visited Nov. 17, 2015).
86 Id.
87 See, e.g., Michela Ferron & Paolo Massa, WikiRevolutions: Wikipedia as a Lens for Studying
the Real-Time Formation of Collective Memories of Revolutions, 5 INT’L J. COMM. 1313 (2011)
(examining Wikipedia as a “lens” through which to understand real-time social and political
upheaval and change); Michaël R. Laurent & Tim J. Vickers, Seeking Health Information Online:

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

21

Like the Marthews and Tucker study, this case study uses a list of
keywords the U.S. Department of Homeland Security use to track and monitor
social media.88 This list categorizes certain search terms in relation to a range
of different issues such as “Health Concern,” “Infrastructure Security,” and
“Terrorism.” According to the DHS documents themselves, the list is meant
to assist analysts to monitor social media to provide “situational awareness and
establish a common operating picture.”89 Though the methodology for
formulating the list is not well known, presumably they represent ideas or
content people associate with “terrorism” and similar national security matters.
Because of this, they are items that government officials may be interested in
tracking online.90
Using governments’ own keyword lists (for online monitoring purposes) to
study government surveillance or censorship is not new.91 Here, the DHS
keywords provide a helpful basis of selecting Wikipedia articles for the study.
To be clear, this keyword list is non-random and it is not chosen based on any
assumption that the general public is aware of the list or the topics attached. In
other words, this study does not assume that people are avoiding topics
relating to these keywords due to the DHS’ media monitoring program.
Rather, the list is simply used as a means of methodological pragmatism.
Similar to how the list was used in the MIT Google search trends study,92 the
DHS documents, and the keywords therein, are used to select Wikipedia
Does Wikipedia Matter? 16:4 J. AM. MED. INFORMATICS ASS’N 471 (2009)(using Wikipedia traffic
data from stats.grok.se to study the relevance of Wikipedia to how people access to health
information online); Helen Susannah Moat et al., Quantifying Wikipedia Usage Patterns Before Stock
Market Moves, 3 SCI. REP. 1 (2013) (investigating Wikipedia article traffic and usage in relation
to stock market changes).
88 The keyword list has been publicly available online since 2012, and was updated and
re-posted by the DHS in 2013: U.S. DEP’T OF HOMELAND SEC., NATIONAL OPERATIONS
CENTER MEDIA MONITORING CAPABILITY ANALYST’S DESKTOP BINDER (2011) [hereinafter
ANALYST’S DESKTOP BINDER], https://epic.org/foia/epic-v-dhs-media-monitoring/AnalystDesktop-Binder-REDACTED.pdf. This was later updated and posted online by the DHS. See
U.S. DEP’T OF HOMELAND SEC., PRIVACY IMPACT ASSESSMENT FOR THE OFFICE OF
OPERATIONS COORDINATION AND PLANNING (2013)[hereinafter PRIVACY IMPACT
ASSESSMENT],
https://www.dhs.gov/sites/default/files/publications/privacy/PIAs/privacy_pia_ops_NOC
%20MMC%20Update_April2013.pdf
89 PRIVACY IMPACT ASSESSMENT (2013), supra note 88, at 23 (Appendix B); Marthews &
Tucker, supra note 25, at 3-4.
90 Marthews & Tucker, supra note 25, at 6.
91 Jedidiah R. Crandall & Masashi Crete-Nishihata et al., Chat Program Censorship and
Surveillance in China: Tracking TOM-Skype and Sina UC, 18:7 FIRST MONDAY (2013),
http://firstmonday.org/ojs/index.php/fm/article/view/4628/3727; Jeffrey Knockel, Jedidah
Crandall, & Jared Saia, Three Researchers, Five Conjectures: An Empirical Analysis of Tom-Skype
Censorship and Surveillance, 16:4 FOCI’11: USENIX WORKSHOP ON FREE AND OPEN COMM.
ON INTERNET (2011), https://www.cs.unm.edu/~crandall/foci11knockel.pdf.
92 Marthews & Tucker, supra note 25, at 3-4.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

22

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

articles that reasonably represent the sort of articles that users may be chilled
from accessing in light of government surveillance.93
This case study selected forty-eight Wikipedia articles that corresponded
with the DHS keywords listed as relating to “terrorism.”94 The full list of the
keywords used— which include terms such as “dirty bomb,” “suicide attack”,
“nuclear enrichment”, and “eco-terrorism”— and the corresponding English
language Wikipedia articles for which Wikipedia article “page view” counts
were collected via stats.grok.se can be found at Appendix D.95
The keywords relating to “terrorism” were used to select the Wikipedia
articles because the U.S. government cited terrorism as a key justification for
PRISM and NSA surveillance. Moreover, much of the media and news
coverage framed the revelations around terrorism and national security.96
Wikipedia articles coinciding with these terrorism-related topic keywords may
include the kind of information or content users may avoid accessing in light
of potential government surveillance. This study aggregated Wikipedia article
traffic (or view counts) on a monthly basis for the forty-eight Wikipedia
articles over a 32 month period, from the beginning of January 2012 to the end
of August 2014. Those forty-eight Wikipedia articles corresponded with all
DHS keywords listed in the “terrorism” category.97

93 For example, if the government surveillance is focusing on terrorism online, people
may have privacy concerns about accessing terrorism-related information online, and are thus
“chilled” or deterred from accessing.
94 PRIVACY IMPACT ASSESSMENT (2013), supra note 88, at 27. Locating Wikipedia
articles coinciding with each keyword was done manually; this was a rather simple exercise as
there was a Wikipedia article that corresponded perfectly with the vast majority of keywords in
the “terrorism” DHS keyword category. The few discrepancies were these: the Wikipedia
article “environmental terrorism” was used for the keyword “environmental terrorist”; the
keyword “target” was excluded as they were too many potentially corresponding Wikipedia
articles; the Wikipedia article “political radicalism” was used for the DHS keyword
“radicalism” (there were too many potentially corresponding Wikipedia articles); the keyword
“enriched” was excluded as it was redundant with the included Wikipedia article / DHS
keyword “nuclear enrichment”; there were also no Wikipedia articles corresponding with DHS
keywords “weapons cache,” “suspicious substance,” “plot,” and “homegrown.” Wikipedia
articles corresponding with the remaining 48 DHS “terrorism” related keywords were all
included in the study.
95 For clarity, the raw Wikipedia article “Page View” statistics track total views or loads
of the Wikipedia articles or pages in question, not unique visitors, see PageviewStatistics,
Wikipedia, https://en.wikipedia.org/wiki/Wikipedia:Pageview_statistics (last updated July 8,
2015).
96 Qin, supra note 44, at 178 (finding that a predominant “framing” in traditional news
media coverage of the Snowden surveillance disclosures focused on national security
terrorism, along with international relations).
97 PRIVACY IMPACT ASSESSMENT (2013), supra note 88, at 27. The findings in this study
primarily concern the English speaking world, as only English Wikipedia article view counts
are tracked in the Wikipedia data. Locating Wikipedia articles coinciding with each keyword

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

23

Although forty-eight is not an extraordinarily large sample size, the
Wikipedia traffic attracted by these articles represents over 81 million total
article page views over the course of the study. This means that the potential
number of Internet users tracked in the study could be in several millions.98
Moreover, to ensure the sample of forty-eight articles could be generalized to a
wider sample of content (both terrorism-related and other topics that may
raise privacy concerns), the study used crowdsourcing following the approach
of Marthews and Tucker.99
This study used crowdsourcing to have independent respondents
individually evaluate the privacy value of the topics in question.100
Crowdsourcing involves completing certain tasks with the assistance of larger
pools of online users or “crowds”—recruited through online services like
Amazon’s Mechnical Turk (MTurk)—and has become a common technique
for researchers to evaluate research instruments and other measures for
privacy or privacy-related concerns.101 MTurk is an “open” online
crowdsourcing platform founded in 2005 that provides a means for “task
creation”, “recruitment”, “compensation and data collection.”102 Several
studies summarized and documented MTurk’s advantages for survey,
was done manually; the vast majority of keywords in the “terrorism” DHS category had a
coinciding Wikipedia article.
98 Thus, though the selection of the forty-eight English Wikipedia articles was not
random (there is no sampling frame for all terrorism-related Wikipedia articles), the data
clearly indicates these articles represent a substantial number of Wikipedia users. Though a
precise number cannot be estimated (the Wikipedia data tracks “Page View” statistics, that is,
total views or loads of the Wikipedia articles or pages, not unique visitors, see Pageview Statistics,
Wikipedia, https://en.wikipedia.org/wiki/Wikipedia:Pageview_statistics (last updated July 8,
2015). The data arguably still involves a large number of Internet users— many millions.
99 Marthews and Tucker similarly recruited independent “raters” to evaluate the privacy
value of Google search terms in their study: Marthews & Tucker, supra note 25, at 3-5.
100 Id.
101 See, e.g., Berker Agir, Jean-Paul Calbimonte, & Karl Aberer, Semantic and Sensitivity
Aware Location Privacy Protection for the Internet of Things, PRIVACY ONLINE: WORKSHOP ON
SOCIETY, PRIVACY AND SEMANTIC WEB PRIVON 2014 (2014), http://ceur-ws.org/Vol1316/privon2014_paper5.pdf; Margherita Bonetto et al., Privacy in Mini-drone Based Video
Surveillance, WORKSHOP ON DE-IDENTIFICATION FOR PRIVACY PROTECTION MULTIMEDIA
(2015), http://infoscience.epfl.ch/record/206109; Pavel Korshunov et al., Crowdsourcing-based
Evaluation of Privacy in HDR images, SPIE PHOTONICS EUROPE (2014),
http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1873752;
Pavel
Korshunov et al., Framework For Objective Evaluation of Privacy Filters, SPIE OPTICAL
ENGINEERING
+
APPLICATIONS
(2013),
http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1744325; Jialiu Lin et al.,
Expectation and Purpose: Understanding Users' Mental Models of Mobile App Privacy Through
Crowdsourcing, PROC. 2012 ACM CONF. ON UBIQUITOUS COMPUTING (2012),
http://dl.acm.org/citation.cfm?id=2370290.
102 Michael Buhrmester, Tracy Kwang, and Samuel Gosling, Amazon’s Mechanical Turk:
A New Source of Inexpensive, Yet High Quality, Data?, 6:1 PERSP. ON PSYCHOLOGICAL SCI.3, 3
(2011), http://pps.sagepub.com/content/6/1/3.abstract.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

24

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

experimental, and other related research. Paolacci and Chandler recently
concluded, after extensively canvassing existing evidence, that researchers may
use MTurk for “virtually any study that is feasible to conduct online.”103
Indeed, the MTurk service has been validated as a tool for a range of research,
including research on behavioral economics, and decision-making, collective
behavior experiments, linguistic and cognitive psychological experiments, and,
importantly for our purposes, conducting survey research.104 Samples recruited
with MTurk have been found to be “at least as diverse as traditional subject
pools” in terms of the general U.S. population and “relatively representative”
of the U.S. Internet using population.105
A total of 415 independent Internet users participated in the
crowdsourcing project through MTurk, and they rated each of the forty-eight
topics, with which the Wikipedia articles in the data set corresponded. The
questions were designed to explore the likelihood that the topics would raise
privacy-related concerns for Internet users. To minimize self-selection and
response bias (a limitation difficult to avoid in non-random sampling), the
brief questionnaires were described as merely an “Online Information Study”
to potential MTurk participants.
The respondents recruited for the evaluations were similar to other MTurk
participant pools that are “relatively representative of the population of U.S.
Internet users.” However, the respondents for this study were younger, more
educated, had slightly lower incomes than the broader U.S. Internet
population, and slightly more male than female with 56% of respondents male
and 44% female.106 The respondents were also highly likely to use “websites
and other online resources” for information more generally (83.8% were “very
likely” and another 15% “somewhat likely”) and, in particular, to “stay
informed about current events” (73.5% “very likely” and another 22.9%
“somewhat likely”). Respondents were asked to indicate on a scale of 1 to 5 (1
being very unlikely and 5 being very likely): how likely they thought they would
be in trouble if the U.S. Government found out that they accessed information
about the topic in question (Government Trouble Rating); how “privacy103 Gabriele Paolacci & Jesse Chandler, Inside the Turk: Understanding Mechanical Turk as a
Participant Pool, 23:3 CURRENT DIRECTIONS PSYCHOL. SCI. 184, 186 (2014),
http://cdp.sagepub.com/content/23/3/184.abstract.
104 Matthew J.C. Crump, John V. McDonnell, & Todd M. Gureckis, Evaluating Amazon's
Mechanical Turk as a Tool for Experimental Behavioral Research, 8:3 PLOS ONE e57410, e57410
(2013), http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0057410
105 Gabriele Paolacci, Jesse Chandler, & Panagiotis G. Ipeirotis, Running Experiments on
Amazon Mechanical Turk, 5:5 JUDGMENT AND DECISION MAKING 411, 411-412 (2010),
http://sjdm.org/~baron/journal/10/10630a/jdm10630a.pdf; Panagiotis G. Ipeirotis, Turker
Demographics vs. Internet Demographics, COMPUTER SCIENTIST BUS. SCH. (2009),
http://www.behind-the-enemy-lines.com/2009/03/turker-demographics-vs-internet.html.
106 Paolacci, Chandler, & Ipeirotis, id. at 412.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

25

sensitive” they viewed each topic as (in this case, 5 being highly sensitive and 1
not at all) (Privacy-Sensitive Rating); how likely they would be to delete the
browser history on their computer after accessing information about the topic;
and how likely they would avoid viewing or accessing information on the topic
if they knew the Government was monitoring peoples’ activities online
(Avoidance Rating).107
The results are set out in the table at Appendix C. On balance, the results
from the first three categories suggest that the topics raised notable privacy
concerns for respondents. The results do not suggest, however, that any one
category raised overly strong concerns. This is not entirely surprising as there
is no particular reason why someone would expect to be in “trouble” with the
government for simply accessing information online when such access is a
legal activity. The rating scores in response to the fourth category are, however,
noteworthy—things change if people are aware of government surveillance.
This is apparent from the higher “Avoidance” rating of 2.62, which suggests
that respondents were overall more likely to avoid the topics in question if
they knew the government was monitoring online activities. In short, the
ratings suggest that the topics of the forty-eight Wikipedia articles raise privacy
concerns for Internet users, particularly when people suspect the government
is monitoring them, which may lead them to avoid or be chilled from accessing
that information.
C.

METHOD OF ANALYSIS

A strength of ITS design is that there are “multiple” assessments or
measures before and after the event or intervention in the time series; such
multiplicity controls for changes in level and secular trends in the data and
increase the robustness of results.108 This case study uses Wikipedia article
view counts or traffic data to create a time series over a 32-month study period
from January 2012 to August 2014 (n=32), with the “interruption” or
intervening event dividing the time series into two segments: before and after
the June 2013 revelations. Two empirical approaches are used to analyze the
interrupted time series. The first is a simple comparison of the mean number
of views for all the Wikipedia articles in the dataset before and after June 2013.
If there is a chilling effect due to surveillance revelations in June 2013, the
average or mean number of views for the forty-eight Wikipedia articles should
be lower for months following June 2013 than that of the months before. The
107 This is not a standard scale, but one developed for this case study. For
methodological consistency, these questions were designed to track ratings categories used by
Marthews and Tucker: Marthews & Tucker, supra note 25, at 33 (Table 15).
108 CAMPBELL & STANLEY, supra note 27, at 37; Fok, Henry, & Allen, supra note 27, at 7;
Lagarde, supra note 82; Penfold & Zhang, supra note 27, at S39; Wagner et al., supra note 27, at
308.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

26

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

second is a model-based empirical analysis. That is, segmented regression of an
interrupted time series, which is the recommended method of analysis for ITS
designs.109 Health economist Mylene Lagarde recently set out and analyzed
this method comprehensively110 and provided an equation that expresses the
specification for the regression analysis:
Yt = 0 + 1 * time +2 * intervention + 3* postslope + 1111
The ITS design controls for secular trends, or the long-term and nonperiodic trends in the data.112 To strengthen the robustness and validity of
results, however, additional controls can be included in the model, and, where
appropriate, auto-correlation can be corrected.113 Here, although a true control
group was not possible,114 to increase the robustness of the analysis, a further
control variable was added that includes monthly raw aggregate total view
counts for all English Wikipedia articles on all platforms (desktop and
mobile).115 This was done to control for broader use trends in Wikipedia over
the course of the thirty-two months, and isolate the impact that the June 2013
revelations had on the forty-eight Wikipedia articles in the case study.116 Again,
109 Penfold & Zhang, supra note 27, at S41-42; Wagner et al., supra note 27, at 299.
110 Lagarde, supra note 82, at 79-80.
111 In this case study, Yt the “outcome” or dependent variable is the raw aggregate total
of Wikipedia article views (or ‘view count’) for the 48 articles in the study. The time variable
includes 32 time points in the time series, representing each of the 32 months in the time
series dataset from January 2012 to August 2014, which is the period of study. So, in this time
series dataset, the “outcome” or dependent variable is the aggregate views of all forty-eight
Wikipedia articles totaled on a monthly basis, for each of the 32 months. For greater clarity, in
this model, 0 captures the baseline level of the outcome variable at time 0— here, that would
be the expected total views for all 48 articles in the dataset at the beginning of the study; 1
estimates the secular trend or growth rate in the total number of views for the 48 Wikipedia
articles, independently from the “intervention” or intervening event (the June 2013
surveillance revelations); 2 estimates the immediate impact of the “intervention” or the
exogenous shock of PRISM/NSA surveillance publicity in June 2013, by reflecting the change
in the “level” or the total number of views for the Wikipedia articles immediately after the
June 2013 events; and, finally, 3 reflects any change in the trend of the data; that is, any
growth or decline in total views for the 48 Wikipedia articles on a month-to-month basis, after
the intervention: Lagarde, supra note 82, at 79-80.
112 Lagarde, supra note 82, at 79.
113 Lagarde, supra note 82, at 79, 81; Mark & Reichardt, supra note 27, at 385; Penfold &
Zhang, supra note 27, at S42; Wagner et al., supra note 27, at 305.
114 Online surveillance potentially affects everyone, and there was no opportunity,
before the June 2013 revelations to isolate a control group.
115 As previously noted, Wikimedia provides a wealth of information about: Page Views
for
Wikipedia,
Both
Sits,
Raw
Data,
WIKIMEDIA
(Nov.
11,
2015)
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyOriginalCombined.htm
116 The segmented regression analysis was performed using the statistical software
package Stata and auto-correlation is controlled for using the Prais-Winsten method where

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

27

a prediction based on chilling effects theory is that there will be a decrease in
the total views of the forty-eight Wikipedia articles after June 2013. In addition
to an immediate drop, if any chilling effects are more substantial and long-term
the overall long-term trend in the data may also be impacted (e.g., a change
from a gradual increase in the number of article views from month to month
to a monthly reduction in total views).
IV.

NON-MODEL EMPIRICAL FINDINGS

The results discussed in this section are “non-model” empirical findings,
that is, these findings do not rely upon a statistical (regression) model. Instead,
a more basic method of analysis is used whereby the average number of
Wikipedia article views before and after the “focusing event” of June 2013 is
compared. A lesser average number of views after June 2013 would be
consistent with a chilling effect. Here, those mean total article views, for all
Wikipedia articles in the study, for the months before and after June 2013 (see
Figure B).
Figure B: Average monthly view counts, Pre and Post June, 2013
The reduction after the June 2013 surveillance revelations may
suggest a chilling effect

4,000,000

Total Views 

3,500,000
3,000,000
2,500,000

2,775,709
2,249,095

2,000,000
1,500,000
1,000,000
500,000
0
Pre‐June, 2013 Post‐June, 2013

The difference in mean values is notable—a reduction of 526,614 in the
average monthly views for the article after June 2013, which represents
necessary: Lagarde, supra note 82, at 79 (recommending controlling for auto-correlation when
employing this statistical analysis). See similar recommendations in: GEORGE G. JUDGE ET
AL., INTRODUCTION TO THE THEORY AND PRACTICE OF ECONOMETRICS (1985).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

28

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

approximately a 19.5% drop in article view counts. This is more than mean
differences found in the Google search terms study before and after June
2013.117 Moreover, this difference or reduction in average monthly article
views before and after June 2013 is both large and highly statistically significant
at the 99% confidence level.118 This by itself offers some evidence of a chilling
effect. Of course, there are alternative explanations for these results. One
possible explanation may be that overall Wikipedia traffic (and thus, all
Wikipedia article view counts) decreased after June 2013 for other reasons.
Perhaps people are using Wikipedia less and less and this data is simply
reflecting this overall declining trend unrelated to any surveillance revelations.
So, while these findings are intriguing, a model-based empirical analysis is
required to control for such variables as the overall trends in the data and to
arrive at more robust empirical results. Statistical regression models, used in
the next section may control for additional variables such as overall trends in
Wikipedia article view traffic.
V.

MODEL-BASED EMPIRICAL FINDINGS

As noted earlier, the “outcome” or dependent variable in this analysis
represents the raw aggregate total view counts per month for all Wikipedia
articles in the data set. The method of a segmented regression analysis of an
interrupted time series data set (“interrupted” by the June2013 PRISM
surveillance revelations) demonstrates the impact of these revelations. Two
sets of results are reported here. Analysis was conducted using Stata statistical
software.119
A.

FIRST SET OF RESULTS

The first set of results is represented in the table at Appendix A.
Interestingly, the results indicate there was a reduction of 853,586 views
immediately following the June 2013 revelations, which is a large, sudden, and
statistically significant drop in the total view counts for the forty-eight
Wikipedia articles.120 The total article views as of May 2013 is 2,960,778 views,
117 But cf., Marthews & Tucker, supra note 25, at 13-14.
118 The Cohen’s d value was 1.3286.
119 Version 11.1.
120 There is little consensus for the appropriate method to measure effect size for single
group ITS designs like the one used for this case study: Larry Hedges, James Pustejovsky, &
William Shadish, A Standardized Mean Difference Effect Size for Single Case Designs, 3:3 RES.
SYNTHESIS METHODS 224, 225 (2012). The most common method in “treatment” studies is
the percentage of non-overlapping data (PND), which here is clearly above the 80% threshold
for a “large” effect size, see generally Thomas E. Scruggs & Margo A. Mastropieri, How to
Summarize Single-Participant Research: Ideas and Applications, 9:4 EXCEPTIONALITY 227 (2001)
(proposing the use of non-overlapping data metric for summarizing single participant
research). However, model diagnostics identified two influential outlier data points, the first

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

29

meaning this decline represents an immediate drop off of almost 30% of total
views overall. However, the results also indicate that there was no statistically
significant change in the secular (or overall long-term) trend in the data. In
short, because of the large drop in total view counts for the forty-eight
Wikipedia articles, the data supports the existence of an immediate and
substantial chilling effect following the June 2013 surveillance revelations. This
decrease may be observed in the following graphic visualization in Figure C,
below, which includes a scatter plot of the data points in the set and a trend
line based on the fitted results produced by the regression analysis:
Figure C: First Results: Pre/Post June 2013 Article View Trends
The sudden drop, and flatter trend or slope in the data, after June
2013 surveillance revelations consistent with a chilling effect

Mid June, 2013

Total Views (All 48 Wikipedia Articles)

4000000

3500000

3000000

2500000

2000000

0

2

4

6

8

10

12

14 16 18
Time (Months)

20

22

24

26

28

30

Confidence Interval (95%)

Trend Pre-June 2013

Trend Post-June 2013

Total Article Views (By Month)

32

In this graph, the large and statistically significant immediate drop
(853,585 page views) over the course of June 2013 can be seen, with the
trend line (regression/line of best fit) substantially lower immediately after
June, 2013, compared to the trend in the data before June. Also, there was no
outlier concerned view counts for the Wikipedia articles in the data set in November 2012
(Cooks D value = 0.1286922), and the other was for view counts in July 2014 (Cooks D value
= 0.3244882). Both of these are extreme values and are visible from the visualization in
Figure C.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

30

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

statistically significant change in the secular trend in the data, as the slopes in
the data before and after June 2013 are comparable: in each, the trend in view
counts for the sample is modestly increasing overall.
Though there was not a statistically significant change in the trend of the
data, the graph still suggests something more than an ephemeral chilling effect
that dissipates quickly. Rather the data suggests a lasting impact on total article
views. For example, the total article views as of August 2014 (month 32) is still
lower than the views in April and May 2013 (months 15 and 16), the months
prior to the PRISM revelations. The results also suggest that the chilling effect
did not influence the long-term trend in the data that increased monthly.
Though the number of views dropped off after June, 2013, the trend in the
data still increased until August 2014 at a modest but apparent rate on a
month-to-month basis.
In sum, the results are consistent with a sharp immediate chilling effect,
possibly with a lasting impact on total views. However, the rising secular trend
in the Wikipedia article traffic is inconsistent with a significant long term
chilling effect.
B.

OUTLIERS: THE ‘EXOGENOUS SHOCK’ OF WAR

Two outliers are clearly apparent in the data. The first is at month 11
(November 2012) and the second is at month 31 (July 2014). The view counts
for the forty-eight Wikipedia articles in the data skyrocketed in these months,
far beyond any other months before and after, clearly out of step with other
data points. In November 2012, the total views dramatically and temporarily
increased to approach almost 4,000,000 total views. In July 2014, total views
exponentially increased far beyond previous or subsequent months. Model
diagnostics confirm that these are highly influential outliers with extreme
values.121
121 Best practices for dealing with outliers in cases like this were observed—the
technique used to identify the “influential” outlier should be indicated (here, Cooks D),
deletion used as a method to address the outlier, and results are reported with and without the
outlier data. See Herman Aguinis, Ryan K. Gottfredson, & Harry Joo, Best-Practice
Recommendations for Defining, Identifying, and Handling Outliers, ORGANIZATIONAL RES. METHODS
8,
20-23
(2014),
http://orm.sagepub.com/content/early/2013/01/11/1094428112470848.abstract (techniques
for identifying outliers should be indicated; Cooks D is noted (at 8 and 21) as an appropriate
“technique” to identify an outlier’s influence globally in a regression (as here); also the authors
state (at 22) that influential outliers can be dealt with through deletion but “emphasize the
importance of reporting the results with and without the chosen handling technique, which
includes providing an explanation for any differences in results because the mere presence of
influential outliers causes a dilemma in determining proper inference about a population based
on a sample...”). These guidelines were, as noted, employed. The Cooks D value of 0.1286922
for the November 2012 view count and 0.3244882 for July 2014, are both extreme values.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

31

What happened during November 2012 and July 2014 that caused the total
view counts in the sample to suddenly skyrocket? War and conflict in the
Middle East are likely the cause. In November 2012, Israel launched
“Operation Pillar of Defense,” an eight day Israeli Defense Force (IDF)
operation in Gaza in response to rocket attacks launched by the Palestinian
militia group, Hamas, into southern Israel.122 The operation officially began on
November 14, 2012 and ended on November 21, 2012, with an Egyptbrokered ceasefire. In July 2014, Israel launched “Operation Protective Edge,”
which is an IDF operation in Gaza against Hamas.123 This operation ended
after fifty days with widespread media coverage of the thousands of rockets
fired from Gaza into Israel, and the likewise several thousand strikes, many by
IDF air force, on Hamas targets in Gaza.124
These two high profile conflicts coincide with a dramatic and anomalous
increase in the view counts during those months for the Wikipedia article on
“Hamas” in the dataset. Examining more closely the view counts for the
Hamas article over the 32 months in the data set, the “Hamas” Wikipedia
article view count was 928,533 for November 2012, and then 1,220,490 for
July 2014, which are far beyond the mean number of view counts for the
article across all months in the study (134,574 monthly views). If we exclude
these two outlier months, the contrast between the view counts for the Hamas
Wikipedia article during those two months and other months in the dataset is
even starker, with the mean being 71,912.125 It can be inferred that the media
122 For a “timeline” to the conflict and the IDF operation against Hamas, see
TIMELINE: Israel Launches Operation Pillar of Defense Amid Gaza Escalation, HAARETZ (Nov. 20,
2012), http://www.haaretz.com/news/diplomacy-defense/timeline-israel-launches-operationpillar-of-defense-amid-gaza-escalation.premium-1.479284. See also Q&A: Israel-Gaza Violence,
BBC NEWS (Nov. 22, 2012), http://www.bbc.com/news/world-middle-east-28439404.
123 Information about the operation is available in this Jerusalem Post article: Ben
Hartman, Fifty Days of Israel's Gaza Operation, Protective Edge – by the Numbers, JERUSALEM POST
(Aug. 28, 2014), http://www.jpost.com/Operation-Protective-Edge/50-days-of-Israels-Gazaoperation-Protective-Edge-by-the-numbers-372574; Gaza Crisis: Toll of Operations in Gaza, BBC
NEWS (Sep. 1, 2014), http://www.bbc.com/news/world-middle-east-20388298.
124 Amos Harel, At the Crossroads of a Gaza Ground Operation, HAARETZ (Jul. 12, 2014),
http://www.haaretz.com/news/diplomacy-defense/.premium-1.604601(“Hamas and Israel
are waging an image battle. Their moves are the subject of constant media coverage, and, more
than in the past, they are using information and photos from civilians, through smart phones
and social media.”). In fact, the widespread media coverage of the Gaza conflict in 2012 led to
a dramatic increase in social media activity during the 2012 conflict: Thomas Zeitzoff, Does
Social Media Influence Conflict? Evidence from the 2012 Gaza Conflict (SSRN, Working Paper 13,
2012),
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2407804
(noting
that
“international interest” in the Israeli-Palestinian conflict “led to multiple, competing news
organizations covering the 2012 Gaza Conflict”).
125 This is confirmed by the z-scores for those two data points (3.01 and 4.11,
respectively). Both are outlier values. See generally PETER H. WESTFALL & KEVIN S. S.
HENNING, UNDERSTANDING ADVANCED STATISTICAL METHODS 247 (2013) (noting the

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

32

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

coverage of these two conflicts involving Israel and Hamas in Gaza led to a
dramatic increase in Internet users seeking information about Hamas on
English Wikipedia. This conclusion is supported by the fact that view counts
for the Wikipedia article “Palestinian Liberation Organization” also increased
in those same months.126 It is also consistent with the findings of Zeitzoff,
Kelly, and Lotan who have found that major conflicts, including the 2012
Gaza conflict, drew “significantly higher levels” of activity on the social media
platform Twitter.127 The two influential outliers relating to the “Hamas”
Wikipedia article are incorporated in the second set of results.
C.

SECOND SET OF RESULTS – A LASTING CHILLING EFFECT?

Consistent with best practices for dealing with outliers,128 results including
the outlier “Hamas” Wikipedia article data were reported above. A second set
of results from the analysis, which excludes the outlier data concerning the
Hamas article, is presented here and set out in the table at Appendix B.
Removing the outliers led to new findings.129 Similar to the first reported
results, there was a sudden and statistically significant decrease in view counts
following the June 2013 revelations: an immediate drop of 540,318 total views.
Using the total article views as of May 2013 (2,960,778), this decrease
represents an immediate drop off of just under 20%. This suggests that the
revelations in June 2013 caused, or at least is consistent with, a chilling effect
or a sharp and sudden decrease in traffic.
More importantly, however, is that after June 2013, there is not only a large
and immediate drop in views but also a statistically significant change in the
overall trend or direction in the month-to-month total views of the Wikipedia
articles. Rather than trending upwards or increasing on a monthly basis, the
trend after June 2013 has completely changed. Due to the statistically
significant decrease of 63,593.4 monthly views, the overall data trend has
shifted an increase of 30,367.3 views per month to a decrease of 33,226.1 per
“rule of thumb” that an observation with a z-score greater than +3.0 or less than -3.0 is
typically considered an outlier).
126 However, these increases although noticeable in the data were not so extreme as to
constitute outlier observations. The PLO Wikipedia article view count for November 2012
had a z-score of 3.0 while for July, 2014, it was 2.61. Neither are outliers. For explanation of
the usual “rule of thumb” for z-scores and outliers, see id. at 247.
127 Thomas Zeitzoff, John Kelly, & Gilad Lotan, Using Social Media to Measure Foreign
Policy Dynamics: An Empirical Analysis of the Iranian-Israel Confrontation (2012-13), 1:16 J. PEACE
RES. 1,2,5 (2015) (among other things, focusing on social media data obtained from Twitter to
track foreign policy discussions across languages online).
128 Aguinis et al., supra note 121.
129 The second set of results set out in Figure D were also corrected for autocorrelation using the Prais-Winsten method. For a discussion of the “importance” of
controlling for auto-correlation in ITS designs using generalized least squares estimators like
the Prais-Winsten method to correct see: Lagarde, supra note 82, at 79.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

33

month. This is important. It means that the NSA/PRISM surveillance
revelations, covered by media in June 2013, are associated in the findings not
only with a sudden chilling effect but also a longer term, possibly even
permanent, decrease in web traffic to the Wikipedia pages studied. This
indicates a possible chill. Figure D illustrates this trend:
Figure D: Pre/Post June 2013 Trends (Outliers Excluded)
The sudden drop in views, and shift in trend from increasing
monthly views over time to decreasing after June 2013, is
consistent with a significant and long-term chilling effect
Total Views (47 Wikipedia Articles (Hamas Excluded))

Mid June, 2013
3500000

3000000

2500000

2000000

1500000
0

2

4

6

8

10

12

14 16 18
Time (Months)

20

22

24

26

28

30

Confidence Interval (95%)

Trend Pre-June 2013

Trend Post-June 2013

Total Article Views (By Month)

32

Here, the increased predictive strength of the model is apparent from the
narrower confidence interval for regression/trend lines in both segments of
the graph. The shifting trend of the data, which in this case is a sudden and
immediate drop, is particularly consistent with a chilling effect arising from
June 2013 revelations. If the outlier data relating to Hamas view counts is
excluded, the decline in page views is less sudden (e.g. 20% immediate drop
off if the Hamas data are excluded compared to the 30% drop off in the
Hamas data remains in the study). However, regardless of whether the Hamas
data is included, there is still a substantial and statistically significant decrease.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

34

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

Moreover, there is a change in the overall trend in the data.130 Before June
2013, total views of the Wikipedia articles in the dataset are slowly increasing
month to month. They are clearly on the rise, at least until June 2013. After
that month, however, with the widespread “exogenous shock” of publicity
surrounding the PRISM/NSA surveillance revelations, there is a change in the
“slope” or data trend. Without the outlier “Hamas” view counts in July 2014,
the total views are on a downward path.
In sum, the visualization of data shows empirical evidence that is
consistent with a long-term chilling effect due to the PRISM surveillance
revelations, which is not only associated with an immediate drop in views but
also a long-term chill on accessing these Wikipedia articles, as users accessed
information on these topics less and less frequently.
D.

FINAL RESULTS

WITH COMPARATOR / QUASI-CONTROL GROUP

This section presents a final set of results, visualized, to strengthen the
chilling effects inference from these findings, that is, the reduction in article
views after the June 2013 revelations are a result of Wikipedia users’
surveillance-related privacy concerns, leading to chilling effects. Two main
steps were taken to strengthen this inference in these final results. First, to
focus on Wikipedia article content most likely to raise privacy concerns for
users, only the 31 articles with the highest combined privacy ratings (from the
MTurk privacy evaluation) were included.131 Second, a quasi-control132 group
of Wikipedia articles was added to the analysis. In a classic controlled
experiment, a control group is randomly selected from the same population or
sampling frame as the experimental group.133 The design’s logic is that if you
draw from an identical or very similar sampling pool, the only significant
difference between the testing and the control group, is the latter is not
exposed to the intervention or treatment; therefore, if the treatment group is
impacted while the control group is not, this strengthens the inference that the
treatment or intervention caused any observed impact or intervention

130 Again, using percentage of non-overlapping data (PND) method to determine effect
size, here the “intervention” in June 2013 clearly has a PND value of over 80% for a very large
effect size. See Scruggs & Mastropieri, supra note 120, at 227.
131 This left 31 Wikipedia articles. The articles included can be viewed in Appendix E.
Of the original 47 articles (48 minus Hamas), the median of the combined privacy score was 2;
so anything below the media was removed to focus on those articles that should raise the most
serious privacy concerns.
132 The control group is only “quasi” as there was no opportunity to isolate a group of
Wikipedia users, before June 2013 revelations, not exposed to surveillance related concerns.
This is because online surveillance potentially affects everyone.
133 See NOREEN L. CHANNELS, SOCIAL SCIENCE METHODS IN THE LEGAL PROCESS 5860 (1985) (introducing experimental design).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

35

effects.134 Though true experiments are rarely found outside laboratories
because they require highly controlled settings,135 employing quasiexperimental features in research designs like ITS—such as employing a
control group—helps strengthen findings and results.136 As with experimental
designs, a control group employed in an ITS design is ideally identical or very
similar to the “experimental” group—if possible drawn from the same
population—but would not “experience” the intervention.137 Results for both
the “testing” group (here, the terrorism-related Wikipedia articles) and the
control group are compared to better understand any “intervention” impact
associated with the June 2013 revelations.138
To create a quasi-control group as similar as possible to the terrorismrelated Wikipedia articles in this study, a grouping of security-related Wikipedia
articles was created using the category of “DHS and Other Agencies”
homeland security related keywords from the same 2013 DHS document.139
Logic of this design choice is straightforward. First, while using identical set of
Wikipedia articles for the control group is impossible, terrorism and domestic
security agencies are very closely related content and likely attract similar
readers and Wikipedia users (thus rendering the article group and quasi-control
134 Id. at 58-60 (describing the classic experimental design and procedure). See also
DAVID DE VALUS, RESEARCH DESIGN IN SOCIAL RESEARCH 53-55, 58 (2003)(introducing
classic experimental design and procedures and also explaining how use of control groups in
experiment design help control for unknown factors that may explain treatment effects; if
both groups are identical and very similar, then they are likely equally exposed to these outside
unknown factors, such that any observed effects would show up for both groups, discounting
intervention or treatment effects as the cause); CAMPBELL & STANLEY, supra note 27, at 13-34
(providing an extensive discussion of different forms of experimental design, and how such
designs guard against threats to internal and external validity).
135 See MATTHEW DAVID & CAROLE D SUTTON SOCIAL RESEARCH: AN
INTRODUCTION206 (2d ed., 2011) (discussing some of the challenges with using experimental
designs in the “social world”); CHANNELS, supra note 133 at 61 (noting the difficulty of doing
experiments “outside the laboratory”).
136 See Wagner et al., supra note 27, at 306-307 (defending single group ITS designs as
robust, but also noting and discussing many benefits of employing a control group in an ITS
design and analysis); Penfold & Zhang, supra note 27, at S43 (noting that while single group
ITS designs (with segmented regression) can still be carried out in the absence of a proper
control group, the “strength of inference is weaker in the absence of the counterfactual
outcome.”).
137 Wagner et al., supra note 27, at 306 (“Ideally, a control group that is identical to the
study group but does not experience the intervention is followed over the same time period as
the intervention group. Comparing the effect in the intervention group with that in the control
group then allows separating the intervention effect from others that may have occurred at the
same time.”).
138 Wagner et al., supra note 27, at 306 (noting the experimental and control groups are
compared); Penfold & Zhang, supra note 27, at S40-S41 (noting the importance of comparison
between the experimental and control groups).
139 PRIVACY IMPACT ASSESSMENT (2013), supra note 88, at 24.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

36

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

article similar). The two groups of Wikipedia articles are also created by
matching articles with keywords in the same DHS document,140 so the articles
are being drawn from the same “source”, again, rendering the groups more
similar and closely related for comparative purposes.141 However, in theory,
viewing terrorism-related Wikipedia content is far more likely to raise privacy
concerns for Wikipedia users concerned about government surveillance than
merely viewing information about security agencies like “Department of
Homeland Security” or security “Fusion Centers”. As such, article traffic
concerning the security-related Wikipedia articles in the quasi-control group
should not lead to a June 2013 related chilling effect. The hypothesis, based on
chilling effects theory, is that users viewing terrorism-related Wikipedia articles
should, by contrast, be more impacted and chilled.142 This hypothesis is again
explored, but this time results are graphed with both linear (first graph) and
fractional polynomial trend analysis (second graph) to provide an even clearer
140 Locating Wikipedia articles coinciding with each keyword was again done manually
and similarly was very simple as there was a Wikipedia article that corresponded perfectly with
the vast majority of keywords in the “DHS and Other Agencies” DHS keyword category. The
few discrepancies were these: the Wikipedia article “National Security Operations Center” was
used for the keyword “National Operations Center”; the Wikipedia article “biological agent”
was used for the DHS keyword “agent”; the Wikipedia article “Joint Task Force” was used for
the keyword “Task Force”; the Wikipedia article “Drug Enforcement Administration” was
used for the keyword “Drug Enforcement Agency” (this article related to an unknown agency
in Liberia); the Wikipedia article “International Federation of Red Cross and Red Crescent
Societies” was used for the keyword “Red Cross”. Wikipedia articles corresponding with the
remaining related keywords were all included in these final set of results, except for the
“United Nations” article. This article was excluded from the final results and analysis, as this
article, like the Hamas article excluded earlier, was a significant and extreme outlier, exerting
substantial influence on a number of observations in the data and distorting results (as
apparent from its extreme Cooks D values). As such, it was excluded. It might also be noted
that observing the keywords in the “DHS and Other Agencies” category in the DHS
document overall, the keyword “United Nations” stands out as more international than
domestic. See PRIVACY IMPACT ASSESSMENT (2013), supra note 88, at 24. The 25 Wikipedia
articles included can be viewed in Appendix F.
141 This attempts a more robust approach to quasi-control group selection than the
more “neutral” set of quasi-control terms employed by Marthews and Tucker. This is because
the greater the difference between the test group of terms (or in this case articles) and the
control group, the greater the possibility that any observed intervention effect can be
explained by some unknown confounding factor variable possessed by the studied group of
search terms or Wikipedia articles but not the control group. See Wagner et al., supra note 27, at
306 (explaining the role of control groups). See also Marthews & Tucker, supra note 25, at 7
(explaining their use of “neutral” terms as a control).
142 This is another reason why the keywords under the “DHS and Other Agencies”
category heading in the DHS document were used to select a quasi-control group as many
keywords in the other categories beyond “Terrorism” (e.g., “Domestic Security”, “HAZMAT
and Nuclear”, etc) concern topics or content that may very well raise privacy concerns for
users aware of government surveillance online. As such, such terms would not be appropriate
as quasi-controls.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

37

picture of trends in the data (the former best captures linear relationships in
the data while the latter best illustrates non-linear relationships, hence the
smooth, rather than rigid, trend lines).143 The final comparative results are
available in Appendix C and are plotted and visualized in Figure E:
Figure E: Terrorism Articles vs. Control Group
The sudden drop and trend reversal for the terrorism-related
articles is consistent with chilling effects. The security-related
articles control group shows little impact (includes both linear
and polynomial trend analysis).
Linear Trend Analysis
Mid June, 2013
1200000

Total Article Views

1000000

800000

600000

400000

200000
0

2

4

6

8

10

12

14 16 18
Time (Months)

20

22

24

26

28

30

32

Confidence Interval (95%)

Terrorism Articles Trend

Security Articles Trend (Control)

Terrorism-related Articles

Security-related Articles (Control Group)

143 Segmented regression was completed, this time employing a fractional polynomial
trend analysis for flexibility and to account for any non-linear relationships. See Douglas G.
Altman & Patrick Royston, Regression Using Fractional Polynomials of Continuous Covariates:
Parsimonious Parametric Modelling (1994) 43:3 APPL. STATIST. 429, 429-430 (1994) (discussing the
usefulness of polynomials to capture non-linear relationships in behavioral data).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

38

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __
Non-Linear/Polynomial Trend Analysis
Mid June, 2013
1200000

Total Article Views

1000000

800000

600000

400000

200000
0

2

4

6

8

10

12

14 16 18
Time (Months)

20

22

24

26

28

30

32

Confidence Interval (95%)

Terrorism Articles Trend

Security Article Trend (Control)

Terrorism-related Articles

Security-related Articles (Control Group)

Once again, the reduction in views and reversal in trend for the terrorismrelated Wikipedia articles, apparent in both Figure E graphs, are consistent
with a significant and potentially long term chilling effect. Both the immediate
drop off (262,075) is large and the trend change from increasing views
monthly (34,764) to fewer (44,060.4) were statistically significant at the .99
confidence level. In May 2013, the 31 terrorism-related Wikipedia articles
were viewed 981,499 times, so the 262,075 views in June represent a sudden
drop off of over 25%.
By contrast, article views for the quasi-control group—security-related
Wikipedia articles—suggest very little impact, or evidence of chill, associated
with the June 2013 revelations. Before June 2013, no clear increasing or
decreasing trend is observable in article views and, over the course of June,
there is a slight drop in views (1,098.8), but this is not statistically significant
and trivial compared to the significant drop off over June for the terrorismrelated Wikipedia articles (262,075). The common drop in June, however
slight, may suggest some minor background factor or event across all
Wikipedia articles more generally, but the difference in the reduction (or
effect) as between the two groups on this count is so stark (one is large and
statistically significant, while the other not), that the results still suggest an
immediate chilling effect for the terrorism articles, but not for the control
group. In fact, unlike the terrorism article group—which experienced a
statistically significant negative trend change (44,060 fewer views a month)—

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

39

the control group actually experienced an increase in monthly views (9,756) after
June 2013. Though not statistically significant, it was nearly so (p = 0.058).
These observations are also apparent from the visualizations in Figure E,
where the trend analysis for the quasi-control security-related Wikipedia
articles group indicate no long term chilling effects after the June 2013
revelations—the article view trends suggest a slight dip and then, as before
June, a slight increase monthly in views, showing no short term or long term
impact. This is in contrast to the terrorism-related articles in the study that
noticeably trend downward Post-June. In fact, the polynomial trend analysis in
the second graph suggests in the future, the trend lines will eventually cross,
despite the strong trend towards increasing article readership before June 2013.
And the slight curve in the polynomial trend line post June 2013, which curves
downward over time, may be evidence of chilling effects interacting with
information diffusion—as information about the government surveillance
spreads and more learn of it through online and traditional sources and media
over time—fewer users view the terrorism-related Wikipedia articles over time
as well (in contrast to the security-related articles, which are slightly increasing
in views over time).144
These findings strengthen the inference that the explanation for the
sudden drop during and after June 2013 for the terrorism-related Wikipedia
articles, and trend reversal, arise due to surveillance related chilling effects.
This inference is supported by the fact that there is no indication in the
findings that the similar and similarly situated quasi-control security-related
Wikipedia articles were likewise impacted by the June 2013 revelations. The
explanation, it can be surmised, is that the security-related Wikipedia articles,
though similar, are simply unlikely to raise privacy concerns for Wikipedia
users worried about online surveillance. Thus, article views before and after
the June 2013 show few noteworthy effects, evening increasing in readership
going forward.

144 For a review of research concerning information diffusion and cascading online, see:
Rajiv Garg, Michael D. Smith, & Rahul Telang, Measuring Information Diffusion in an Online
Community, 28:2 JOURNAL OF MANAGEMENT INFORMATION SYSTEMS 11, 11-14 (2011)
(reviewing literature on information diffusion theory and related works). See also Jennifer Earl,
The Dynamics of Protest Related Diffusion on the Web, 13:2 INFORMATION, COMMUNICATION, &
SOCIETY 209, 209-221 (2010) (reviewing research relevant to information diffusion among
social networks online).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

40

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

VI.

IMPLICATIONS

A.

EMPIRICAL EVIDENCE FOR REGULATORY CHILLING EFFECTS

Skepticism among courts, legal scholars, and empirical researchers has
persisted about the nature, extent, and even existence of chilling effects due, in
large part, to a lack of empirical substantiation.145 The results in this case
study, however, provide empirical evidence consistent with chilling effects on
the activities of Internet users due to government surveillance. And, to be
clear, the activity here is not only legal—accessing information on Wikipedia—
but arguably desirable for a healthy democratic society. It involves Internets
users informing themselves about important topics subject to today’s
widespread social, political, moral, and public policy debates.146 The large,
statistically significant, and immediate drop in total views for the Wikipedia
articles after June 2013 implies a clear and immediate chilling effect.
Moreover, the broad and statistically significant shift in the overall trend in the
data (e.g. the shift from the second results excluding outliers) suggests any
chilling effects observed may be substantial and long-term, rather than weak,
temporary, or ephemeral. This study also bolsters support for the existence of
the chilling due to the data upon which it relies. It is among the first studies to
demonstrate evidence of such a chilling effect using web traffic data (instead of
survey responses or search), and the first to do so in relation both to the
potential chilling effects on Wikipedia use, and, more broadly, how such
government surveillance and other actions impact how people access and
obtain information and knowledge online.
These results are consistent with chilling effects theory but arguably
contradict other research concerning online privacy behaviors. First, the
substantial body of “privacy paradox” research, involving a diverse range of
online platforms and contexts, has demonstrated that Internet users’ stated
concerns about privacy are often not reflected in their online behavior.147

145 See Kaminski & Witnov, supra note 9, at 517 (calling for further research on the
“types of surveillance and surveillance cues that cause chilling effects”, as well further research
on both the magnitude and persistence of such surveillance related chilling effects). See also
Kendrick, supra note 14, at 1657; Nickel, supra note 17, at 263; Richards, supra note 21, at 1964.
146 Clark McCauley, Terrorism, Research, and Public Policy: An Overview, in TERRORISM
RESEARCH AND PUBLIC POLICY 134 (C. McCauley, ed., 1991) (“Taken together, the financial,
social, political, and moral costs of response to terrorism constitute a challenge to the
democratic capacity to govern...”).
147 See generally Kokolakis, supra note 44 (providing a comprehensive explanation and
review of “information privacy paradox” literature). See Annika Bergström, Online Privacy
Concerns: A Broad Approach to Understanding the Concerns of Different Groups for Different Uses, 53
COMPUTERS
IN
HUMAN
BEHAVIOR419
(2015),
http://www.sciencedirect.com/science/article/pii/S0747563215300364

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

41

There have been a range of explanations offered to explain this disconnect,148
but one common explanation offered by behavioral economists is that online
users suffer from “incomplete information” and “bounded rationality” in
making decisions about privacy, that is, such decisions are often complex and
people are limited both by cognitive ability and knowledge.149 However, the
results here, consistent with the chilling effects hypothesis that users are
avoiding certain online content due to privacy concerns about online
surveillance, suggest online users are acting both rationally and logically even
with incomplete information about the true nature and scope of covert NSA
surveillance practices. In other words, privacy concerns are being reflected in
online behavior, contrary to the “privacy paradox”.
Second, as noted earlier in this article, privacy researchers and legal
scholars have also expressed skepticism about the possibility of large scale or
long term chilling effects caused by online surveillance due to “desensitization”
about privacy concerns in online contexts150 or because online users adapt
quickly to shifting norms, rendering chilling effects arising due to concerns
about criminal or civil risks stemming from online activities “temporary”.151
On this count, research has found, for example, a “lax attitude” among users
toward the benefits of online privacy (compared, for example, to the benefits
of information disclosure);152 muted user responses to reputational or privacy
risks associated with embarrassing behavior being exposed online;153 and that
while negative privacy experiences online prompted users to adjust their
sharing practices, their “social or psychological privacy behaviors” online were
unaffected.154 If there even are any privacy or surveillance related chilling

148 See e.g. id. at 7-9 (reviewing the various interpretations and explanations for paradox).
149 Id. at 9.
150 Nickel, supra note 17, at 263.
151 See Bernescu, supra note 18, at 671 (“However, because consumers in the Internet
context quickly adapt to changing norms, any such chilling effect will likely be temporary.”).
152 Debatin, supra note 18, at 83, 100-102 (finding that a majority of Facebook users in
their study disclosed a great deal of personal information despite being aware of privacy risks;
they attribute this to a “lax attitude”). See also Kokolakis, supra note 44, at 7 (discussing research
on the “privacy calculus” where people weigh the benefits of privacy over disclosure).
153 Hermstrüwer & Dickert, supra note 67, at 22-23 (an experimental study on chilling
effects finding that risks of “networked publicity” (exposure online of users’ embarrassing
activities) did not affect users’ “privacy valuations” nor “dampen” either “behavioral
idiosyncrasies” or the “panoply of different behaviors” involved in the study; they conclude
that concerns about surveillance related chilling and conforming effects may be “overstated”).
154 Sabine Trepte, Tobias Dienlin, & Leonard Reinecke, Risky Behaviors: How Online
Experiences Influence Privacy Behavior, in B. Stark, O. Quiring, & N. Jackob, eds., VON DER
GUTENBERG-GALAXIS ZUR GOOGLE-GALAXIS [FROM THE GUTENBERG GALAXY TO THE
GOOGLE GALAXY] 225-244 (2014) (“After encountering harassing or humiliating status posts
or messages, users adjusted the information they posted online. However, negative experiences
did not affect social or psychological privacy behaviors. It was shown that the ways that users

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

42

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

effects, such research suggests they may muted, ephemeral, or short-term. The
findings here also contradict these studies, suggesting not only an immediate
chilling effect associated with the June 2013 surveillance revelations, but a
possible longer term chill as well. A determination of whether this trend will
continue further into the future is inherently limited by the data set in this
study, which only extends to August 2014. Nonetheless the downward trend in
the data, overall, does provide evidence of a more permanent impact.
This case study also provides important insights on how to understand
chilling effects, particularly how they operate online. Though Schauer and
Solove’s accounts of chilling effects are closely related, there are important
distinctions between the two. Schauer approached chilling effects as mainly
resulting from uncertainty in the legal system (e.g., vagueness of legislative
enactments) and people’s fear of prosecution and legal sanction.155 Solove, on
the other hand, broadened chilling effects theory’s outlook by focusing his
efforts on surveillance and “executive information gathering.”156 The findings
here are more consistent with Solove’s approach to chilling effects. On his
account, people censor themselves and avoid certain activities not out of fear
of prosecution but out of concern for potential future harms due to privacy
violations, embarrassing public disclosures, risks of fraud or identity theft, or
being labeled a criminal, deviant, or non-conformist by state authorities.157
Given the lack of evidence of people being prosecuted or punished for
accessing information on Wikipedia or similar sites before, during, or after the
June 2013 revelations, it is unlikely that actual fear of prosecution can fully
explain the chilling effects suggested by these findings of this study. Rather,
Solove’s notion of surveillance-related “pollution,” that is, a broader societal
context of self-censorship and conformity arising from ubiquitous and largescale surveillance, may be the better explanation.158

managed their audiences and friends (social privacy) and the kinds of information they shared
(psychological privacy) remained unaffected by negative experiences.”).
155 See Schauer, supra note 12, at 693-695 (discussing, among other things, how fear, risk,
and uncertainty in the legal process as contributing to potential chilling effects).
156 See Solove, supra note 14, at 142-151 (exploring and analyzing cases wherein
government information gathering implicates the First Amendment and related “chilling
effects”). See also Solove, supra note 32, at 487-489 (identifying surveillance related chilling
effects as a “more modern privacy problem” that “does not fit” with more traditional
conceptions of privacy harms).
157 See Solove, supra note 32, at 493-499 (reviewing a range of instances where
surveillance and related information gathering activities can lead to chill, self-censorship,
inhibition, and other forms of privacy harms).
158 Id. at 487-488 (discussing how the broader sets of risks caused by government
practices like surveillance, which can be likened to “environmental harm” or “pollution”, also
should be understood as having broader societal impact beyond any “mental pain and distress”
caused to individuals).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

43

This inference is supported by the independent privacy evaluation
completed by the 415 respondents recruited through MTurk. Respondents’
assessment of all forty-eight keyword topics (the forty-eight keywords
corresponded with the forty-eight terrorism-related Wikipedia articles in the
case study) and their responses indicated that if they knew the government was
monitoring online activities, they would be more likely to avoid the forty-eight
topics in question. In other words, their responses suggested a potential for
chilling effects relating not to fear of prosecution but fears, risks, harms, and
threats associated with government surveillance. This point should not be
taken too far, however, as the independent evaluators were recruited among
MTurk users and not Wikipedia users tracked by the article traffic data in this
study. So the findings may not hold for Wikipedia users or Internet users
more generally.
B.

THE IMPACT OF WAR AND OTHER EXOGENOUS EVENTS

Another important insight from the study is how intervening dramatic
external pressures or “exogenous shocks” impact chilling effects.
Notwithstanding the evidence both of immediate, substantial, and potentially
long-term chilling effects due to awareness of government surveillance, those
chilling effects can be affected or impacted by other dramatic intervening
events or “exogenous shocks” like war. For instance, the impact that the
November 2012, and July 2014 Israeli-Hamas conflicts had on the Wikipedia
use chilling effects can be interpreted in two ways. First, it could be argued
that these instances of high profile armed conflict “ameliorated” any chilling
effects possibly caused by publicity and public awareness of the PRISM/NSA
surveillance, as article traffic, at least for one or two of the Wikipedia articles
among the forty-eight retrospectively analyzed in the case study, dramatically
increased as those conflicts unfolded. People’s desire to learn about the IsraeliHamas conflicts simply overrode any concerns they may have had about the
government monitoring information they were accessing online, thus reducing
chilling effects.
A second way of interpreting the results, is that the Israeli-Hamas conflicts
essentially “hid” or “masked” a broader chilling effect (apparent once the
“Hamas” article view data was excluded), by bringing new populations of
atypical users to who were less aware of, or concerned by, government
surveillance. This would seem to support the findings of Zeitzoff, Kelly, and
Lotan who have explored how major conflicts resulted in “significantly higher
levels” of social media activity. For example, their findings included the
emergence of what they called “ephemeral” users who seemed to only “tweet”

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

44

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

about the 2012 Gaza conflict and nothing else.159 Perhaps those visiting the
Hamas and Palestinian Liberation Organization Wikipedia articles in
November 2012 and July 2014 were atypical Wikipedia users and less affected
by online government surveillance (and any attendant chilling effects). But
there is nothing in this study to so suggest. Clearly, additional research
focusing on this aspect of this case study could offer further insights on this
dimension of the results.
C.

CONSTITUTIONAL LITIGATION

The results in this case study should, first of all, provide important
empirical support for the “chilling effects doctrine” in First Amendment law.
Skepticism about the chilling effects doctrine dates back decades, and, more
recently, scholars have concluded that more research is required to support the
“unsubstantiated empirical judgments” of chilling effects claims under the
First Amendment (and chilling effects more generally).160 These results meet
this call. Second, the evidence of chilling effects in this case study has
important implications for a wide array of present constitutional litigation
brought in relation to government surveillance practices. Indeed, a significant
challenge for recent lawsuits filed against the NSA and the U.S. government,
especially those based on a chilling effects theory like the Wikimedia
Foundation complaint, is the issue of standing. A key part of this challenge is
the nature of government surveillance online—it is covert and secretive so
victims are most often unaware if they have been personally targeted.161 This is
compounded by how the Supreme Court has applied standing in cases
involving covert surveillance. The findings of this study may help plaintiffs
overcome challenges to standing by providing empirical evidence to ground
constitutional claims based on chilling effects and related harms in objective
evidence not subjective claims and fears.
The U.S. Supreme Court’s recent decision in Clapper exemplified existing
legal and judicial skepticism of chilling effects. In that decision, a five judge
majority dismissed as “too speculative” the plaintiffs’ assertion of standing
based on a likelihood that their activities would be subject to surveillance in
the future.162 The Court similarly dismissed the additional arguments for
159 Zeitzoff et al., supra note 127, at 5; Zeitzoff, supra note 127, at 13.
160 Kendrick, supra note 14, at 1657.
161 See Richards, supra note 23, at 1934 (“Although we have laws that protect us against
government surveillance, secret government programs cannot be challenged until they are
discovered.”). See generally Slobogin, supra note 38 (analyzing legal standing issues in relation to
constitutional challenges to NSA and other surveillance related practices).
162 Clapper, supra note 4, at 1143 (“Respondents assert that they can establish injury in
fact because there is an objectively reasonable likelihood that their communications will be
acquired under § 1881a at some point in the future. But respondents' theory of future injury is

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

45

standing based on “chilling effects”, observing that while prior cases found
constitutional violations may arise from chilling effects, such violations could
not arise “merely” from a person’s “knowledge” or “concomitant fear” about
government activities.163 Relying on its 1973 decision in Laird, the Court noted
that “[a]llegations of a subjective ‘chill’ are not an adequate substitute for a
claim of specific present objective harm or a threat of specific future harm.”164
Such surveillance related chilling effect claims based on subjective fears were
“self-inflicted” injuries, the Court concluded, and thus could not provide
standing for the constitutional claims.165
Clapper is unlikely the final word on standing based on widespread
government surveillance. To begin with, the case was decided in February of
2013, several months before the Snowden disclosures and the widespread
publicity concerning the PRISM and other government surveillance programs,
which have rendered such secretive surveillance more public and thus more
subject to scrutiny and challenge.166 Moreover, commentators like Neil
Richards, Luke Milligan, and Christopher Slobogin, among others, have
offered persuasive criticisms of the Supreme Courts’ approach to standing in
Clapper.167 Richards argues that Clapper’s approach to standing affirms a “brutal
paradox” whereby litigants must prove harms (like chilling effects) arising
from secretive covert surveillance but the only party that knows—the
government—is not telling.168 Milligan, on the other hand, offers a compelling
argument that the Clapper approach to standing and chilling effects claims is
inconsistent with the text and history of the Fourth Amendment, which was
too speculative to satisfy the well-established requirement that threatened injury must be
‘certainly impending’.”).
163 Clapper, supra note 4, at 1152.
164 Id. (quoting Laird v. Tatum, 401 U.S. 1, 13-14 (1972)).
165 Id. at 1152-1153 (quoting Laird v. Tatum, 401 U.S. 1, 13-14 (1972)).
166 See Slobogin, supra note 38, at 4 (noting “Thanks to Edward Snowden...” the U.S.
federal government has been forced to acknowledge certain surveillance practices, while
journalists have shed important additional light).
167 See Richards, supra note 23, at 1963-1964 (argues, based on the notion of “intellectual
privacy” and its importance to the democratic principle of “self-government”, that surveillance
privacy harms should be recognized under legal standing doctrines and that Clapper fails to do
so); Luke M. Milligan, The Forgotten Right to be Secure, 65 HASTINGS L. J. 713, 732-750 (2014)
(argues for a broader approach to standing than recognized in Clapper—that would allow
earlier Fourth Amendment challenges to concealed government investigative techniques—
based on the Fourth Amendment’s original understanding as defined by its
text,
history, and structure); Slobogin, supra note 38, at 535-541 (draws on both Richards and
Mulligan to argue, among other things, that Clapper and its standing requirements undermine
the “political process”, which is what the standing process was meant to protect). See also Lexi
Rubow, Standing in the Way of Privacy Protections: The Argument for a Relaxed Article III Standing
Requirement for Constitutional and Statutory Cause of Action, 29 BERK. TECH. L. J. 1007 (2014)
(analyzing and critiquing current standing doctrine in light of the difficulties of proving privacy
harms).
168 See Richards, supra note 23, at 1944-45.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

46

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

originally understood to guarantee freedom not just from individual
unreasonable searches but also freedom from “fear” of such searchers.169
Lastly, Slobogin offers a strong criticism of Clapper based on political process
theory and the separation of powers, arguing that chilling effects caused by
covert surveillance undermine the political process; as a result citizens should
have standing to challenge such surveillance in court.170
Still, Clapper remains the law and thus presents a difficult standard to show
injury and standing for “chilling effects” constitutional claims.171 On this
count, this case study’s empirical findings will have important implications for
present and future litigation. As noted, Clapper emphasized the need for
evidence beyond “self-inflicted” injuries based on “subjective fears” about
chilling effects to support standing.172 This case study provides empirical
support for surveillance-related chilling effects on Wikipedia’s users—not on
Wikimedia Foundation itself, though it is impacted by virtue of its users being
chilling—meaning any constitutional claims are not necessarily subjective, selfinflicted, nor speculative about future harms. The findings also suggest those
chilling effects are not trivial or temporary, but significant, sudden, and with a
long term impact. The plaintiffs in Clapper, whose claims about harms due to
chilling effects were based on costs incurred to avoid government surveillance,
argued such harms mostly in an empirical vacuum. This approach left their
claims vulnerable to Laird and its holding that subjective allegations could not
create standing. By contrast, this case study provides empirical support for
Wikimedia Foundation to assert harm in its lawsuit against the NSA and
Justice Department based on chilling effects claims:
The notion that the N.S.A. is monitoring Wikipedia’s users is not,
unfortunately, a stretch of the imagination. The harm to Wikimedia
and the hundreds of millions of people who visit our websites is
clear: Pervasive surveillance has a chilling effect. It stifles freedom of

169 See Milligan, supra note 167, at 750 (“On the basis of both text and history, the
Fourth Amendment right “to be secure” can be fairly read to encompass the right to be
‘protected’ from unreasonable searches and seizures, and quite possibly the right to be ‘free
from fear’ of such government actions. This broader interpretation of ‘to be secure’ has
important implications for prevailing Fourth Amendment rules and procedure (arguing that
the text and history of the Fourth Amendment supports standing for technological chilling
effects claims”).
170 See Slobogin, supra note 38, at 535-541 (draws on both Richards and Mulligan to
argue, among other things, that Clapper and its standing requirements undermine the “political
process”, which is what the standing process was meant to protect).
171 Slobogin, supra note 38, at 6 (“As the outcome in Clapper illustrates, because NSA
surveillance is, by design, covert, the standing requirement that plaintiffs allege a ‘concrete’
injury can pose a serious obstacle to parties trying to challenge it...”).
172 Clapper, supra note 4, at 1152-1153 (quoting Laird v. Tatum, 401 U.S. 1,13-14
(1972)).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

47

expression and the free exchange of knowledge that Wikimedia was
designed to enable.173

The results of this case study suggest that the harm produced by chilling
effects is not a “stretch of the imagination” at all. The June 2013 surveillance
revelations, extensively covered by media, appear to have had a salient and
observable chilling effect on Wikipedia users accessing certain Wikipedia
articles. Additionally, this case study provides a more general empirical
foundation for companies, organizations, and other institutions whose users
may have been “chilled” by government surveillance to assert constitutional
harms.
D.

SURVEILLANCE, WIKIPEDIA, AND DEMOCRATIC SOCIETY

This case study also has implications for the health of democratic
deliberation among citizens. Surveillance related chilling effects, in deterring
people from exercising their rights and freedoms, have clear implications for
individual citizens.174 However, these same chilling effects also have
implications for the broader health of society, threatening what Richards calls
“intellectual privacy,” which is the freedom to read, think, and communicate
privately, essential privileges to democracy and “self government.”175 Chilling
effects are indeed a force for conformity and therefore corrosive to “political
discourse.”176
This, in particular, is a problem for Wikipedia. Democratic theorists have
long pointed to public deliberation as an essential “tool” to enhance collective
understanding and decision-making, and Wikipedia has been found to be an
important complement to this democratic process.177 On this count, Wikipedia
provides a collaborative model of knowledge production that strengthens
democracy. A study by Nathaniel Klemp and Andrew Forcehimes found that
Wikipedia offers enhanced democratic deliberation and collective decisionmaking through its “model” of citizen engagement and information
exchange.178 And, beyond these important contributions, Wikipedia not only
173 Wales & Tretikov, supra note 3.
174 See Richards, supra note 23, at 1950 (noting that protection against chilling effects is
necessary to preserve freedom of speech and thought, two important First Amendment
values).
175 Richards, supra note 23, at 1959, 1963.
176 BRUCE SCHNEIER, DATA AND GOLIATH: THE HIDDEN BATTLES TO CAPTURE YOUR
DATA AND CONTROL YOUR WORLD 95-99 (2015). See also RON DEIBERT, BLACK CODE:
INSIDE THE BATTLE FOR CYBERSPACE 130-32 (2013); Solove, supra note 32, at 494-99.
177 Nathaniel Klemp & Andrew Forcehimes, From Town-Halls to Wikis: Exploring
Wikipedia's Implications for Deliberative Democracy, 6:2 J. PUB. DELIBERATION 1, 27 (2007) (finding
that the Wikipedia model of online interaction offers a ”powerful” supplement to traditional
face-to-face forms of public deliberation).
178 Id. at 31-32.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

48

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

remains incredibly popular online—every month, Wikipedia is visited by nearly
half a billion people from almost every country on earth179—but is an
increasingly important resource for Internet users to quickly and efficiently
inform themselves about government policies, laws, and actions, this better
equipping them to “enter into deliberations over political decisions.”180
Moreover, Wikipedia provides a collaborative model of knowledge production
that strengthens democracy. A study by Nathaniel Klemp and Andrew
Forcehimes found that Wikipedia offers enhanced democratic deliberation and
collective decision-making through its “model” of citizen engagement and
information exchange.181
The importance of Wikipedia as a source of online knowledge and
information is highlighted by the data in this case that showed people taking to
Wikipedia in dramatic numbers in November 2012 and July 2014, as compared
to other months in the study’s data set, to inform themselves about “Hamas”
during the Israeli and Hamas conflicts. This suggests large populations of
Internet users sought to inform themselves about the conflict as media
coverage on it unfolded during those months. Whether these users were
typical or atypical Wikipedia users, the findings imply that Wikipedia was a key
source of information gathering about an contentious and globally covered
armed conflict. This conclusion is consistent with prior research on how
people seek information about breaking news stories on Wikipedia and how
such events impact its contributor communities and content.182
But these findings have potentially troubling implications too. Beyond
these war-related outliers represented by the Hamas and Palestinian Liberation
Organization Wikipedia articles during the Gaza conflict, the case study
indicates government surveillance may have a long-term chilling effect on this
type of important Wikipedia use. If people are chilled from informing
themselves about breaking news stories and other important news events, or
from doing simple research online on matters of law, security, and public
policy like “terrorism,” then chilling effects also have serious implications for
public policy and public deliberation on point. With people potentially chilled
or deterred from such basic acts of information gathering, people will be less
informed, and our broader processes of democratic deliberation will be
179 See Michelle Paulson & Geoff Brigham, Wikimedia v. NSA: Wikimedia Foundation
Files Suit Against NSA to Challenge Upstream Mass Surveillance, WIKIMEDIA BLOG (Mar. 10 2015),
http://blog.wikimedia.org/2015/03/10/wikimedia-v-nsa/.
180 See Klemp & Forcehimes, supra note 177, at 31-32 (“Ideally, such potential
applications of the Wikipedia model would enhance existing forms of face-to-face
deliberation. The information gathered through such political wikis would help to inform
citizens and better equip them to enter into deliberations over political decisions.”)
181 Id.
182 See works cited supra note 72-74.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

49

weakened. If “intellectual privacy,” as Richards argues, is essential to
democracy and “self government” in guaranteeing the space and freedom to
read, think, and communicate privately,183 so too is the freedom (free from
insidious surveillance-related chilling effects) to gather basic information from
important platforms and resources like Wikipedia that make engaging in acts
of thinking, communicating, and decision-making, meaningful.
Finally, Wikipedia has proven valuable beyond merely being an online
source of knowledge. As earlier noted, researchers have employed Wikipedia
and information the Wikimedia Foundation makes available online about
Wikipedia articles and editing activities for a broad range of online and offline
research interests, including peer-production, mapping global and local
knowledge production, and determining how geography and place are
represented online.184 As a popular and highly successful collaborative peerproduction online platform, Wikipedia is also invaluable as a focal point for
research exploring collaborative networks and knowledge production. All of
this invaluable research is put in doubt if people are chilled from using the site
due to government surveillance.
VII.

LIMITATIONS

Notwithstanding the significance of this case study’s findings and their
attendant implications, they have three important limitations. First, the period
of the study only extends until August 2014. This means that the persistence
of any chilling effects beyond that point remains an open question. Though
the findings here suggested a long-term, even permanent, chilling effect, this
possibility cannot be confirmed or denied using existing data. Additional
research using more recent data could shed some light on this aspect of the
study. Second, a true experimental design, one with a true control group—
randomly drawn from the identical subject pool or population—was not
possible. Given the secrecy surrounding government surveillance practices
and their potential wide scope, the research design could not be strengthened
by comparing Wikipedia users affected by surveillance with a true control
group that had not been exposed to online surveillance; the covert nature of
the surveillance rendered it impossible to isolate or identify such a group of
individuals. This is one of the challenges of studying chilling effects and the
impact of surveillance—much of the practices at issue are secret and thus
difficult to study systematically. Still, a quasi-control group was included in the
final analysis, rendering the final results more robust.

183 Richards, supra note 23, at 1950, 1963.
184 See generally sources cited supra note 72.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

50

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __

A related limitation, is that it was impossible to know (because the data is
simply not available) whether Wikipedia users chilled or deterred from viewing
the articles included in the study may have been viewing the very same content
elsewhere. Ideally, this would be controlled for in an experimental setting, but
like all naturalistic or observational studies involving data derived from the
field (as here), there are certain variables and factors that cannot be known or
controlled. The ITS design, however, provides a robust means to analyze data
where, as here, true experimental designs are not possible. Third, the nature of
the Wikipedia data also limited the ways in which the research design could
have been strengthened in this case study. For example, though the study
focused on English Wikipedia articles, article view counts used to construct
the time series data set did not distinguish the geographic origins of article
views. That distinction would have provided some insight as to whether
evidence of chilling effects varied across geographical regions. Further
research attempting to address or overcome these limitations would be
valuable.
VIII. CONCLUSION AND FUTURE DIRECTIONS
The compelling evidence of chilling effects illustrated in this case study has
important implications on multiple fronts. The study’s ITS design, combined
with a segmented regression analysis and a quasi-control group, also provides a
powerful research design and analytical method that can be employed by
researchers in other contexts to explore chilling effects and related regulatory
impacts.
Yet, there are clear future directions for research too. This case study has
focused primarily on whether the June 2013 surveillance revelations had a
chilling effect on Wikipedia users and whether there were any immediate
implications of the findings on that question. But the economic impact of those
chilling effects was not explored and further could shed important light on
such monetary harms stemming from regulatory chill. As Tucker and
Marthews observe in relation to chilling effects on Google search,185 the
findings here suggest that NSA/PRISM programs and other forms of
government surveillance may have a substantial impact on the bottom line of
online service providers and other businesses. Since Wikipedia is a non-profit
collaborative effort, this angle was not explored in depth here; but Wikipedia,
185 Marthews & Tucker, supra note 25, at 23 (“From a US competitive standpoint, the
longer-run effect observed on international Google users’ search behavior indicates that
knowledge of US government surveillance of Google could indeed affect their behavior. At
the most limited end of the spectrum, it could steer them away from conducting certain
searches on US search engines; at the most severe end of the spectrum, they might choose to
use non-US search engines.”).

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

CHILLING EFFECTS

4/26/201611:20 PM

51

like many Online Service Providers, survives through user traffic and
contributions, both of which may be chilled temporarily or permanently by
government surveillance. In fact, Wikimedia Foundation, the ACLU, and
other organizations (like The Nation magazine and Human Rights Watch) that
are party to the lawsuit against the NSA and U.S. Justice Department make
similar assertions, claiming they have incurred significant costs to preserve
privacy and confidentiality in response to the NSA surveillance activities.186
Moreover, it is also unknown whether the findings in this case study hold for
other comparable forms of online information resources. Was Wikipedia,
given its prominence as a popular online information provider, unique in being
impacted? And were Wikipedia editors and contributors affected differently
from general Wikipedia users (e.g., users who merely read Wikipedia articles,
but do not produce or edit them). Despite the evidence established in this
case study concerning the existence and scope of chilling effects, significant
gaps remain in literature. Further work can be done both on Wikipedia and in
other online contexts to extend our understanding of regulatory chilling effects
in both North America and abroad. Though the true scope of chilling effects
still remains to be fully explored and analyzed, this case study nevertheless
offers an important contribution to its deeper understanding.

186 See Complaint, supra note 1, at 15, 22, 24, 26.

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

52

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __
====================
APPENDIX A
First Results, Segmented Regression Analysis

Independent Variable
Coefficient (0)

Coefficients

Standard
Error

1586224.0*

1156233.0

0.181

36498.2*

23105.8

0.126

853585.5*

322969.0

0.014

31192.5*

27308.2

0.263

Pvalue

Expected Total Views at Beginning of Study

Secular trend in data (1)
Change in Total Views (Month to Month) Before
6/2013

Change in level (2)
Change in Total Views Immediately After 6/2013

Change in slope (3)
Change in Total Views (Month to Month) After
6/2013
Note 1: *p< 0.05, **p < 0.01

====================
APPENDIX B
Second Results, Excluding Outliers and Correcting for Auto-Correlation
Independent Variable
Coefficient (0)

Coefficients

Standard
Error

1969435.0**

691764.1

0.008

30367.3**

16015.5

0.069

540318.4**

210584.8

0.016

63593.4**

23221.5

0.011

Pvalue

Expected Total Views at Beginning of Study

Secular trend in data (1)
Change in Total Views (Month to Month) Before
6/2013

Change in level (2)
Change in Total Views Immediately After 6/2013

Change in slope (3)
Change in Total Views (Month to Month) After
6/2013
Note 1: *p< 0.05, **p < 0.01

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

53

====================
APPENDIX C
Final Set of Results– Segmented Regression (compared)
31 Terrorism-related Wikipedia Articles Study Group
Independent Variable
Coefficient (0)

Coefficients

Standard
Error

806987.4**

276978.7

0.007

34764.9**

5518.1

0.000

262075.0**

81288.47

0.003

44060.4**

6513.11

0.000

Pvalue

Expected Total Views at Beginning of Study

Secular trend in data (1)
Change in Total Views (Month to Month) Before
6/2013

Change in level (2)
Change in Total Views Immediately After 6/2013

Change in slope (3)
Change in Total Views (Month to Month) After
6/2013
Note 1: *p< 0.05, **p < 0.01

25 Security-related Wikipedia Articles Quasi-Control Group
Independent Variable
Coefficient (0)

Coefficients

Standard
Error

116529.2**

153860.5

0.456

8192.9**

3963.6

0.049

1098.8**

46894.5

0.981

9755.7**

4917.0

0.058

Pvalue

Expected Total Views at Beginning of Study

Secular trend in data (1)
Change in Total Views (Month to Month) Before
6/2013

Change in level (2)
Change in Total Views Immediately After 6/2013

Change in slope (3)
Change in Total Views (Month to Month) After
6/2013
Note 1: *p< 0.05, **p < 0.01

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

54

4/26/201611:20 PM

BERKELEY TECHNOLOGY LAW JOURNAL [Vol. __
====================
APPENDIX C
Independent Ratings Results
The rating is on a scale of 1 to 5, with 1 being “very
unlikely” and 5 being “very likely.” The “mean rating”
is the average of of ratings for all 48 topics.
Rating Type

Mean Rating

Government Trouble Rating

1.95

Privacy-Sensitive Rating

2.01

Browser History Delete Rating

2.00

Avoidance Rating

2.62

====================
APPENDIX D

Topic Keyword – 48 Article Group
T o p ic Ke ywo rd

W ik ip e d ia Article s

Al Qaeda
terrorism
terror
attack
Iraq
Afghanistan
Iran
Pakistan
agro
Environmental terrorism
Eco terrorism
Conventional weapon
Weapons grade
dirty bomb
Nuclear Enrichment
Nuclear
Chemical weapon
Biological weapon
Ammonium nitrate
Improvised explosive device
Abu Sayyaf
Hamas
FARC
Irish Republican Army
Euskadi ta Askatasuna
Hezbollah
Tamil Tigers
PLO
Palestine Liberation Front
Car bomb
jihad
Taliban
Suicide bomber
Suicide attack

http://en.wikipedia.org/wiki/Al-Qaeda
http://en.wikipedia.org/wiki/terrorism
http://en.wikipedia.org/wiki/terror
http://en.wikipedia.org/wiki/attack
http://en.wikipedia.org/wiki/iraq
http://en.wikipedia.org/wiki/afghanistan
http://en.wikipedia.org/wiki/iran
http://en.wikipedia.org/wiki/pakistan
http://en.wikipedia.org/wiki/agro
http://en.wikipedia.org/wiki/Environmental_terrorism
http://en.wikipedia.org/wiki/Eco-terrorism
http://en.wikipedia.org/wiki/Conventional_weapon
http://en.wikipedia.org/wiki/Weapons-grade
http://en.wikipedia.org/wiki/Dirty_bomb
http://en.wikipedia.org/wiki/Nuclear_enrichment
http://en.wikipedia.org/wiki/nuclear
http://en.wikipedia.org/wiki/Chemical_weapon
http://en.wikipedia.org/wiki/Biological_weapon
http://en.wikipedia.org/wiki/Ammonium_nitrate
http://en.wikipedia.org/wiki/Improvised_explosive_device
http://en.wikipedia.org/wiki/Abu_Sayyaf
http://en.wikipedia.org/wiki/hamas
http://en.wikipedia.org/wiki/FARC
http://en.wikipedia.org/wiki/Irish_Republican_Army
http://en.wikipedia.org/w/Euskadi_ta_Askatasuna
http://en.wikipedia.org/wiki/hezbollah
http://en.wikipedia.org/wiki/Tamil_Tigers
http://en.wikipedia.org/wiki/Palestine_Liberation_Organization
http://en.wikipedia.org/wiki/Palestine_Liberation_Front
http://en.wikipedia.org/wiki/Car_bomb
http://en.wikipedia.org/wiki/jihad
http://en.wikipedia.org/wiki/taliban
http://en.wikipedia.org/wiki/Suicide_bomber
http://en.wikipedia.org/wiki/Suicide_attack

AL Qaeda in the Arabian Peninsula
Al Qaeda in the Islamic Maghreb
Tehrik-i-Taliban Pakistan
Yemen
Pirates
Extremism
Somalia
Nigeria
Political radicalism
Al-Shabaab
nationalism
Recruitment
Fundamentalism
Islamist
MEAN S

http://en.wikipedia.org/wiki/Al-Qaeda_in_the_Arabian_Peninsula
http://en.wikipedia.org/wiki/Al-Qaeda_in_the_Islamic_Maghreb
http://en.wikipedia.org/wiki/Tehrik-i-Taliban_Pakistan
http://en.wikipedia.org/wiki/yemen
http://en.wikipedia.org/wiki/pirates
http://en.wikipedia.org/wiki/extremism
http://en.wikipedia.org/wiki/somalia
http://en.wikipedia.org/wiki/nigeria
http://en.wikipedia.org/wiki/Political_radicalism
http://en.wikipedia.org/wiki/Al-Shabaab
http://en.wikipedia.org/wiki/nationalism
http://en.wikipedia.org/wiki/recruitment
http://en.wikipedia.org/wiki/fundamentalism
http://en.wikipedia.org/wiki/islamist

Go v e rnme nt
T ro ub le

Bro ws e r
D e le te

P riv a c y
Se ns itiv e

Av o id a nc e

2.20
2.19
1.98
1.92
1.60
1.61
1.62
1.59
1.51
2.20
2.22
2.03
2.18
2.72
2.22
1.84
2.43
2.44
2.49
2.82
2.02
1.90
1.83
1.62
1.86
1.86
1.76
1.77
1.81
2.72
2.15
2.06
2.25
2.30
2.01
2.05
1.96
1.60
1.44
1.64
1.50
1.48
1.75
1.84
1.48
1.74
1.60
1.79
1.95

2.11
2.05
1.96
1.91
1.74
1.71
1.73
1.71
1.80
2.20
2.20
2.16
2.22
2.55
2.21
1.97
2.36
2.39
2.44
2.64
1.96
1.93
1.88
1.77
1.88
1.90
1.86
1.87
1.89
2.61
2.19
2.03
2.31
2.36
1.98
1.98
1.96
1.72
1.67
1.90
1.68
1.66
1.91
1.89
1.71
1.90
1.79
1.89
2.00

2.21
2.16
2.01
1.92
1.76
1.75
1.78
1.75
1.76
2.24
2.22
2.07
2.17
2.50
2.21
1.91
2.39
2.39
2.26
2.53
1.99
1.97
1.90
1.83
1.88
1.96
1.87
1.91
1.95
2.50
2.17
2.10
2.24
2.29
2.06
2.06
1.97
1.74
1.67
1.86
1.67
1.64
1.97
1.89
1.73
1.87
1.80
1.93
2.01

2.84
2.79
2.64
2.56
2.25
2.23
2.25
2.22
2.29
2.92
2.92
2.81
2.99
3.45
2.92
2.55
3.16
3.18
3.24
3.46
2.57
2.49
2.46
2.24
2.43
2.46
2.39
2.42
2.47
3.40
2.89
2.70
2.97
3.04
2.63
2.60
2.59
2.18
2.10
2.40
2.12
2.07
2.48
2.48
2.20
2.54
2.32
2.45
2.62

 PENNEY_CHILLING EFFECTS_SSRN-POSTED-APRIL-26-2016-NON-TIF-GRAPHS (DO NOT DELETE)

2016]

4/26/201611:20 PM

CHILLING EFFECTS

55

====================
APPENDIX E

Topic Keyword – 31 Article Group
Co mb ine d
Priv a c y R a ting

T o p ic Ke y wo rd

W ik ip e d ia A rtic le s

Al Qaeda
terrorism
terror
Environmental terrorism
Eco terrorism
Conventional weapon
Weapons grade
dirty bomb
Nuclear Enrichment
Nuclear
Chemical weapon
Biological weapon
Ammonium nitrate
Improvised explosive device
Abu Sayyaf
FARC
Euskadi ta Askatasuna
Hezbollah
Palestine Liberation Front
Car bomb
jihad
Taliban
Suicide bomber
Suicide attack

http://en.wikipedia.org/wiki/Al-Qaeda
http://en.wikipedia.org/wiki/terrorism
http://en.wikipedia.org/wiki/terror
http://en.wikipedia.org/wiki/Environmental_terrorism
http://en.wikipedia.org/wiki/Eco-terrorism
http://en.wikipedia.org/wiki/Conventional_weapon
http://en.wikipedia.org/wiki/Weapons-grade
http://en.wikipedia.org/wiki/Dirty_bomb
http://en.wikipedia.org/wiki/Nuclear_enrichment
http://en.wikipedia.org/wiki/nuclear
http://en.wikipedia.org/wiki/Chemical_weapon
http://en.wikipedia.org/wiki/Biological_weapon
http://en.wikipedia.org/wiki/Ammonium_nitrate
http://en.wikipedia.org/wiki/Improvised_explosive_device
http://en.wikipedia.org/wiki/Abu_Sayyaf
http://en.wikipedia.org/wiki/FARC
http://en.wikipedia.org/w/Euskadi_ta_Askatasuna
http://en.wikipedia.org/wiki/hezbollah
http://en.wikipedia.org/wiki/Palestine_Liberation_Front
http://en.wikipedia.org/wiki/Car_bomb
http://en.wikipedia.org/wiki/jihad
http://en.wikipedia.org/wiki/taliban
http://en.wikipedia.org/wiki/Suicide_bomber
http://en.wikipedia.org/wiki/Suicide_attack

2.34
2.30
2.15
2.39
2.39
2.27
2.39
2.81
2.39
2.07
2.59
2.60
2.61
2.86
2.14
2.02
2.01
2.05
2.03
2.81
2.35
2.22
2.44
2.50

AL Qaeda in the Arabian Peninsula
Al Qaeda in the Islamic Maghreb
Tehrik-i-Taliban Pakistan
Political radicalism
Al-Shabaab
Recruitment
Islamist
ME A N

http://en.wikipedia.org/wiki/Al-Qaeda_in_the_Arabian_Peninsula
http://en.wikipedia.org/wiki/Al-Qaeda_in_the_Islamic_Maghreb
http://en.wikipedia.org/wiki/Tehrik-i-Taliban_Pakistan
http://en.wikipedia.org/wiki/Political_radicalism
http://en.wikipedia.org/wiki/Al-Shabaab
http://en.wikipedia.org/wiki/recruitment
http://en.wikipedia.org/wiki/islamist

2.17
2.17
2.12
2.03
2.03
2.01
2.02
2.30

====================
APPENDIX F

Topic Keyword – 25 Article Quasi Control Group
T o p ic Ke ywo rd

Wikip e d ia Article s

Department of Homeland Security
Federal Emergency Management Agency
Coast Guard
U.S._Customs_and_Border_Protection
Border patrol
Secret Service
National Security Operations Center
Homeland defense
Chemical weapon
Biological agent
Joint Task Force
Central Intelligence Agency
Fusion center
Drug Enforcement Administration
Secure Border Initiative
Federal Bureau of Investigation
ATF
U.S. Citizenship and Immigration Services
Federal Air Marshal Service
Transportation Security Administration
Air marshals
Federal Aviation Administration
National Guard
International Federation of Red Cross and Red Crescent
Societies
U.S. Immigration and Customs Enforcement

https://en.wikipedia.org/wiki/United_States_Department_of_Homeland_Security
https://en.wikipedia.org/wiki/Federal_Emergency_Management_Agency
https://en.wikipedia.org/wiki/Coast_guard
https://en.wikipedia.org/wiki/U.S._Customs_and_Border_Protection
https://en.wikipedia.org/wiki/Border_Patrol
http://en.wikipedia.org/wiki/United_States_Secret_Service
https://en.wikipedia.org/wiki/National_Security_Operations_Center
https://en.wikipedia.org/wiki/Homeland_defense
https://en.wikipedia.org/wiki/Chemical_weapon
http://en.wikipedia.org/wiki/Biological_agent
https://en.wikipedia.org/wiki/Joint_Task_Force
https://en.wikipedia.org/wiki/Central_Intelligence_Agency
https://en.wikipedia.org/wiki/Fusion_center
https://en.wikipedia.org/wiki/Drug_Enforcement_Administration
https://en.wikipedia.org/wiki/Secure_Border_Initiative
https://en.wikipedia.org/wiki/Federal_Bureau_of_Investigation
https://en.wikipedia.org/wiki/Bureau_of_Alcohol,_Tobacco,_Firearms_and_Explos
https://en.wikipedia.org/wiki/United_States_Citizenship_and_Immigration_Service
https://en.wikipedia.org/wiki/Federal_Air_Marshal_Service
https://en.wikipedia.org/wiki/Transportation_Security_Administration
https://en.wikipedia.org/wiki/Air_marshal
https://en.wikipedia.org/wiki/Federal_Aviation_Administration
https://en.wikipedia.org/wiki/National_Guard
https://en.wikipedia.org/wiki/International_Federation_of_Red_Cross_and_Red_Cr
escent_Societies
https://en.wikipedia.org/wiki/U.S._Immigration_and_Customs_Enforcement