New York State Office of the State Comptroller Thomas P. DiNapoli Division of State Government Accountability Marketing Service Performance Monitoring Empire State Development Report 2014-S-10 May 2015 2014-S-10 Executive Summary Purpose To determine whether Empire State Development (ESD) has established adequate internal controls to oversee, monitor, and manage contracted marketing services, including the extent to which ESD employs appropriate performance measurement systems that provide management with information about program effectiveness and cost-efficiency. Our audit scope covered the period December 1, 2011 through November 6, 2014. Background ESD’s mission is to promote a vigorous and growing economy, prevent economic stagnation, encourage the creation of new job opportunities, increase revenues to the State and its municipalities, and achieve stable and diversified local economies. To this end, ESD plans and conducts programs to promote travel, tourism, and business investment. In support of these programs, ESD awarded a contract in December 2011 to BBDO USA LLC (BBDO) for an amount not to exceed $50 million, as its non-exclusive, full-service advertising, marketing, branding, media, and communications agency. Due to the value of the contract and the appropriated State funds involved, it was reviewed by the Office of the Attorney General to ensure that it was in proper form and included all provisions mandated by law, and by the Office of the State Comptroller to ensure, among other things, that fair, proper procurement procedures were followed and that the cost of the procured services was reasonable on its face. Neither of these Offices was responsible for determining whether the contract was the most appropriate method of accomplishing the intended purpose or whether it would successfully achieve the program goals ESD intended, which in this instance were not set forth in the contract. Instead, ESD management remains ultimately responsible for ensuring that the contract is necessary and accomplishes its intended purpose, and for establishing a system of internal controls to monitor, oversee, and manage the contract. These controls should include employing a performance measurement system that provides management with timely and accurate information about contract performance and outcomes. By September 2014, ESD had executed four amendments to this contract, bringing the total contract amount to $211.5 million. Of this, $36.5 million is specifically targeted to promote tourism and business in the wake of Hurricane Sandy. The remaining $175 million is available to be spent at ESD’s discretion. As of October 2014, ESD had committed $182.9 million of the $211.5 million, including $33.2 million of the $36.5 million set aside for post-Hurricane Sandy campaigns. Key Findings • ESD has an appropriate system of internal controls in place to ensure that it receives the advertising services for which it paid, and that those services are appropriately priced in keeping with the terms of its contract with BBDO. However, these controls focus on the specific services that are provided (i.e., outputs) rather than on the results that are achieved (i.e., outcomes). • ESD has not quantified what it expects to achieve from its advertising efforts, except in the broadest terms like increasing tourism or creating jobs. As a result, ESD does not have an appropriate system to monitor, measure, and evaluate the extent to which any accomplishments Division of State Government Accountability 2 2014-S-10 or outcomes resulting from these efforts compare to expectations. • ESD is unable to evaluate the extent to which its $211.5 million planned investment has contributed toward achieving the purposes of the underlying programs or whether it has been cost-effective. In fact, ESD officials reject the idea that advertising programs should be measured against the results achieved by the underlying programs they aim to benefit, except in the broadest terms. • ESD does track certain measures that officials believe are indicative of possible program impact, including website traffic, attendance at certain tourism attractions, and business leads developed. However, officials consider any reported improvement in these measures, as well as other factors such as the extent to which people perceive New York to be a good place for business development or to visit, as evidence that the advertising programs are a success. In doing so, ESD officials have not considered ways to account for any other factors that may influence these measures, or to assess whether the State has received an appropriate return on its investment in these marketing services. Key Recommendations • Develop strategic plans that include performance measures for monitoring the extent to which marketing efforts have a positive impact on desired outcomes, such as ESD’s stated goal of improving the perceptions of New York as a good place to visit and for business development. • Set specific targets, goals, and benchmarks for evaluating performance outcomes and use these measures to monitor program performance. • Regularly evaluate the program outcomes associated with marketing efforts and use this information to periodically adjust program goals, strategies, and resource allocations. Authority Response In responding to our draft report, ESD officials reiterated many of the positions they put forth during our audit. However, ESD’s response avoids addressing the core issue of this report: the fact that ESD has not established any measurable goals or outcomes that quantify the progress it expects to achieve through its advertising efforts. As a result, ESD is unable to determine the extent to which program outcomes meet expectations or are commensurate with the significant State resources devoted to these efforts. In fact, these critical factors are never mentioned in ESD’s response. ESD’s effort to deflect attention away from the core issues of strategic planning, performance outcomes, and accountability for cost-effectiveness is further evidenced by the fact that officials respond to five specific items that they label as “OSC RECOMMENDATIONS,” none of which resemble the three actual recommendations included in this audit. Other Related Audit/Report of Interest Empire State Development: Oversight of International Offices (2012-S-7) Division of State Government Accountability 3 2014-S-10 State of New York Office of the State Comptroller Division of State Government Accountability May 11, 2015 Mr. Howard Zemsky President and CEO Empire State Development 633 Third Avenue New York, NY 10017 Dear Mr. Zemsky: The Office of the State Comptroller is committed to helping State agencies, public authorities, and local government agencies manage their resources efficiently and effectively. By so doing, it provides accountability for tax dollars spent to support government operations. The Comptroller oversees the fiscal affairs of State agencies, public authorities, and local government agencies, as well as their compliance with relevant statutes and their observance of good business practices. This fiscal oversight is accomplished, in part, through our audits, which identify opportunities for improving operations. Audits can also identify strategies for reducing costs and strengthening controls that are intended to safeguard assets. Following is our audit report entitled Marketing Service Performance Monitoring. The audit was performed pursuant to the State Comptroller’s authority under Article X, Section 5, of the State Constitution and Section 2803 of the Public Authorities Law. This audit’s results and recommendations are resources for you to use in effectively managing your operations and in meeting the expectations of taxpayers. If you have any questions about this report, please feel free to contact us. Respectfully submitted, Office of the State Comptroller Division of State Government Accountability Division of State Government Accountability 4 2014-S-10 Table of Contents Background 6 Audit Findings and Recommendations 7 Monitoring Advertising Services and Payments 7 Monitoring Impact on Tourism and Economic Development 8 Recommendations 12 Audit Scope and Methodology 13 Authority 13 Reporting Requirements 14 Contributors to This Report 15 Authority Comments 16 State Comptroller’s Comments 44 State Government Accountability Contact Information: Audit Director: John Buyce Phone: (518) 474-3271 Email: StateGovernmentAccountability@osc.state.ny.us Address: Office of the State Comptroller Division of State Government Accountability 110 State Street, 11th Floor Albany, NY 12236 This report is also available on our website at: www.osc.state.ny.us Division of State Government Accountability 5 2014-S-10 Background The New York State Urban Development Corporation, now doing business as Empire State Development (ESD), was formed by legislative act in 1968 to address conditions of unemployment, underemployment, and blight which impede the economic and physical development of municipalities, increase the burdens on the State, and adversely affect the welfare and prosperity of the State’s citizens. ESD seeks to promote a vigorous and growing economy, prevent economic stagnation, encourage the creation of new job opportunities, increase revenues to the State and its municipalities, and achieve stable and diversified local economies. To advance these goals, ESD plans and conducts programs to promote travel, tourism, and business investment. To facilitate this effort, ESD awarded a contract to BBDO USA LLC (BBDO) as its non-exclusive fullservice advertising, marketing, branding, media, and communications agency. The purpose of the services supported by the contract, as reported to the Public Authorities Reporting Information System, is “to create a campaign that will position New York State favorably in the global marketplace to spur investment, job creation, and income generation in New York’s economic rebuilding process.” The initial contract was awarded in December 2011 for an amount not to exceed $50 million. Due to the value of the contract and the appropriated State funds involved, it was reviewed by the Office of the Attorney General to ensure that it was in proper form and included all provisions mandated by law, and by the Office of the State Comptroller to ensure, among other things, that fair and proper procurement procedures were followed and that the cost of the procured services was reasonable on its face. Neither of these Offices was responsible for determining whether the contract was the most appropriate method of accomplishing the intended purpose or whether it would successfully achieve the program goals ESD intended, which in this instance were not set forth in the contract. Instead, ESD management remains ultimately responsible for ensuring that the contract is necessary and accomplishes its intended purpose. To this end, ESD management is also responsible for establishing a system of internal controls to monitor, oversee, and manage the contract. These controls should include employing a performance measurement system that provides management with timely and accurate information about contract performance and outcomes. As of September 2014, ESD had executed four amendments to this contract, bringing the total contract amount to $211.5 million. Of this, $36.5 million is specifically targeted to promote tourism and business in the wake of Hurricane Sandy. The remaining $175 million is available to be spent at ESD’s discretion. As of October 2014, ESD had committed $182.9 million of the $211.5 million, including $33.2 million of the $36.5 million set aside for post-Hurricane Sandy campaigns. Division of State Government Accountability 6 2014-S-10 Audit Findings and Recommendations ESD has an appropriate system of internal controls in place to ensure both that it receives the advertising services for which it paid, and that those services are appropriately priced in keeping with the terms of its contract with BBDO. However, these controls focus on the specific services that are provided, such as advertisements created and placed (i.e., outputs), rather than on the results that are achieved, such as increased employment (i.e., outcomes). This situation exists primarily because ESD did not quantify what it expected to achieve from its advertising efforts, except in the broadest terms like increasing tourism or creating jobs. As a result, ESD does not have an appropriate system to monitor, measure, and evaluate the extent to which any accomplishments or outcomes resulting from these efforts compare to expectations. Most importantly, ESD is unable to evaluate the extent to which its $211.5 million planned investment has contributed toward achieving the purposes of the underlying programs or whether it is cost-effective. ESD officials indicated they do not believe that advertising programs should be measured in any way against the results achieved by the underlying programs. Rather, officials consider any reported improvement in certain factors, such as the perception of New York as a good place for business development or to visit, to be an indication that the advertising programs are a success. ESD officials have not considered ways to account for any other factors that may influence these measures or to assess whether the State has received an appropriate return on its investment in those marketing services. Monitoring Advertising Services and Payments As of September 2014, ESD had identified five different economic development programs to be covered under the BBDO contract: Masterbrand, Taste NY, START-UP NY, Hurricane Sandy, and Tourism. There is no formal allocation of the total contract amount among these five programs. ESD officials indicate that this was done purposefully to allow them the most flexibility in allocating these resources and determining which programs would benefit most from additional advertising services beyond the levels that the programs themselves may provide. In keeping with this philosophy, ESD has not developed an overall strategic plan for the marketing services covered by the BBDO contract, nor has it developed an overall budget for each individual program. Instead, ESD relies on the marketing expertise of BBDO, and of its own employees, to determine the most effective allocation of contract funding. According to ESD officials, the advertising done under the BBDO contract is not intended to directly produce economic results, such as increasing tourism or creating jobs. Instead, advertising is intended to support these programs by improving public perceptions of New York as a place to visit and for business development. ESD expects that improved public perception will indirectly lead to more tourism and job creation as more individuals and businesses consider New York an attractive place to visit and do business. As a result, its internal controls in this area focus mainly on service delivery outputs, that is, ensuring that ESD receives all the services and related advertising it pays for, and that they are fairly priced in accordance with the terms of the contract. Division of State Government Accountability 7 2014-S-10 Under ESD’s system, BBDO submits a quarterly work plan detailing what advertising projects it proposes to develop for each economic development program that ESD has identified. The work plan specifies what advertising will be produced, any key assumptions, and what support is needed from ESD. ESD then reviews and approves these quarterly work plans before BBDO implements them. According to ESD officials, this system allows them the most flexibility to react to changing State priorities, programs, and initiatives. After a quarterly work plan is reviewed and approved, BBDO prepares cost estimates for each individual project to accomplish the work outlined in the plan. ESD officials review these estimates and may suggest changes where they deem necessary. Once an estimate is agreed to by both parties, it is approved and becomes the basis for future billings and payments. Upon completion of the project, BBDO submits an invoice to ESD, along with all the supporting documentation, to receive the final payment for the project. For certain media purchases, BBDO also provides certifications from the television or radio stations regarding the dates and times that commercials aired, along with the estimated number of viewers and listeners. If the estimated number of viewers or listeners is less than contractual requirements, BBDO will ensure that the stations air the commercials again at no additional charge to ESD. Early in the contract, ESD had been advancing almost all of the funds for media purchases to BBDO at the time project estimates were approved. However, such a system not only requires significant effort to reconcile at the close of an advertising program, but when some projects inevitably get delayed, cancelled, or significantly scaled back, funding is needlessly tied up for an extended period of time. ESD recognized these issues and has since moved to a system where funds are not advanced until much closer to the date the services will be delivered. We performed various tests of ESD’s payments, comparing them to billings and documentation provided by BBDO and to approved project estimates, as well as independent testing of billings and payment reconciliations completed by ESD. We also met with BBDO officials to understand the processes they used to place, price, and monitor the actual delivery of advertising services under the contract. Generally, we found that staff at both ESD and BBDO were complying with the control procedures as designed and those controls provided ESD with reasonable assurance that the advertising services it pays for have been received from BBDO and are fairly priced in accordance with the contract and general industry practices. Monitoring Impact on Tourism and Economic Development ESD officials consider the BBDO contract to be an integral part of the State’s overall economic development strategy. At the same time, officials stress that marketing in and of itself is not expected to create jobs or increase tourism. Instead, ESD officials indicate that the primary expectation they have for this contract is to improve the perceptions of the State as a place to visit and for business development. As a result, the BBDO contract does not include specific outcome-oriented performance goals or even specify which economic development programs should receive advertising services. When the contract with BBDO was first developed at the $50 million level, ESD made a conscious Division of State Government Accountability 8 2014-S-10 decision not to include specific performance measures or outcome-oriented deliverables. According to ESD officials, this gives them the greatest flexibility to react to changing State priorities and even to terminate the contract if necessary. As the contract has grown, this desired level of flexibility, and the resulting ambiguity in the actual services to be provided, makes it even more critical that ESD know precisely what outcomes it wants to achieve from the contract and that it monitor performance in these areas closely so it can redirect resources if goals are not being met. This cyclical approach to performance monitoring – where goals are established and strategies devised, resources committed, performance measured, and then goals and resources re-evaluated – is a cornerstone of good strategic planning. We found that, although ESD does develop quarterly work plans in conjunction with BBDO, it has not developed a comprehensive strategic plan that quantifies precisely what outcomes it seeks to achieve through what has now become a planned investment of $211.5 million. As a result, ESD has no parameters in place to guide how much it should spend on marketing to achieve any specific outcome. Most importantly, ESD is unable to assess whether the State has received an appropriate return on its investment in these marketing services through significantly improved perceptions of the State. Further, ESD does not assess its continued investment in terms of other more traditional measures such as cost per new job, tourism market share, or increased economic activity. Although ESD officials indicate that the marketing programs exist to support the broader results of the various economic development programs they serve, they do not believe that success should be measured by the outcomes of those programs, except in the most general terms. As a result, in lieu of specific performance measures, ESD has identified certain activities that it routinely monitors as indicators of public interest. Some of these indicators include visits to relevant websites operated by the State; purchases of travel guides and maps; visits to the State’s Facebook, Twitter, and other websites; and employment statistics. ESD also occasionally gathers data on other economic and tourism indicators to assess whether trends are moving in the proper direction (e.g., attendance at specific tourism venues or targeted surveys assessing traveler and business attitudes). ESD has also not identified benchmarks or set targets with which to assess these indicators. Instead, they define successful performance by BBDO as any positive change to those indicators. Further, ESD does not take into account factors outside BBDO’s marketing efforts that may affect these indicators, such as broad economic trends or other State and local economic development initiatives. ESD has not developed a methodology for identifying what portion of the changes to Division of State Government Accountability 9 2014-S-10 these indicators is attributable to the advertising services received from BBDO. Although this may be a difficult task given the multiple variables involved, ESD, as the administrator of this contract, must be able to determine the extent to which its investment in advertising services is producing effective and cost-efficient results. On several occasions, we met with ESD officials to discuss goals and objectives of the various programs, as well as targeted outcomes and performance measures used to assess progress, accomplishments, and return on investment. Of the five programs currently covered by the BBDO contract, ESD only provided details about one: START-UP NY, which began in October 2013. This program encourages qualified businesses to start, expand, or relocate operations to certain areas of the State in exchange for a ten-year exemption from taxes. Legislation governing the program authorizes as many as 10,000 new jobs per year to be created by the effort. ESD officials indicated marketing services in this area were therefore intended to encourage businesses to apply for the program, although they stressed that it is up to the program director and staff to evaluate those applications, verify eligibility, and determine program success. Between October 2013 and October 2014, ESD committed $45.1 million to advertise the STARTUP NY program, which is 40 percent of the total $111.6 million committed to all advertising services under the BBDO contract during those 13 months. Based on the annual cap of 10,000 jobs authorized by the Legislature to receive benefits, this would equate to marketing costs of about $4,500 per job created in association with the program. During that time, ESD records indicate it received 18,203 applications for START-UP NY. Of those, only about 10 percent (1,843) came from businesses that were actually eligible for the program. Through the close of our fieldwork, ESD had received pledges from 41 of these businesses to create a total of 1,750 jobs over the next five years, which works out to a marketing cost of over $25,000 per job. ESD officials told us the marketing program for START-UP NY was a success and they consider the investment in advertising services to be worthwhile. However, they were unable to provide any analysis to support their conclusion. Thousands ESD provided us with detailed information on funding commitments made, and applications received, during the first nine months of the START-UP NY program, from October 2013 through June 2014. In total, it committed about $33.4 million to funding advertisements during this period and received slightly more than 15,000 START-UP NY - Program Applications Received applications. Our 6 analysis found that the 5 number of applications 4 for START-UP NY fell 3 from a peak of nearly 2 5,300 during January 1 2014 to slightly more than 500 in June 2014. By Month Oct-13 832 Nov-13 Dec-13 287 3,944 Division of State Government Accountability Jan-14 5,284 Feb-14 Mar-14 Apr-14 May-14 Jun-14 2,118 811 769 483 518 10 2014-S-10 START-UP NY - Marketing Funds Committed Millions Despite this significant and persistent decline in applications, ESD allocated more and more funding to advertise this program during the intervening months, reaching a total of $33.4 million by the end of June 2014. $14 $12 $10 $8 $6 $4 $2 $0 Over the next Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14 four months, ESD committed another $11.7 million to START-UP NY advertising and received another 3,157 applications. Had ESD established and monitored specific performance targets consistent with its stated intention of encouraging businesses to apply for the program, it may have considered allocating resources differently as the apparent interest declined. When we discussed these observations with ESD officials, they indicated that the marketing likely benefited other programs as well. For example, officials told us some applicants were referred to other economic development programs because they either did not qualify for START-UP NY or were better suited for those other initiatives. However, ESD did not provide us with any information on those referrals or the resulting economic impact through those other programs. Even though ESD did not set specific targets or benchmarks against which to evaluate performance, we found officials did make some efforts to assess the quality of the advertising services it purchased and, in turn, their potential for impact. In particular, ESD commissioned three surveys of focus groups to gauge which commercials would most likely impact public perceptions of the State as a desirable place to visit and do business. Two of these surveys were conducted in 2013 and one in 2014. In each case, a group of people were shown a series of ads and asked to respond to questions about each one. ESD officials provided us with results of these surveys, two of which were conducted to gauge the potential impact of the START-UP NY advertising and one to assess the tourism advertising. These tests did not involve ads created for the other economic development programs advertised through the BBDO contract (Masterbrand, Taste NY, and Hurricane Sandy). As with the other indicators that ESD monitors, officials did not set targets or goals for how much they thought these perceptions should increase, only that the perceptions should improve. The results of the START-UP NY surveys showed some evidence that viewers had a more positive perception of New York as a place for business development after viewing the advertisements. Depending on the commercial, up to 57 percent of business representatives reported a positive impression of New York’s business climate, as did up to 39 percent of individuals. However, the tourism survey showed that the advertising services had little or no impact on perceptions of Division of State Government Accountability 11 2014-S-10 New York as a place to visit. Groups of individuals drawn from New York City, Philadelphia, and Ontario, Canada did not report an increase in the likelihood that they would visit Upstate New York after viewing the tourism ads. Even so, ESD officials pointed to five tourism destinations featured in its marketing campaign and the related increases in attendance reported at those venues from 2013 to 2014 as evidence that the ads had been effective. However, our review showed that, in evaluating this information, ESD assumed all of the reported gains were due solely to its advertising efforts and made no attempt to consider any other factors that could have positively affected the attendance at these destinations. Among the gains cited were: • The Baseball Hall of Fame: 17 percent increase in attendance in 2014. Attendance at the Baseball Hall of Fame is routinely affected by the inductees in any particular year. The inductees for 2013 (Jacob Ruppert, Hank O’Day, and Deacon White) all died prior to 1940 and were therefore lesser known to current baseball fans. In contrast, the inductees in 2014 (Bobby Cox, Tony La Russa, Joe Torre, Tom Glavine, Greg Maddux, and Frank Thomas) were all well-known players and managers of more current eras and thus more widely recognized by even casual baseball fans. In fact, reports project even greater attendance for 2015 as several recently retired pitchers (Randy Johnson, Pedro Martinez, and John Smoltz) are scheduled to be inducted. • Whiteface Mountain: 13 percent increase in attendance. Attendance at ski slopes is affected by the weather and by snow depth. Published reports show 2014 produced more snow and greater average snow depths at Whiteface than did 2013. In addition, according to the Olympic Regional Development Authority, Whiteface was open for about 3 percent more days in 2014 than it was in 2013. • Jones Beach: 14 percent increase in attendance. Attendance at beaches is also affected by the weather. Jones Beach was open for ten weekends during both 2013 and 2014. According to weather service reports, there was measurable rain during six of these weekends in 2013, while only four had precipitation the following year. As ESD moves forward with its future marketing efforts, we believe it is critical that officials develop strategic plans that include performance measures which can demonstrate whether advertising is having a positive impact on specific desired outcomes, such as the perception of New York as a good place to visit and for business development. ESD also needs to identify benchmarks and set targets against which to evaluate performance to ensure that the marketing services purchased under the BBDO contract have been effective. Officials also must determine if that impact is meeting expectations and is commensurate with the allocated funding. As ESD officials have pointed out, the success of each individual economic program in meeting its goals lies with the program’s director, not with the BBDO contract. However, ESD should determine whether the programs it is advertising through the BBDO contract are benefiting from those services sufficiently to justify the money spent. Recommendations 1. Develop strategic plans that include performance measures for monitoring the extent to which Division of State Government Accountability 12 2014-S-10 marketing efforts have a positive impact on desired outcomes, such as ESD’s stated goal of improving perceptions of New York as a good place to visit and for business development. 2. Set specific targets, goals, and benchmarks for evaluating performance outcomes and use these measures to monitor program performance. 3. Regularly evaluate the program outcomes associated with marketing efforts and use this information to periodically adjust program goals, strategies, and resource allocations. Audit Scope and Methodology We audited ESD to determine whether it has established adequate internal controls to oversee, monitor, and manage contracted marketing services, including the extent to which it employs appropriate performance measurement systems that provide management with information about program effectiveness and cost-efficiency. Our audit covers the period from December 1, 2011 through November 6, 2014. To accomplish our audit objective and assess related internal controls, we reviewed ESD’s procurement policies and procedures and interviewed ESD officials and employees. We also reviewed contracts, contract amendments, financial statements, invoices and documentation supporting invoiced amounts, and survey results. We also interviewed BBDO employees and reviewed documentation BBDO maintains on its contract with ESD and the invoices it has submitted. We conducted our performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. In addition to being the State Auditor, the Comptroller performs certain other constitutionally and statutorily mandated duties as the chief fiscal officer of New York State. These include operating the State’s accounting system; preparing the State’s financial statements; and approving State contracts, refunds, and other payments. In addition, the Comptroller appoints members to certain boards, commissions, and public authorities, some of whom have minority voting rights. These duties may be considered management functions for purposes of evaluating organizational independence under generally accepted government auditing standards. In our opinion, these functions do not affect our ability to conduct independent audits of program performance. Authority The audit was performed pursuant to the State Comptroller’s authority under Article X, Section 5, of the State Constitution and Section 2803 of the Public Authority Law. Division of State Government Accountability 13 2014-S-10 Reporting Requirements A draft copy of this report was provided to ESD officials for their review and comment. At their request, we extended the normal 30-day comment period established by statute to provide ESD with an additional 30 days to consider our findings and formulate their response to the issues discussed in this report. A complete copy of their response is attached at the end of this report, including a research report executive summary that officials included with their response. Our rejoinders to specific issues discussed in the response are also attached as State Comptroller’s Comments. In responding to our draft report, ESD officials reiterated many of the positions that they put forth during the course of our audit. Most notably, officials assert that the success of an advertising program should not be gauged by the results that are achieved by the underlying programs it is intended to benefit, and that ESD in turn tracks various factors like website traffic which it views as indicators that advertisements are generating interest in New York State and therefore are successful. We acknowledge ESD’s efforts and discussed them at length in various sections of this report. At the same time though, ESD’s response avoids addressing the core issue of this report: the fact that ESD has not established any measurable goals or outcomes that quantify the progress it expects to achieve through its advertising efforts, even as it relates to ESD’s broad vision to generate interest and shift perceptions. When viewed in its entirety, ESD’s response provides further evidence of our conclusion that officials consider any reported improvement to be evidence that the advertising programs are successful, without any regard for program cost (to date $211.5 million) or the extent of return on their investment. In fact, these critical factors are not mentioned in ESD’s almost 30-page response. ESD’s effort to deflect attention away from the core issues of strategic planning, performance outcomes, and accountability for cost-effectiveness is further evidenced by the fact that officials respond to five specific items that they label as “OSC RECOMMENDATIONS,” none of which resemble the three actual recommendations included in this audit. We therefore remind officials that Section 170 of the Executive Law requires that, within 90 days of the final release of this report, the President and CEO of Empire State Development must report to the Governor, the State Comptroller, and the leaders of the Legislature and its fiscal committees, advising what steps were taken to implement the recommendations contained herein, and where the recommendations were not implemented, the reasons why. Our expectation is that ESD will comply with this requirement and, in doing so, be more responsive to the core issues at hand. Division of State Government Accountability 14 2014-S-10 Contributors to This Report John F. Buyce, CPA, CIA, CFE, CGFM, Audit Director Walter Irving, Audit Manager Jennifer Paperman, CPA, CIA, Audit Supervisor Brandon Ogden, Examiner-in-Charge Peter Carroll, Staff Examiner Andrew Philpott, Staff Examiner Division of State Government Accountability Andrew A. SanFilippo, Executive Deputy Comptroller 518-474-4593, asanfilippo@osc.state.ny.us Tina Kim, Deputy Comptroller 518-473-3596, tkim@osc.state.ny.us Brian Mason, Assistant Comptroller 518-473-0334, bmason@osc.state.ny.us Vision A team of accountability experts respected for providing information that decision makers value. Mission To improve government operations by conducting independent audits, reviews and evaluations of New York State and New York City taxpayer financed programs. Division of State Government Accountability 15 2014-S-10 Authority Comments * Comment 1 *See State Comptroller’s Comments, Page 44. Division of State Government Accountability 16 2014-5-10 Empire State Development 633 Third Avenue New York City, NY 10017 (212] 803-3130 1. Building website traf?c to encourage people to learn more about New York State; 2. Changing perceptions of New York State through improved ratings on the State as a tourism destination or a place to operate a business; and 3. Increasing consideration of New York State as a place to visit or move/ expand a business. Our measurements demonstrate clear success of the marketing campaigns in meeting these criteria. Some examples follow. Business Development . Marketing has been extremely successful at driving traf?c to with a 530% visitor increase when we are on-air compared to periods in the same year when New York State did not run advertising.2 - marketing efforts have successfully reached business executives with one-half Comment of New York State-based executives and one-quarter from out-of-state 2 recalled having seen one or more commercials from the Start-Up NY campaign. . There has been a strong positive shift in perceived momentum of New York State?s business climate. One-half of professionals believe the state is moving in the right direction, a 72% increase in just 12 months after the launch of the campaign. Tourism . Periods with a media campaign spend saw an average weekly website traf?c increase of 48% compared to periods in the same year when the State did not run advertising. - New York State is the most top-of?mind destination for summer vacations and or getaways as one-half of New York state residents and two-fifths who live outside the state named New York State as a vacation destination on an unprompted basis. In sum, Open for Business marketing is consistently evaluated against performance metrics that indicate very strong results for our marketing efforts. Further, we are con?dent that OFB marketing has helped drive New York State to its lowest level of unemployment, the success of programs like Start-Up NY and the ongoing growth in the 3 tourism industry. Comment 2 All sources for statistics in the Overview can be found in the Discussion section below. 2 ?ange? i Empire State . Alt Development Division of State Government Accountability 17 2014-5-10 Empire State Development {533 Third Avenue New York City, NY 10017 (212) 803-3130 ll. Res on aft Re ort ESD has carefully reviewed the findings in the Draft Report. This response contains three parts. First, we review the history of this audit, the changes in the scope of the audit, the findings made to date and the separate approval process undertaken by contract division to authorize the initial contract and three contract extensions, including the most recent extension in November of 2014. Second, we address audit ?ndings. Finally, we provide detailed responses to key recommendations. A. nd Overview of the Audit 0505 audit was effectively commenced by Audit Manager Walter J. Irving?s letter dated May 27, 2014 to former ESD President and CEO Kenneth Adams. This letter informed BSD that OSC would be auditing "marketing services, including its contract with BBDO USA, The audit would, as we understood from the letter and from the opening conference on June 6, 2014, examine "whether the performance measures established are effective and cost-efficient, program expenditures represent reasonable and necessary costs, and revenue generation opportunities are being realized." Over the ensuing eight months, OSC conducted a thorough review of ESD's full-service marketing and advertising agency BBDO. review included many meetings, conference calls and correspondence between OSC and ESD staff, as well as meetings with BBDO. OSC favorably resolved a number of inquiries based on its thorough review of management of the BBDO contract. 05C confirmed the following in the Draft Report: that the BBDO contract was properly and competitively procured; [ii] that fair and proper procurement procedures were followed in the selection of that the BBDO contract is in proper form and includes all provisions mandated by law;3 [iv] that the cost of the procured services are fair and proper on their face; that ESD has an appropriate system of internal controls in place to ensure that it receives the advertising services for which it pays; and [vi] that those services provided by BBDO are appropriately priced in keeping with the terms of the contract. We are in agreement with these findings and believe that they re?ect the important safeguards and oversight that ESD has put in place with respect to this contract. 3 This determination is made by the Of?ce of the Attorney General under the procedures established in connection with review of public authority contracts under section 2879-21 of the Public Authorities Law. 3 ?ew? Empire State ??gim Development Division of State Government Accountability 18 2014-5-10 Empire State Development 633 Third Avenue New York City, NY 10017 (212) 803-3130 Having favorably concluded its review of ESD procurement and management of the BBDO contract, the Draft Report focuses primarily on how ESD monitors performance of the BBDO contract and what ESD expects to achieve from its advertising efforts, beyond the broad goals of ?increasing tourism or creating jobs.? Draft Report, Executive Summary, p. 2 [emphasis in original). OSC asserts in the Draft Report that ESD has not satisfactorily Comment quanti?ed or measured its market goals and makes five "key recommendations? to ESD for 4 future marketing efforts. We address each of the ?ndings and recommendations in the Draft Report in detail below. We will also address what we believe is a fundamental misunderstanding upon which OSC bases these findings and recommendations about marketing service contracts and the role that the BBDO contract plays in the State?s economic deveIOpment strategy. Finally, we review the information provided to OSC in the contract approval process. It should be noted from the outset that 0505 Contracts Division conducted its own independent review before approving the BBDO contract at its inception in November, 2011, and before approving four subsequent amendments adding funds to the contract Comment [most recently in December, 2014). Specifically, 0505 review of public authority contracts is conducted pursuant to the regulations promulgated by OSC that govern its review and 5 approval of agreements such as the ESD-BBDO contract, under section 2879-a of the Public Authorities Law. One of the required criteria for approval of such contracts is that ?the terms of the agreement are reasonable and in the best interests of the authority.?4 Contracts Division requested performance metrics from ESD as part of the 2014 review and. as discussed in detail below, approved the contract. It is our hope that OSC will concur with the conclusions reached by Contracts Division: that the contract has been and continues to be reasonable and in the best interests of BSD, and ful?lls its mandate to the public. B. FINDINGS in and NewYork? Develo At the outset of the Cuomo Administration, New York State faced a number of significant impediments to economic growth, and ultimately job growth. These included high taxes, deferred infrastructure investment, an oversized state government and poor fiscal disciplines. These impediments created policy challenges for the new 4 2 N.Y.C.R.R., Part mentors). 4 Taiwan Empire State Development Division of State Government Accountability 19 2014-5-10 Empire State Development 633 Third Avenue New York City, NY 1001? (212} 803-3130 Administration. The Cuomo Administration put in place a set of programmatic and legislative measures to eliminate these impediments. But, these measures could not succeed if businesses and visitors had poor perceptions of New York State. The Administration needed to improve the perception of New York State and encourage businesses and visitors to give the state another look and a second chance. This task became more critical, given the lack of marketing activity in the years prior to the Cuomo administration. Reported New York State tourism and business attraction marketing spending totaled $24 million between 2006and 2010, compared to over $400 million in spending by top state competitors. Accordingly, the Administration developed an ambitious marketing plan as part of its economic development strategy. The Administration?s economic development strategy includes the creation of Regional Economic Development Councils, Start-Up NY, a property tax rate cap, the installation of broadband across the State, a smaller State government, and a host of other programs and policies. The strategy also uses marketing to promote New York?s new business-friendly environment and programs. The focus of our marketing effort is to improve the perceptions of New York State and thereby increase the likelihood of consideration to move, expand, start a business or just take a vacation in the State. Through our marketing, we promote the great reasons to visit or grow a business here. In other words, marketing is aimed at putting New York into what in the marketing world is considered ?the consideration set.? Accordingly, BSD has developed metrics for the success of our marketing campaigns and consistently measures growth in website visits, positive perceptions of the State, and increased consideration of New York State as a destination. OSC's Draft Report characterizes the nature of marketing as an economic development program that should in and of itself produce jobs or direct investment. We disagree. We do not believe that our marketing program can directly produce jobs or investments. Instead, it is intended to change perceptions, in the mold of brand marketing, and is akin to marketing costly or complex products or services. This may be contrasted with a marketing campaign for a speci?c low-cost product like toothpaste or a lottery ticket. In a perception or brand marketing campaign, the goal is to encourage individuals who are ?in the market? to consider your product. Often the consideration will include online and other research, consultation with friends, comparison price shopping and other factors. The marketing campaign should increase the number of individuals and businesses considering New York State as a place to open or expand a business or to travel. The role of perception or brand marketing is to ensure that potential businesses or tourists consider New York, instead of dismissing it or ignoring it as they may have done in the past, as they pare-down possible places for a vacation or to locate a business. marketing efforts are directed at this process of evaluation, improving the perception of New York as a 5 Ail-Vim? I Empire State .?lAviUr ?mm Development Division of State Government Accountability 20 2014-5-10 Empire State Development 633 Third Avenue New York City, NY 10017 (212} 803-3130 destination worthy of consideration. The success of this effort can be, and has been, properly and purposefully monitored. nitorin iness Dev] mentM tin The overall objective of our business development marketing has been to build the New York brand through the creation of a compelling message that would first drive traffic to our State websites [where there is more information about New York State economic development programs than can be conveyed through advertising] and to then begin the long-term process of changing entrenched perceptions that NY State is unfriendly to business. Accordingly, our economic development advertising was directed to business leaders inside the State (to drive consideration for retention and business expansion] and outside the State [to drive consideration for relocation]. ESD has implemented the marketing campaign in three separate phases: First, the marketing campaign communicated that there is a new attitude towards business in New York State. These ads featured Robert DeNiro and introduce the "new? New York with testimonials from companies that benefitted from the State?s new programs. Second, the marketing campaign provided specific examples of how the State?s new approach to business was working and showed that the ?new New York? was on the move. The third phase of the marketing campaign focused on Start-Up NY, highlighting this innovative economic development program as evidence of the State?s commitment to reduce taxes, the single greatest barrier to New York State consideration by out-of?state businesses, and to reinforce a positive attitude towards businesses. Evi nce of Effe 'veness Measuring the impact of marketing on brand building is a passionately discussed topic in marketing literature. There is, however, general agreement that long-term brand value should be measured through perceptual shifts [rather than pro?t or sales volume] and that investment in brand marketing is a long-term investment on behalf of a service or business.5 5 This perspective is evident in the academic literature. Se_e, http:z {wwmetro- asng It is also the subject of commercial white-papers from companies like Survey Monkey. brand-advertising, as well as industry associations such as the AMA. See, eg. 6 i Empire State ?oswmv Development Division of State Government Accountability 21 2014-S-10 * Comment 6 * Comment 6 * Comment 7 Division of State Government Accountability 22 2014-5-10 Empire State Development 633 Third Avenue New York City, NY10017 (212} 803-3130 4. The current Start-Up NY advertising is also generating leads for Start-Up NY and for other State programs. To date, 18,300 businesses or individuals have expressed interest in Start-Up NY by ?lling out a detailed questionnaire to determine their level of Comment quali?cation. As a result, more than 2,100 qualified leads have been shared with ESD 8 personnel who manage Start-Up NY, over 110 leads have been provided to Small Business Assistance programs, and close to 1,200 leads have been channeled to the Incubator program for start-up companies. OSC has expectations for the marketing program that do not match the goals for our campaign or for any campaign that is seeking to build a brand and change perceptions. Advertising cannot close a deal or change general economic circumstances that impact corporate job creation. What marketing efforts can accomplish is to create interest and Openness on the part of business owners to do business in New York. Our business deveIOpment marketing is driving website traffic, changing perceptions, increasing consideration and helping generate leads. These are the only metrics appropriate to evaluate our effectiveness, and on these metrics we have succeeded. These are the results that we sought, and that are achievable with brand advertising. Touri ketin Tourism is an important industry in New York State and is critical to the economy, not only in New York City, but to many upstate areas as well. However, prior to the current administration, the State had not consistently marketed tourism. As a result, tourists and the tourism industry had little understanding of the art, culture, history and nature that New York State has to offer beyond New York City. When Governor Cuomo came to office, he set about fixing that deficit. The objective of our tourism marketing efforts is simple: to create awareness and enthusiasm for the array of New York State destinations, encourage website visits and, ultimately, consideration of New York State as a vacation spot or weekend getaway. Our campaign has been directed to the leisure travel market including family vacationers, as well as valuable audience segments with speci?c interests [Path through History, Taste NY, LGBT Travelers, etc]. Advertising on television is the primary vehicle for our tourism marketing, where we continue the iconic I Love NY campaign, supported by public relations initiatives and social media to ensure high levels of earned media. 8 Russell Research, Advertising Evaluation, October, 2013; Russell Research Tracking Study, October, 2014 8 YORK Empire State . Development Division of State Government Accountability 23 2014-5-10 Empire State Development 633 Third Avenue New York City, NY 10017 (212) 803-3130 Evidence of E?ectiveness We measure the effectiveness of our tourism marketing just as we do for our business marketing: we examine website traf?c, changes in perceptions about New York State and changes in consideration of NY State as a destination for travel. The advertising itself does not in and of itself generate visitors. tourism marketing has been effective, as evidenced by the following: 1. Periods with a paid media campaign saw an average weekly website traffic increase of 48% compared to periods in the same year when the State did not run advertising. In fact, organic average weekly traffic (defined as website traffic during non?promoted periods] has decreased in each of the last full two years (2012-2013), underscoring the importance of paid media in keeping New York State top of mind for visitors.9 2. A slight majority of New York State residents and nearly one-half from target non-NY markets recalled seeing or hearing an advertisement for the state in Comment the three months prior to the study, far higher than any other competing 9 destination. 10 3. Exposure of the summer campaign to target travelers confirms the advertising is both appealing and motivating. Four-fifths of the target audience indicate they like the I Love NY campaign, including more than four-?fths of state residents and three-quarters from out-of?state A study conducted by the Advertising Research Foundation has suggested that likeability is the single largest predictor of a campaign?s success. 11 4. More than three in ?ve New York State residents and one-half from out-of- state indicate they would be more likely to visit New York State in the summer based on the campaign. 12 9 Google Analytics, data January 1, 2012-October 25, 2014. 9 Russell Research, I LOVE NY Tracking Research Study June, 2014 10 Russell Research, I LOVE NY Tracking Research Study June, 2014. Additionally, it is worth noting that while ESD has sited increases in visitors to Speci?c destinations that have been featured in the marketing campaign, we are not suggesting that the increase number of visitors was a rank of the campaign alone but Com ment rather an indication that more individuals put New York destinations on their consideration lists. On the 10 other hand, 0503 citation of well-known inductees to the Baseball Hall of Fame, or better snow conditions contributing to more skier visits, are plausible explanations for visitor increases, but are not apposite to advertising of those destinatiOns. 12 Russell Research, i LOVE NY Tracking Research Study June, 2014. 9 Empire State wmumw Development Division of State Government Accountability 24 2014-5-10 Empire State Development 633 Third Avenue New York City, NY 10017 (212) 803-3130 Overall, we are assured by the data on website visits, perceptions, and consideration that the State?s promotion of tourism is having its intended effectDATIONS 1. QSC Recommendation: ESD focuses on the Specific: services that are provided, such as advertisements created and piaced outputs), rather than on the results that are achieved, such as increased employment outcomes). This situation exists primarily because ESD did not quanti?/ what it expected to achieve from its advertising ejjforts, except in the broadest terms like in creasing tourism or creating jobs. ESD RESPONSE: Contrary to this assertion, ESD does focus on the results to be achieved from the marketing campaign. As stated earlier, increasing website traf?c, changing perceptions, increasing consideration and helping generate leads for economic development programs are the key focus of our marketing. Contrary to assumptions, job creation is not the definition of success, since advertising by itself cannot close a deal to create jobs, nor change general economic conditions that impact corporate job creation, such as interest rates, in?ation, availability of a skilled workforce and the like. What marketing efforts accomplish is the creation of awareness of, and interest in, New York State as a place to grow business. Those accomplishments are in fact "outcomes,? every bit as real as job growth numbers. We would also point out that, as part of its review and approval of the most recent amendment to the BBDO contract in November, 2014, OSC Specifically requested that we provide performance monitoring data for the marketing contract. We duly provided that information, which included most of the same market research, website hit ?gures and tourism increases that we provided to 0503 audit team in connection with this audit. As you know, OSC in fact did exercise its discretion to approve the amendment to the contract, applying the same criteria for approval discussed above. We believe that OSC was correct in approving and extending the contract, and recognized that our approach to performance monitoring was appropriate, and in the best interests of ESD. Nevertheless, and based on our experience with the BBDO contract, we believe that it is appropriate to incorporate a requirement for our marketing agent to report on performance measures and metrics in the new marketing contract to be entered into at the end of 2015, following the expiration of the current BBDO contract. 10 3530?? Empire State Development Comment 4 Division of State Government Accountability 25 2014-5-10 Empire State Development 633 Third Avenue New York City, NY 10017 (212) 803-3130 2. OSC RECOMMENDATION: ESD is unable to evaluate the extent to which its $211.5 million planned investment has contributed toward achieving the purposes of the underlying programs or whether it is cost-effective. ESD o?icials indicated they do not believe that advertising programs should be measured in any way against the results achieved by the underlying programs. ESD RESPONSE: ESD's marketing is not measured by the success of the marketed programs. In fact, Start-Up NY has been an extremely successful program, generating commitments to invest in New York, as well as commitments to create 2,085 new jobs.13 As discussed earlier, it is not possible to assert that this success is solely the result of marketing, and therefore, ESD is correct in its position that the success of the marketing efforts should not be measured in this way. 3. OSC RECOMMENDATION: As of September 2014, ESD had identi?ed?ve different economic development programs to be covered under the BBDO contract: Masterbrand, Taste NY, START-UP Y, Hurricane Sandy, and Tourism. There is no formal allocation of the total contract amount among these five programs. ESD RESPONSE: The advance allocation of the total contract amount among programs was not intended by ESD to be a goal of the BBDO contract. BBDO was contracted to act as full-service advertising and marketing agency. It Comment would be impossible to allocate marketing funds to SuperStorm Sandy relief, or 11 to a new program such as Start-Up NY when the BBDO contract was initiated in November of 2011. Rather, ESD directs marketing resources to economic development programs economic development programs on a case-by-case basis, as needs arise. 4. OSC RECOMMENDATION: in keeping with this philosophy, ESD has not developed an overall strategic plan for the marketing services covered by the BBDO contract, nor has it developed an overall budget for each individual program. E512 RESPONSE: ESD did develop an overall strategic plan for the BBDO marketing services. As discussed above, plan was to improve perceptions of New York as a place to visit and to do business. ESD did not intend to create overall budgets for each program, because the funds to be spent on each program were decided on a case-by-case basis. This is not a failure of the marketing effort. 0n the contrary, it is part of the marketing strategy?s success, as OSC has implicitly acknowledged by five times approving the BBDO contract and its funding as being in best interests. 13 Start-Up NY 2014 Annual Report, April 1, 2015. 1 1 ?15130? Empire State Tmm" lDevel0pment Division of State Government Accountability 26 2014-5-10 Empire State Development 633 Third Avenue New York City, NY 10017 (212) 803-3130 5. 93!; REE According to ESD o?icials, the advertising done under the BBDO contract is not intended to directly produce economic results, such as increasing tourism or creating jobs. Instead, advertising is intended to support these programs by improving public perceptions of New York as a place to visit and for business development. ESD expects that improved public perception will indirectly lead to more tourism and job creation as more individuais and businesses consider New York an attractive place to visit and do business. ESD RESPONSE: 0505 statement as to expectation is correct, and ESD stands behind its conviction that the marketing services provided by BBDO have supported tourism and job creation, and will continue to do so. CONCLUSION ESD appreciates evaluation of the BBDO contract and marketing program, as it has allowed us to take a critical look at contract management processes and to work to make them even better. Equally, the audit focused our attention on improving methods to assess the success of the marketing program in meeting its goals. While ESD and OSC disagree on what should be the primary focus of those assessment and monitoring measures, your audit recommendations will certainly be taken into account as we proceed with our marketing program. ESD looks forward to cooperating fully with OSC audits in the future, as we strive to enter into and monitor the performance of contracts that help stimulate economic development and provide the best value for the peeple of the State. Ends. 12 5 EW YD RK sure be Empire State Development Division of State Government Accountability 27 2014-S-10 Research Implications EXECUTIVE SUMMARY Research conducted by Russell Research in 2013-14 indicates that Empire State Development’s (ESD) marketing efforts have positively impacted perceptions and consideration of New York State both as a place to do business and visit for a vacation or getaway. Business Development        Marketing has been very successful at driving traffic to the website with a 530% increase compared to periods in the same year when New York State did not run advertising. ESD’s marketing efforts have successfully reached business executives with one-half of New York State-based executives (52%) and one-quarter from out-of-state (26%) recalled having seen one or more commercials from the StartUp NY campaign. This has resulted in strong awareness of the StartUp NY program. Three in five New York State executives (60%) are aware of the program, and while only three in ten outside of the state (30%) are aware of StartUp NY, this includes two-thirds of executives from large businesses (65%). There has been a strong positive shift in perceived momentum of New York State’s business climate. One-half of professionals (50%) believe the state is moving in the right direction, a 72% increase in just 12 months after the launch of the campaign. A majority of executives (55%) believe that New York State is an excellent or very good place to do business – a 62% increase since October 2013. This increase was more pronounced out-of-state, with a 122% increase in the same time frame (23%  51%). More than three-fifths of NY State executives (63%) would consider New York as a place to do business, a finding that has not statistically changed over time. However, a slight majority of out-of-state executives (51%) indicate they would consider opening or relocating a business to New York State, a 132% increase since October 2013 (22%), which is statistically significant. For most metrics, results are stronger among professionals who recall seeing StartUp NY television advertising. Tourism      Periods with a media campaign spend saw an average weekly website traffic increase of 48% compared to periods in the same year when the State did not run advertising. New York State is the most top-of-mind destination for summer vacations and/or getaways as one-half of New York state residents (51%) and two-fifths who live outside the state (41%) named New York State as a vacation destination on an unprompted basis. Nearly four in five New York State residents (78%) would consider a New York State summer vacation in the future and nearly three-fifths (56%) of out-of-state residents would consider the state. This is higher than all neighboring states/regions among in-state residents and only trails New England among those who live outof-state. A slight majority of New York State residents (52%) and nearly one-half from target non-NY markets (46%) recalled seeing or hearing an advertisement for the state in the three months prior to the study, by far higher than any other competing destination. More than two-fifths of New York State residents (44%) and more than one-third from out-of-state (36%) recalled seeing one or more commercials from the 2014 summer tourism campaign. Across multiple 1 Division of State Government Accountability 28 2014-S-10 Research Implications campaigns with a range of support levels from a budgetary and timing perspective, this is an above average level of campaign recall. BUSINESS DEVELOPMENT MARKETING The stated objective of Empire State Development’s marketing efforts has been to create a compelling message that would first drive traffic to the ESD website and ultimately begin the long-term process of changing entrenched perceptions that NY State is anti-business. Economic development advertising was directed to business leaders inside the State (to drive consideration for retention and business expansion) and outside the State (to drive consideration for relocation). This advertising was very successful at driving traffic to the website with a 530% increase compared to periods in the same year when New York State did not run advertising (source: Google Analytics). 14000 12000 10000 8000 Campaign 6000 No Campaign 4000 2000 0 2011 2012 2013 2014 Source: Google Analytics, data January 1, 2012-November 29, 2014 (excluding August 31st-November 8th due to Google tagging issue) Russell Research has conducted a series of research studies for Empire State Development which has assessed perceptions of New York State among executives within the business community both inside and outside New York State. • Advertising Evaluation: October 2013 • Advertising Evaluation: March 2014 • Tracking Study: October 2014 A range of metrics captured over the course of the several research studies suggests marketing efforts have been effective in improving perceptions of the New York State business climate. This is particularly evident when examining perceptions and consideration of the state among large out-of-state businesses (a primary target for 2 Division of State Government Accountability 29 2014-S-10 Research Implications relocation) and when results of most key metrics are compared between those who recall seeing one or more advertisements versus those who had not – with the former having far more positive perceptions than the latter. 3 Division of State Government Accountability 30 2014-S-10 Research Implications Campaign Reach ESD’s marketing efforts have been successful in reaching the target audience of business executives, particularly among the valuable large company segment (250 or more employees). One-half of New York State-based executives (52%) and one-quarter from out-of-state (26%) recalled having seen one or more commercials from the StartUp NY campaign. This includes more than three-fifths of executives in large businesses located in New York State (63%) and one-half headquartered outside the State (52%). Aware of advertisement 100 90 80 70 60 50 40 30 20 10 - 57 52 39 33 26 63 55 52 39 29 12 Total New York NonSmall New York Medium Large Small Medium Large Small 19 Medium Large This broad advertising reach has resulted in strong awareness of the StartUp NY program. Three in five New York State executives (60%) are aware of the program, and while only three in ten outside of the state (30%) are aware of StartUp NY, this includes two-thirds of executives from large businesses (65%). 100 90 80 70 60 50 40 30 20 10 - Aware of "StartUp New York" 62 60 45 30 39 63 58 65 59 38 14 Total New York NonSmall New York Medium Large Small Medium Large Small 19 Medium Large 4 Division of State Government Accountability 31 2014-S-10 Research Implications Brand Momentum Momentum is a key indicator of a brand’s health (with the New York State business climate being the brand). When the first research was conducted for ESD, less than three in ten business professionals (29%) believed New York State’s business climate’s momentum was moving in a positive direction, with the other seven in ten believing the state was either holding its ground (43%) or moving in a negative direction (27%). During this same time period, Texas was statistically significantly more likely to be seen as having positive brand momentum (48%). 100 90 80 70 60 50 40 30 20 10 - 11 27 Worse 41 Stayed the same 43 Better 48 29 New York Texas This dramatically changed by the Fall of 2014, with one-half of professionals (50%) now believing the state is moving in the right direction, a 72% increase in just 12 months after the launch of the campaign. This includes more than double the percentage of out of state professionals (23%  48%) and nearly tripling among executives in large businesses located out-of-state (25%  69%). Momentum was also much higher among outof-state professionals who were aware of the StartUp advertising (78%). 100 90 80 70 60 50 40 30 20 10 0 Fall 2013 55 51 49 50 43 36 29 Winter 2014 Total New York 38 27 Non-New York 65 69 57 48 23 Fall 2014 Small 30 37 41 25 Medium Large 5 Division of State Government Accountability 32 2014-S-10 Research Implications Professionals aware of the StartUp advertising campaign were more likely to believe the State is moving in a positive direction. 100 90 80 70 60 50 40 30 20 10 - Campaign Aware Campaign Unaware 78 63 56 48 41 Total 37 New York Non-New York 6 Division of State Government Accountability 33 2014-S-10 Research Implications A Place To Do Business This increase in momentum has led to a majority of executives (55%) to believe that New York State is an excellent or very good place to do business – a 62% increase since October 2013. This increase was more pronounced out-of-state, with a 122% increase in the same time frame (23%  51%), and includes threequarters of those out-of-state who recalled the 2014 campaign (74%) rating the state as excellent or very good. 100 90 80 70 60 50 40 30 20 10 - Fall 2013 Winter 2014 59 55 44 34 45 44 51 43 23 Total 100 90 80 70 60 50 40 30 20 10 - Fall 2014 New York Campaign Aware Non-New York Campaign Unaware 74 67 63 56 48 Total 44 New York Non-New York 7 Division of State Government Accountability 34 2014-S-10 Research Implications Campaign Impact on State Imagery The Fall 2014 research exposed executives to a range of perceptual statements that could be used to describe New York State from a business perspective. Executives who had seen the StartUp NY advertising campaign rated 10 of 13 attributes statistically significantly higher than those who had not seen the advertising – further pointing to the efficacy of ESD marketing (boxes below indicate statistically significant difference from unaware professionals). Is open to growth 78 66 Provides a good infrastructure for business 77 56 Is dedicated to bringing jobs to their state 77 60 Provides a quality workforce for business 75 64 Is open to innovation 75 58 Provides good opportunities for business 73 55 Invests in higher education to provide a workforce with the skills… 71 52 Is friendly to business 71 57 Has tax incentives which are friendly to business 70 41 Is good for my business 70 50 Has a government that understands what business needs 52 Is taking bold steps for business 51 Is an easy place to start a business 52 - 10 20 30 40 50 60 Campaign Aware Campaign Unaware 67 66 62 70 80 90 100 8 Division of State Government Accountability 35 2014-S-10 Research Implications Consideration The combination of increased momentum and a perceived better business climate has resulted in out-of-state executives being far more likely to consider New York State as a place of business. Specifically, a slight majority of out-of-state executives (51%) indicate they would consider opening or relocating a business to New York State, a 132% increase since October 2013 (22%), which is statistically significant. Further, three-quarters of executives who recall having seen the StartUp NY television campaign (74%) would consider the state, compared to less than one-half (44%) who haven’t seen state advertising. Consideration of New York State as a place to do business has always been strong among New York State-based professionals with no statistically significant differences over time. Meanwhile, consideration among non-New York professionals has experienced significantly positive growth over time. 100 90 80 70 60 50 40 30 20 10 - Fall 2013 49 45 57 Winter 2014 69 63 56 51 41 22 Total 100 90 80 70 60 50 40 30 20 10 - Fall 2014 New York Campaign Aware 68 Campaign Unaware 74 65 60 50 Total Non-New York 44 New York Non-New York 9 Division of State Government Accountability 36 2014-S-10 Research Implications TOURISM MARKETING Marketing efforts in the area of tourism have a stated objective of creating awareness and enthusiasm for the array of New York State destinations that will ultimately translate to consideration of the state as a vacation of spot or weekend getaway. The “I Love NY” campaign has been directed toward the leisure travel market including family vacationers and other valuable audience segments. Recent campaigns have targeted state residents as well as potential vacationers in surrounding markets. Periods with a media campaign spend saw an average weekly website traffic increase of 48% compared to periods in the same year when the State did not run advertising. In fact, organic average weekly traffic (defined as traffic to the website during non-promoted periods) has decreased in each of the last full two years (20122013), underscoring the importance of paid media in keeping New York State top of mind for visitors (source: Google Analytics). 40,000 35,000 30,000 25,000 20,000 Campaign 15,000 No Campaign 10,000 5,000 0 2012 2013 2014 Source: Google Analytics, data January 1, 2012-October 25, 2014 A tracking study was conducted in June 2014 to establish baseline metrics for New York State and assess the impact of advertising on perceptions and consideration of the state as a tourist destination. A second wave to further evaluate trends will be conducted in March 2015. 10 Division of State Government Accountability 37 2014-S-10 Research Implications Unaided Destination Recall The research indicates the advertising campaign has reached the target audience and is motivating, resulting in New York State being widely top-of-mind and considered among target travelers. Recent learnings in the field of behavioral economics conclude that consumers often take “mental shortcuts” when making a decision. Therefore, top-of-mind associations are particularly important in travel planning as consumers will often not take the time to do the research needed to find new travel destinations, instead investigating / considering the destinations that come to mind. From a market research perspective, this results in unaided awareness being a very important metric. New York State is the most top-of-mind destination for summer vacations and/or getaways as one-half of New York state residents (51%) and two-fifths who live outside the state (41%) named New York State as a vacation destination on an unprompted basis. Top-of-mind recall was significantly higher among all audiences who recalled the summer 2014 tourism campaign. NYS (Other than NYC) New York City MA (Outside of Boston) Boston Connecticut Maine New Hampshire Rhode Island Vermont New Jersey Pennsylvania Florida Virginia South Carolina Caribbean Canada 88 7 5 47 56 58 Campaign Aware 12 21 16 14 21 32 17 18 32 33 911 13 12 20 19 20 35 100 90 80 70 60 50 40 30 20 10 - 41 51 8 12 20 40 NY State Non-NYS 60 80 100 Campaign Unaware 62 57 42 40 Total New York 49 37 Non-New York 11 Division of State Government Accountability 38 2014-S-10 Research Implications Vacation Destination Consideration The strength of top-of-mind recall is reinforced when examining future consideration of New York State as a summer vacation destination. Nearly four in five New York State residents (78%) would consider a New York State summer vacation in the future and nearly three-fifths (56%) of out-of-state residents would consider the state. Consideration of New York is higher than all neighboring states/regions among in-state residents and only trails New England among those who live out-of-state. Further, consideration of New York State for a summer vacation is near equal among New York City residents and those who live in other parts of the state. 74 New England 68 78 NYS (Other than NYC) 58 50 50 NY State Non-NYS 51 - 20 40 60 79 59 57 49 New York City 54 New Jersey 77 Pennsylvania 55 New York City 75 NYS (Other than NYC) 56 Pennsylvania 73 New England 80 51 58 New Jersey 100 Rest of NY 38 - 20 40 NYC 60 80 100 12 Division of State Government Accountability 39 2014-S-10 Research Implications Advertising Awareness The television advertising has been very successful in reaching the target audience. A slight majority of New York State residents (52%) and nearly one-half from target non-NY markets (46%) recalled seeing or hearing an advertisement for the state in the three months prior to the study, by far higher than any other competing destination. NYS (Other than NYC) 32 19 New York City 46 27 11 13 MA (Outside of Boston) 5 Boston Connecticut 11 8 12 15 17 Maine 9 8 New Hampshire Rhode Island 4 6 Vermont 8 12 31 29 New Jersey NY State 15 18 Pennsylvania - 20 Non-NYS 40 60 80 100 13 Division of State Government Accountability 40 2014-S-10 Research Implications Several destinations featured in Tourism advertising experienced increases in visitorship between 2013 and 2014 (source: New York State Tourism). 2014 Visitorship 2013 Visitorship % Increase Baseball Hall of Fame 211,687 180,621 17% DIA: Beacon 75,002 400,000 est. 14% Corning Museum of Glass 85,425 416,000 est. Olana State Historic site 84,679 75,794 12% Watkins Glen 256,949 212,720 21% Whiteface Mountain 218,348 192,427 13% Belleayre Mountain 131,257 118,509 11% Gore Mountain 202,718 198,211 2% Jones Beach State Park 2,303,067 2,019,613 Letchworth State Park 226,673 221,331 2% Minnewaska State Park 96,737 93,993 3% 4% 14% 14 Division of State Government Accountability 41 2014-S-10 Research Implications Target consumers who recalled recent advertising specifically remembered seeing a number of these destinations that have experienced increases in traffic between 2013 and 2014. When exposed to commercials from the 2014 summer tourism campaign, more than two-fifths of New York State residents (44%) and more than one-third from out-of-state (36%) recalled seeing one or more commercials. Across multiple campaigns with a range of support levels from a budgetary and timing perspective, this is an above average level of campaign recall. 100 80 60 44 40 36 45 43 NYC Rest of NY 20 NY State Non-NYS 15 Division of State Government Accountability 42 2014-S-10 Research Implications Advertising Quality Exposure of the summer campaign to target travelers confirms the advertising is both appealing and motivating. Four-fifths of the target audience (81%) indicate they like the I Love NY campaign, including more than fourfifths of state residents (85%) and three-quarters from out-of-state (75%). A study conducted by the Advertising Research Foundation has suggested that likeability is the single largest predictor of a campaign’s success. 100 90 80 70 60 50 40 30 20 10 - 1 1 1 11 16 22 45 Total 49 85 NY State 28 43 36 81 Disliked it very much Disliked it somewhat 36 36 0 12 2 11 75 39 43 Non-NYS NYC 86 57 Neither liked nor disliked it Liked it somewhat 85 Liked it very much Rest of NY Advertising Motivation More than three in five New York State residents (63%) and one-half from out-of-state (50%) indicate they would be more likely to visit New York State in the summer based on the campaign. 100 1 1 41 36 1 - 1 90 80 70 31 40 49 Much less likely to consider Somewhat less likely to consider Neither more nor less likely to consider 60 50 40 35 57 34 10 63 50 58 67 36 30 20 35 32 22 29 Total NY State - 26 14 Non-NYS NYC 32 Somewhat more likely to consider Much more likely to consider visiting NYS in summer Rest of NY 16 Division of State Government Accountability 43 2014-S-10 State Comptroller’s Comments 1. We do not suggest that ESD has no performance measurement criteria and in fact acknowledge ESD’s efforts to track certain measures that it purports to be indicators of success throughout our report. Rather, we express concern that these measures make no attempt to quantify desired outcomes and therefore preclude accountability for the costeffectiveness of ESD’s significant investment. 2. We question the extent to which the statistics cited by ESD are indicative of broader program results. We cannot confirm the basis for, nor accuracy of, the data cited by ESD, which appears to come largely from the partial research report that was appended to ESD’s response. This undated document was not previously provided to our auditors, despite multiple requests for all relevant information. However, our examination of other studies prepared by ESD’s vendor, Russell Research, indicates that it conducted its START-UP NY surveys using very limited and specifically targeted groups of participants; its October 2013 survey had 401 participants, while its March 2014 survey had only 227. Further, its November 2013 Tourism study specifically stated that the results were “qualitative in nature and cannot be considered representative or predictive of the larger target audience.” Information about the June 2014 Tourism study referenced in ESD’s response is presented only in executive summary form, and does not include any details about the number of individuals surveyed nor any limitations on the results, as was otherwise provided in the November 2013 survey. 3. ESD officials have stated that the success of the Open for Business marketing efforts should not be measured by job creation or increased tourism. Yet they express confidence that those efforts have been successful due to job creation and increased tourism. This further underscores the importance of developing relevant outcome measures that could be used to evaluate whether the marketing efforts are achieving the goals desired by ESD officials. 4. The five items which ESD refers to as “OSC RECOMMENDATIONS” are not the recommendations from our report. Our audit’s three recommendations are presented on pages 12 and 13 of the report. 5. As discussed in the Background section of our report (see page 6), OSC’s pre-approval of the contract ensures that it was fairly procured and that the cost is reasonable on its face. OSC’s approval does not signify that the contract is the most appropriate method for ESD to employ to accomplish its stated purpose, nor does it ensure ESD will achieve its program goals. ESD management is responsible for ensuring that the contract is necessary and for accomplishing its intended purpose. 6. We do not dispute that advertising can positively impact interest and perceptions. However, the fact remains that ESD had no objective/quantitative benchmarks regarding the adequacy of any increases (improvements) in interest and perceptions. Consequently, the success of the advertising campaign is unclear, and ESD cannot state with sufficient certainty whether or not the campaign has been worth the considerable investment of public funds. 7. Similar to the June 2014 Tourism study discussed in Comment 2 (which is later referred to as the Russell Research I LOVE NY Tracking Study June 2014 in footnote 10 of ESD’s response), the referenced October 2014 Russell Research Tracking Study was also not Division of State Government Accountability 44 2014-S-10 provided to us despite multiple requests for all relevant information. 8. According to the START-UP NY 2014 Annual Report issued by ESD on April 1, 2015, to date the program has created 76 new jobs and generated $1.8 million in capital investments. In comparison, through October 2014, ESD had spent $45.1 million advertising this program. 9. While our audit was underway, ESD only provided our auditors with one tourism survey conducted by its vendor, which was done in November 2013. That report stated that “Overall, the commercials did not particularly appeal to most target consumers.” As a result, while the participants in the surveys may have remembered the advertisements, the report concluded they were no more likely to visit New York State after having seen them. This is reflected in our report (see pages 11-12). 10. During the course of our audit, ESD officials did, in fact, assert that the success of their marketing efforts contributed to the increases in attendance at these locations. Further, on page 14 of the research report summary (appended to their response), officials continue to call attention to the increased attendance at several locations in the State as evidence of the success of their advertising. 11. Nowhere in our report do we recommend that ESD formally make fixed allocations of advertising funding to each specific program. Rather, we simply state that it is not ESD’s practice to do so. Division of State Government Accountability 45