July 26, 2019 VIA HAND DELIVERY and E-MAIL J. Mitchell Armbruster, Esq. Smith Anderson, L.L.P. 150 Fayetteville Street, Suite 2300 Raleigh, NC 27601 marmbruster@smithlaw.com RE: Final Decision in Response to Amplify’s Protest Dear Mr. Armbruster: Istation is the best diagnostic tool for the state of North Carolina. Istation received the contract award through a fair and objective process conducted by an evaluation committee that adhered to all laws, rules, and policies. The evaluation committee recommended Istation to me and to the State Board of Education (the “State Board”). I approved that recommendation, and the State Board unanimously approved that recommendation. Pursuant to 09 NCAC 06B .1102(c)(2), this letter and its attachments constitute my decision on behalf of the North Carolina Department of Public Instruction (the “Department”) to the protest by your client, Amplify Education, Inc. (“Amplify”), of the Department’s June 7, 2019 Read to Achieve contract award to Imagination Station, Inc. (for this letter, “Istation” refers to the company or Imagination Station Inc.’s reading diagnostic tool, Istation, as the context requires). I have reviewed the claims in your protest letter, the claims expressed during the in-person protest meeting, and the claims in your supplemental letter. After careful review, I stand by the Department’s original contract award and deny Amplify’s protest. This letter will give you and the public the clarity necessary to understand the entirety of this procurement process. Amplify’s claims about Imagination Station, Inc.’s reading diagnostic tool, Istation, and its claims about the procurement process, are incorrect and many are based on misstatements of fact. Please consider these four pages a summary. Details regarding each section can be found in the provided attachments. I. Read to Achieve Diagnostic Tool Procurement North Carolina law requires one reading diagnostic tool for every kindergarten, first, second, and third grade class. This same law directed the Department to conduct a Request for Proposal (“RFP”), consider certain factors, and select one tool to be implemented beginning with the 20192020 school year. I was responsible for assembling an evaluation committee of Department employees who would recommend a tool for my approval. (Please see Attachment A). OFFICE OF THE NORTH CAROLINA SUPERINTENDENT Mark Johnson, Superintendent of Public Instruction mark.johnson@dpi.nc.gov 6301 Mail Service Center, Raleigh, North Carolina 27699-6301 (919) 807-3430 Fax (919) 807-3445 AN EQUAL OPPORTUNITY/AFFIRMATIVE ACTION EMPLOYER The State Board, the Department, and I awarded this contract to Istation because it is the best diagnostic tool for North Carolina. • • • • • • Istation provides immediate results for parents to view, track, and better help their students’ literacy progress; even allowing families to log in at home. Istation provides immediate data and results for teachers and administrators to view, analyze, and personalize their teaching to better meet the needs of North Carolina students. Istation minimizes the amount of instructional time impacted by assessments. Put another way, teachers can spend more time teaching, rather than administering tests. Istation is a highly integrated tool that provides assessment results with instructional support for teachers and students. Istation provides the Department and schools across North Carolina with the best value. Istation provides more options for less money than does your client. This list goes on. Unfortunately, this positive story for North Carolina teachers and students has been overlooked due to misstatements of fact from your client, unintentional or not, and inappropriate actions by employees no longer with the Department. At the start of the process in 2017, evaluation committee members promised to prioritize fairness and objectivity over improper biases; going so far as to sign non-disclosure agreements and statements affirming that they had disclosed all possible conflicts of interest. Unfortunately, a small number of career Department employees did not honor these commitments. • • • • • • One did not disclose that she had previously been paid by Amplify, her most recent employer before the Department, even though she signed the no conflict of interest document. One violated the non-disclosure agreement to leak updates to outsiders about confidential issues. A whistleblower provided evidence of a text message discussion detailing how committee members had voted and characterizing my priorities and efforts for educators as attempts to “appease lazy ass teachers.” (Their words, not mine.) Members of the selection committee employed biased procedures that benefited the incumbent vendor (Amplify) over other vendors by not clarifying questions on specifications. If the questions had been clarified, as required by the procurement rules and procedures, it would have benefitted other vendors. Members of the selection committee employed biased procedures that benefited the incumbent vendor (Amplify) over other vendors by allowing untrue statements or nonfactual opinions to be presented as fact during their ranking process. If factually correct statements had been presented, it would have benefitted other vendors. Members of the selection committee violated procurement policies outlined in the Department of Information Technology procurement manual by improperly adding weight to select criteria when ranking the vendors. 2 These employees are no longer with the Department. And, to be clear, the majority of career Department employees are honorable civil servants who place the needs of educators and students as their top priority. Please do not let the bad actions of a small minority portray all Department employees in a negative light. Regrettably, so as to conduct a fair and unbiased procurement process, the Department had to cancel two RFPs. (Please see Attachment B). Pursuant to state law and procurement rules and polices, and in consultation with the Department of Information Technology and the State Board, the Department entered negotiations with Amplify and Istation. (Please see Attachment C). The evaluation committee for the negotiations process presented to me its unanimous recommendation to select Istation, and I approved the recommendation to select Istation. Department staff then presented the recommendation to the State Board, and the State Board unanimously approved the recommendation to select Istation. For clarity, there was only one recommendation in the entire procurement process, and it was Istation. (Please see Attachment B). II. Amplify’s Incorrect Claims about Istation Throughout this protest for Amplify, you have made incorrect claims about Istation to argue in favor of Amplify’s request. Istation has already been used successfully in other states and more than satisfies the requirements to be the reading diagnostic tool for North Carolina. (Please see Attachment D). Contrary to your incorrect claims, and without limitation: • • • • • • • • • • • • III. Istation satisfied the requirements of the procurement. Istation meets the requirements of law. Istation predicts students who are at risk of a reading difficulty. Istation screens for dyslexia. Istation is developmentally appropriate. Istation assesses required measures. Istation is effective at measuring a student’s progress. Istation allows for students to interact with and read aloud to teachers. Istation provides sufficient data to allow determinations of short-term student progress. Istation accurately and reliably determines which students will be reading on grade level by the end of the school year. Istation is already on track for successful implementation in North Carolina. Istation provides the Department and schools across North Carolina with the best value. Issues with Amplify and mCLASS in North Carolina In your protest letter, you state that the Department “must also consider factors including whether the vendor complies with industry standards, the vendor’s past performance with the state, and the probability of the vendor providing the required service on time.” Amplify has 3 presented problems meeting the needs and requirements for the Department and North Carolina school systems. (Please see Attachment E). IV. Final Decision As described and detailed in the attachments, the claims in your protest letter that the award to Istation does not meet the requirements of the procurement process or North Carolina law are incorrect. The Department and the State Board followed all laws, policies, and rules related to this procurement process and contract award. Because of the reasons I have incorporated herein, and without waiving the procedural defects of Amplify’s untimely protest letter, I stand by the Department’s original contract award and deny your client’s protest. Istation is the best reading diagnostic tool for North Carolina educators, students, and parents. Sincerely, Mark R. Johnson North Carolina Superintendent of Public Instruction 4 ATTACHMENTS Attachment A Overview Attachment B Read to Achieve Diagnostic Tool RFPs Attachment C Negotiations Process Attachment D Istation Attachment E Amplify EXHIBITS Exhibit A Evaluation Committee Member’s Confidentiality Agreement Exhibit B Evaluation Committee Member’s Statement Regarding Conflict of Interest Exhibit C January 8, 2019 Text Message Chain – Breach of Confidentiality Exhibit D November 19, 2018 Evaluation Consensus Meeting Notes (pp. 1-2) Exhibit E Read to Achieve 2018 Weighted Criteria Chart Exhibit F June 7, 2019 Contract Award Recommendation Exhibit G Positive Responses from North Carolina Educators After Istation Training Exhibit H North Carolina Reading Scores Exhibit I 40-IT00107-16 / BAFO 3 (pp. 20-21) 5 Attachment A Overview State law requires North Carolina to have one reading diagnostic tool for every K-3 class and required the Department to evaluate diagnostic tools and select one to be implemented beginning with the 2019-2020 school year. “The State Superintendent shall form and supervise an Evaluation Panel to review the proposals received pursuant to the RFP issued in accordance with subsection (b) of this section.” Section 7.27(c) of S.L. 2017-57, as amended by Section 2.6 of S.L. 2017-197. This law also required that “[t]he Evaluation Panel shall be composed of persons employed within the Department of Public Instruction.” Id. “In determining which vendor to select, the Evaluation Panel shall consider, at a minimum, all of the following factors: (1) the time required to conduct formative and diagnostic assessments with the intention of minimizing the impact on instructional time; (2) the level of integration of assessment results with instructional support for teachers and students; (3) the timeliness in reporting assessment results to teachers and administrators; and (4) the ability to provide timely assessment results to parents and guardians.” Id. As directed by the General Assembly in Section 7.27(c) of S.L. 2017-57, as amended by Section 2.6 of S.L. 2017-197, “the Evaluation Panel, with the approval of the State Superintendent, shall select one vendor to provide the assessment instrument or instruments for the 2019-20 school year.” For clarity, any recommendation from the evaluation committee would also require my approval to become the selected vendor, which underscored my obligation, and enhanced my ability, as State Superintendent to ensure a fair and objective process. The Department, the State Board, and I fulfilled our statutory duties. In doing so, we consulted with the Department of Information Technology (“DIT”), the agency ultimately in charge of approving information technology (“IT”) procurement awards. The Department regularly consults with DIT to ensure the Department adheres to North Carolina’s procurement laws, rules, and policies. 6 Attachment B Read to Achieve Diagnostic Tool RFPs 1. Introduction The procurement process for the Read to Achieve diagnostic tool faced unexpected challenges from within the Department. The process had to be cancelled and restarted twice to ensure fairness and objectivity, while also ensuring the Department and the State Board adhered to all state procurement laws, rules, and policies. At the start of the process in 2017, evaluation committee members promised to prioritize fairness and objectivity over improper biases; going so far as to sign non-disclosure agreements and statements affirming that they had disclosed all possible conflicts of interest. Unfortunately, a small number of career Department employees did not honor these commitments. Each evaluation committee member signed a Confidentiality Agreement. (See Exhibit A). Each member also signed a statement indicating that he or she had no conflict of interest. (See Exhibit B). The first RFP (“RFP-1”) was issued on December 6, 2017 and publicly opened on January 11, 2018. RFP-1 was cancelled on March 14, 2018 due to a lack of competition and errors in specifications. Even if RFP-1 had not been cancelled for this reason, the Department would have had to cancel RFP-1 due to an undisclosed conflict of interest. During the evaluation phase of RFP-1, the Department was made aware that a voting member of the evaluation committee, who was also one of the two business owners responsible for managing the process, did not disclose that she had previously been employed by your client, Amplify. She was paid by Amplify immediately preceding her employment with the Department. The Department was made aware of this conflict by a whistleblower who is a career school district employee. The second RFP (“RFP-2”) was issued on September 6, 2018 and publicly opened on October 2, 2018. RFP-2 was cancelled on March 21, 2019. RFP-2 had many legal and fairness issues (detailed below) that tainted the process, and the evaluation committee did not reach the consensus required by procurement rules to recommend one of the four diagnostic tools to me. The Department discovered improper and unfair procedures were used during the evaluation and that a voting member of the evaluation committee breached the Confidentiality Agreement by disclosing confidential details about the procurement process. 2. RFP-2 Cancellation An agency can cancel an RFP when there are “circumstances that prevent determination of . . . the best value offer; or [there is] any determination that rejection would be in the best interest of the State.” 09 NCAC 06B .0401. The RFP-2 evaluation process was plagued by multiple issues, resulting in the Department recommending to DIT that the RFP be cancelled. 7 Among these issues: a. b. c. d. e. breach of confidentiality agreement; unfair and improper ranking procedures; misstatements of fact leading to improper rankings; improper weights leading to improper rankings; and lack of consensus and unclear basis for comparison a. Breach of Confidentiality Agreement During RFP-2 An anonymous whistleblower provided the Department with a screenshot of a text message chain between two former Department employees showing that a voting member of the evaluation committee breached the Confidentiality Agreement. Pursuant to 09 NCAC 06B .0103, the Confidentiality Agreement provides that “. . . all information and documentation (verbal and written) relative to development of a contractual document is deemed ‘confidential’ and shall remain confidential until successful completion of the procurement process. Therefore, Evaluation Committee Members are required to keep all comments, discussions, and documentation confidential . . . .” (See Exhibit A). “If it is discovered that there has been a breach of confidentiality by a member of this Committee . . . [t]he solicitation may be cancelled . . . .” (See Exhibit A). During the text exchange, dated January 8, 2019, which is the same day as a meeting of the evaluation committee, one of the texting participants stated: “Well, just got off another marathon call with [employee’s initials redacted]. 1 hour 45 minutes all about RFP what a mess!” (See Exhibit C). The text messages describe who was on the evaluation committee and how each member voted during the meeting. The texting participant editorialized that some evaluation committee members “voted for children” while others “voted for helping teachers” so as to “appease lazy ass teachers.” (See Exhibit C). The Department still has an on-going investigation into this matter and the extent of the breach. b. Unfair and Improper Procedures Corrupted the Evaluation Committee’s Process in RFP-2 The text exchange discussed above added to the concerns the Department and I already had about improper bias jeopardizing the legality and impartiality of RFP-2. The ranking process for RFP-2 was deficient. During the November 2018 meeting, the evaluation committee members were directed to vote “yes,” “no,” or “maybe” on each specification. Members were directed to vote “maybe” if clarification was needed. (See Exhibit D). However, the evaluation committee decided to rank vendors during the draft specification assessment without clarifying any specifications that received “maybe” votes. Instead, “maybe” 8 votes were treated like “no” votes, and only “yes” votes were awarded points for ranking purposes. A proper clarification process would have yielded more “yes” votes for vendors. The Department discovered many specifications that received “maybe” votes (and counted as “no” votes) should have received “yes” votes. A clarification process should have occurred prior to any ranking by the evaluation committee. Vendors should have been receiving credit in this ranking process but were not, which corrupted the outcome. Further, misstatements of fact resulted in improper “no” votes and “maybe” votes (which counted as “no” votes) for Istation. c. Misstatements of Fact A PowerPoint presented to me in December 2018 contained multiple misstatements of fact and omissions of important vendor information. The presentation also had rankings of vendors based on these misstatements and omissions. As documented in the released public records, multiple misstatements of fact were put forth by members of the evaluation committee during the RFP-2 meetings. Many of these misstatements, such as those regarding dyslexia screening, required subsequent clarification and correction. The numerous misstatements of fact and biased information found in the December 2018 PowerPoint presentation were never corrected on that document because the presentation was never delivered publicly. The voting members of the evaluation committee for RFP-2 followed guidance from whomever they considered to be a given topic’s subject matter expert on the evaluation committee. Unfortunately, in multiple cases, the opinion voiced by a subject matter expert was not grounded in fact. Members of the evaluation committee, whether intentionally or not, were putting forth untrue statements about vendor products, but these falsehoods were treated as facts by the rest of the evaluation committee. Consequently, RFP-2 voting and rankings were based on misstatements of fact and incorrect data. The Department will likely never know the full extent of the number of points improperly denied to Istation during the RFP-2 process due to these misstatements of fact. At best, this was an unprofessional and unfair process. At worst, this allowed for improper bias on the part of an evaluation committee member to skew the results. This jeopardized the objectivity and fairness of the entire RFP-2 process. Even if the Department ignored these issues, the evaluation committee for RFP-2 attempted but failed to reach a consensus, as per procurement guidelines, on a recommendation. Then the Department learned that a committee member had broken the Confidentiality Agreement. The related text exchange, unfortunately, featured participants accusing that certain voting members would “vote for children” while others were voting to “appease lazy ass 9 teachers.” This is disappointing evidence that misstatements of fact could have been intentional forms of improper bias. The following points, a non-exhaustive list, are key examples of misstatements of fact that led to improper votes. Misstatements of Fact = Improper Votes = Flawed Results Misstatement (evaluation committee opinion (summarized in December 2018 PowerPoint)): Istation is not a valid diagnostic assessment (based on 10 maybe votes, 1 no vote). Fact: Istation is a valid and reliable diagnostic literacy assessment and is used in all 50 states at the campus, district, or state level. It has been implemented statewide in eight states and is an approved K-3 screener in five additional states. The ISIP ER technical manual provides research underlying the assessment and includes results of research conducted on ISIP ER over multiple years showing strong concurrent and construct validity. ISIP ER reliability results for overall reading range from .927 to .970, which well exceeds required reliability at .80 or validity at .60. Results of predictability studies show ISIP ER can predict with confidence students’ performance on End of Grade exams and is a stronger predictor than DIBELS ORF for Texas Achievement of Knowledge and Skills (TAKS). Misstatement (evaluation committee opinion (summarized)): Istation is not a valid dyslexia screener (based on 2 maybe votes, 9 no votes). Fact: Istation is a valid dyslexia screener and ISIP ER is already used as a screener of risk for dyslexia in two states per legislation requiring dyslexia screening. ISIP ER provides reliable, continuous assessment in each of the critical areas of reading, including development of phonemic awareness and alphabetic knowledge and skills. Therefore, ISIP ER can be a powerful aid in identifying students with dyslexia early in their development. In a telephonic interview of Dr. Joseph Torgesen, one of the top experts in the field of dyslexia, conducted on July 12, 2019, he stated: “I consider this to be as good a screener as any I know of.” Misstatement (evaluation committee opinion (summarized)): Istation does not meet the requirements for progress monitoring (based on 5 maybe votes, 5 no votes, 1 yes vote). Fact: Istation is a valid progress monitoring tool and is currently being used for progress monitoring in all 50 states at the campus, district, or state level. Istation meets the requirements as a strong progress monitoring tool because it: (a) is brief – the assessment may be given to students in a one-on-one setting or with an entire classroom in approximately 30 minutes, (b) is repeatable – numerous items are available for each subtest, making the subtests repeatable throughout the school year, (c) is sensitive to improvement over time – ISIP ER has been designated to automatically provide continuous measurement of student progress throughout the year in all critical areas of reading, (d) documents improvement over time with instantaneous 10 reports to show student progress, and (e) allows for off grade progress monitoring as the assessment will automatically adapt to student ability even if it is below the students current grade. Misstatement (evaluation committee opinion (summarized in December PowerPoint)): Istation parent reports require the teacher to manually fill in the report for each child (based on 6 maybe votes, 2 no votes, 3 yes votes). Fact: Istation automatically generates reports for teachers and parents that are available in English and Spanish, including the Student Summary Report which tracks each student’s progress in critical areas being assessed with complete graphic and contextual analysis. Several user-friendly printable and customizable templates are also available that teachers can choose to use in addition to the automatically provided reports. Also, parents in North Carolina will have access to the online parent portal where they can access student level reports that show specific skills in which each student struggles, and it links directly to the appropriate lessons and resources. Misstatement (evaluation committee opinion (summarized)): Screen-based assessments are not developmentally appropriate for Kindergarteners and struggling learners (based on 4 maybe votes, 7 no votes). Fact: ISIP ER computer-adaptive assessment is developmentally appropriate due to the very nature of the way items are administered that are individually selected to be at the right level of difficulty for each child. The measures are presented in an engaging format, thus reducing many of the problems, such as distraction and lack of interest, that are sometimes encountered when young children, especially struggling learners, are assessed. The computer-adaptive format engages young students as sights and sounds come to life with animated game-like interaction for young readers. The use of technology in the classroom has increased and will continue to increase in the future with all students. In a recent study of kindergarten students, Putnam (2016) found the level of teacher literacy support with the use of Istation had a statistically significant effect on early literacy achievement, explaining 17.7% of the variance in group differences. As the use of technology becomes more prevalent, the relationship between technology and literacy is at the forefront of educational decisions. This study contributes to the growing evidence that “the teacher’s role in the classroom is strengthened by the introduction of new technologies,” and “technology supplements, not supplants the teacher in the classroom.” d. Improper Weights Given to Criteria Pursuant to the North Carolina Administrative Code and the DIT manual, an IT procurement process conducted by the Department must use both the source selection method and the evaluation method designated in the solicitation. RFP-2 directed the use of the “tradeoff” source selection method and the use of the “ranking” evaluation method (see RFP-2 pp. 13-14). 11 Under the tradeoff source selection method, “the criteria are stated in relative order of importance in the bid” but the evaluation must consider the criteria “without weighting them.” Statewide IT Procurement Office Manual 2017, Section 12.7.2. The evaluation committee improperly added weight to certain criteria in the RFP-2 process even though the solicitation document required use of the tradeoff source selection method. (See Exhibit E). Each evaluation criterion should be given the same weight, and each bid receives a relative overall ranking. In the next phase, the ranking evaluation method, the evaluation committee is then allowed to adjust those rankings “up or down when considered with, or tradedoff against, other non-price categories.” 09 NCAC 06B .0302(2)(c). The State emphasizes the importance of equal weight as the first part of the process followed by potential adjustments for non-price categories because a goal of State IT procurements is to drive best value principles when buying services on behalf of the citizens and taxpayers of North Carolina. A best value procurement is “a procurement process with the objective of reducing the total cost of ownership.” 09 NCAC 06A .0102(2). The “particular procurement methods used [in a best value procurement] are selected so as to result in the best value for the State in terms of the function to be performed or delivered.” Id. Procurement laws and policies, when followed, drive the best value by having equal weight for cost at first while allowing an evaluation committee to later adjust for other factors if necessary. If multiple vendors can provide products, which meet the requirements of North Carolina law, an agency must consider cost in proper stewardship of the taxpayer dollars entrusted to it. This process allows agencies to procure products and services that are the best value to the State and its people. The RFP-2 evaluation committee improperly added weights to the scoring that prevented a best value procurement process. Because of this violation combined with the unfair and improper procedures described above, even a worthy product that was offered at no cost could have scored lower than Amplify. e. A Flawed Ranking, Not a Recommendation Even though, as discussed, there was improper bias, flawed procedures, lack of fairness, misstatements of fact, and other issues from RFP-2 that jeopardized the results, the evaluation committee did bring both Amplify and Istation to my attention as the top-ranked vendors. However, to reach an agreement to recommend one tool to me, the evaluation committee had to reach a consensus. The only vote in RFP-2 occurred in January 2019 – it was a vote on which vendor(s) to continue negotiating with – and did not result in consensus, so there was no recommendation. f. Cancellation of RFP-2 With guidance from DIT, RFP-2 was properly cancelled. 12 Attachment C Negotiations Process 1. Introduction The Department, after consulting with the State Board and DIT and properly cancelling RFP-2, entered the negotiations process pursuant to 09 NCAC 06B .0316. An important aspect of the negotiations process is that “[n]egotiations shall not materially alter the intent or scope of the original solicitation document.” 09 NCAC 06B .0316(e). The Department did not materially alter the intent or scope of the original solicitation document. 2. Intent of the Original Solicitation The intent of the original solicitation document was to effectuate the goal of the Read to Achieve legislation. RFP-2 stated that “[t]he purpose of this RFP and any resulting contract award is to solicit offers for Read to Achieve Diagnostic Software . . . to meet NCDPI’s obligations under state laws, which can be found at N.C.G.S. 115C-83.1 et. seq.” (see RFP-2 p. 6). The solicitation was intended to “ensure that difficulty with reading development is identified as early as possible; students receive appropriate instructional and support services to address difficulty with reading development and to remediate reading deficiencies; and each student and his or her parent or guardian be continuously informed of the student’s academic needs and progress.” N.C. Gen. Stat. § 115C-83.2 (2018). The intent of the negotiations process was to effectuate the goal of the Read to Achieve legislation. The intent did not materially change. 3. Scope of the Original Solicitation In RFP-2, the Department listed six criteria for evaluation. In the negotiations process, with guidance from DIT, the Department reworded certain criteria for brevity and clarity and removed two criteria that had already been fully addressed. The scope did not materially change. The negotiations process, however, did correct a flaw in RFP-2 by removing the weights from the scoring of the criteria. The chart below shows how the changes were not material alterations. RFP-2 Criteria • Negotiations Criteria • • Substantial Conformity to Solicitation Specifications RFP Desired Specification • Proof of Concept/Demonstration • • Vendor Cost Proposal • Vendor Cost Proposal • Vendor Relevant Experience and Reference Checks • • Vendor Financial Stability • Removed since this part of the process was properly completed by the evaluation committee during RFP-2 and did not need repeating Vendor Financial Stability • Reworded as “Formative & Diagnostic Assessment” for clarity* Reworded as “Personalized Learning” for clarity** Removed since this part of the process was properly completed by the evaluation committee during RFP-2 and did not need repeating 13 *As you state in the supplemental letter, “the whole purpose of the RFP is to procure ‘valid, reliable, formative and diagnostic reading assessments.’” (See RFP-2 p. 6). **In RFP-2, the Department had included personalized learning as a substantive desired specification. Personalized Learning is a strategic direction of the State Board and clarification was added through this rewording in the negotiations process. 4. Amplify’s Claims in the Supplemental Letter The theories purported in your protest are incorrect. The scope of the negotiations did not materially change from those of RFP-2. The Department did not change the rules of the procurement. Your claim that the Department “threw out” criteria is wrong. With guidance from DIT, certain criteria were removed because they were already satisfied for the evaluation committee and others were reworded for clarity. Properly following IT procurement policies, as discussed above, the criteria were not weighted during the negotiations process and were equal as a result. References and product demonstrations for Amplify and Istation were both properly completed and satisfactory. Being unweighted and equal, the vendors were equal for these criteria. With the guidance of DIT, the Department properly moved the cost proposal and vendor financial stability, both criteria from RFP-2, higher in priority without materially changing the scope. This did not have an impact on the outcome, though, because no adjustments were made by the evaluation committee based on the order of priorities. All the criteria were unweighted and equal. Amplify and Istation were both satisfactory for vendor financial stability. Being unweighted and equal, the vendors were equal for this criterion. Istation’s cost proposal provided during the negotiations had more strengths and fewer weaknesses than Amplify’s cost proposals provided during the negotiations. When given all the facts and the strengths and weaknesses for all criteria, the evaluation committee for the negotiations process unanimously agreed to continue further negotiations with Istation and recommended awarding to Istation. The evaluation committee had no need to adjust for other factors. 5. Cost and Value As discussed above, the State emphasizes the importance of equal weight first. This can potentially be followed by adjustments for non-price categories because a goal of State IT procurements is to drive best value principles when buying services on behalf of the citizens and taxpayers of North Carolina. A best value procurement is “a procurement process with the objective of reducing the total cost of ownership.” 09 NCAC 06A .0102(2). The “particular procurement methods used [in a best value procurement] are selected so as to result in the best value for the State in terms of the function to be performed or delivered.” Id. Procurement laws and policies, when followed, drive the best value by having equal weight for cost at first while allowing an evaluation committee to later adjust for other factors if necessary. Cost was a significant factor in RFP-2. If multiple vendors can provide reading diagnostic tools that each meets the requirements of North Carolina law, the Department must consider cost 14 in proper stewardship of the taxpayer dollars entrusted to it. Per procurement policies, this maximizes value for the citizens of North Carolina. The Department made clear to Amplify and Istation that cost would continue to be a significant factor in the negotiations process. To emphasize the importance of cost in the process, and with the guidance of DIT, cost was given a high priority in the Request for Negotiation communication sent to both vendors. To maximize fairness during the negotiations process, the Department gave each vendor the opportunity to improve their price offered to the State. Amplify offered a reduced bid of $3.8 million annually (plus professional development costs) compared to its bid of $6.1 million annually (plus professional development costs) in RFP-2. This bid was still objectively higher than Istation’s bid of $2.8 million annually (plus professional development costs). Though cost was given a high priority, the evaluation committee for the negotiations process did not use such priority in its determination. Every criterion was unweighted and equal. The evaluation committee made its recommendation based on evaluations of the unweighted criteria and had no need to adjust for other factors. Your claims in the protest are wrong. Also, in your protest letter dated June 24, 2019, you take issue with a newspaper article that reported Istation’s contract amount for the contract award and Amplify’s current contract amount. The Department answered the questions asked by the reporter. While we cannot control what a reporter decides to compare, you use the opportunity to point out that Amplify’s bid was $3,755,560 a year, which you say represents more than a 40% reduction from prior years. In the next paragraph, however, you claim that Amplify’s tool offers more than Istation, specifically, the TRC. You do not properly qualify this argument with the fact that the additional offerings would cost more than the amount of Amplify’s bid. Amplify’s bid referenced in the protest letter was $3.8 million (plus over $500,000 for professional development) and did not include the TRC. Amplify also proposed an alternate bid, which included the TRC and added Amplify Reading National Edition, along with additional professional development. This bid was for $10.1 million in the first year (plus $1.2 million for professional development). A top priority of any IT procurement is to find the best value for the State in terms of the functions to be performed or delivered. After a fair and unbiased process, Istation proved to be that best value to the State. 6. Recommendation to and Approval by the State Superintendent and the State Board of Education The evaluation committee for the negotiations process presented to me its unanimous recommendation to select Istation, and I approved the recommendation to select Istation. (See Exhibit F). Department staff and I then presented the recommendation to the State Board, and the State Board unanimously approved the recommendation to select Istation. 15 Attachment D Istation 1. Introduction Throughout this protest for Amplify, you make incorrect claims about Istation to argue in favor of Amplify’s request. Istation has already been used successfully in other states and more than satisfies the requirements to be the reading diagnostic tool for North Carolina. Contrary to your incorrect claims: • • • • • • • • • • • Istation satisfied the requirements of the procurement, Istation meets the requirements of law, Istation predicts students who are at risk of a reading difficulty, Istation screens for dyslexia, Istation is developmentally appropriate, Istation assesses required measures, Istation is effective at measuring a student’s progress, Istation allows for students to interact with and read aloud to teachers, Istation provides sufficient data to allow determinations of short-term student progress, Istation accurately and reliably determines which students will be reading on grade level by the end of the school year, and Istation is already on track for successful implementation in North Carolina. 2. Istation’s Use in Other States Istation has been adopted for statewide implementation in eight states: Texas, Florida, Colorado, Kansas, New Mexico, Idaho, Arkansas, and North Carolina. In addition to the eight statewide implementations, Istation has been approved by state departments as a K-3 literacy screener in the following states: Massachusetts, Mississippi, Michigan, Ohio, and Oklahoma. Idaho The State Department of Education in Idaho recently released preliminary spring reading scores on July 1. Scores went up across the board from fall to spring. “The preliminary results from this first year of the new, comprehensive Idaho Reading Indicator are promising,” state superintendent Sherri Ybarra said in a news release. “The improvements were considerable. In each grade, the number of students reading at grade level improved by at least 12 percentage points. In first grade, the improvement approached 24 percentage points. Put another way, more than 14,500 students who scored below grade level on the fall IRI advanced to grade level by spring.” Kevin Richert, What the New Reading Scores Say – And What They Don’t Say, IDAHO ED. NEWS (July 11, 2019), https://www.idahoednews.org/news/what-the-new-reading-scores-sayand-what-they-dont-say/. 16 Florida Based on a recent report from Florida, results are promising for third grade students who have used Istation, as districts partnering with Istation indicate higher growth for third grade standardized testing in May. In the districts of Alachua, Monroe, Pinellas, and Polk, third grade scores “averaged a growth rate of 5.58% over the course of the FSA 2015-2019 testing results. This is over 2% more than average growth rate of districts that have not partnered with Istation. This average growth rate comes out to 3.4% over the same span of years.” “Istation helps raise the potential for growth in each student by allowing both teachers and students to monitor student growth, and by providing individualized education plans for each student based off of needs. These plans are scripted so a new teacher can easily use them, or if there is a seasoned educator using the technology they can smoothly add their own material to the personalized lesson so that each student gets the education they need rather than the traditional “one-size-fits-all” approach. “I love that it (Istation) tiers my students and provides individualized interventions that I can use to help my struggling students,” said Jessica Lockwood, a third-grade teacher at Poinciana Elementary School on Key West.” Online Staff, Higher Growth Reported for Third Grade Standardized Testing In Florida Districts Partnering With Istation, THE BOCA RATON TRIB. (June 18, 2019), http://www.bocaratontribune.com/bocaratonnews/2019/06/higher-growth-reported-third-gradestandardized-testing-florida-districts-partnering-istation/. Arkansas A recent press release out of Arkansas speaks to reliability and validity of using Istation as a K-3 assessment. “Up to 40% of Arkansas's public kindergarten, first and second graders need additional support to achieve at a desired level in reading. That’s based on the 2019 results from the three different nationally standardized tests from which school districts can choose to give to pupils to gauge their achievements in reading and math. The state-authorized tests -Istation, Renaissance, and NWEA -- produced similar results, which indicates the tests are reliable measures,” said the state’s director of assessment, Hope Worsham. Cynthia Howell, Education Notebook, ARKANSAS DEMOCRAT GAZETTE (July 15, 2019), https://www.arkansasonline.com/news/2019/jul/15/english-reading-scores-stir-queries-tes-1/. New Mexico New Mexico awarded Istation with a statewide contract in 2016 and faced a four-week timeline for implementation. Cloudcroft Municipal Schools K-8 Principal and DTC shared the following comments from her faculty and staff: “This experience has changed our minds about Istation. We weren't a fan [sic] because of the quick turnaround we had to do. But the people with Istation have been very helpful and so nice so that was not as scary as we thought it would be. My teachers loved how quickly they got their data and access to intervention for students” (response on file with the Cloudcroft Municipal Schools). 17 ". . . I have an autistic student in third grade who is very negative, and it is hard to test because he refuses. He finished the entire test without redirecting from us. And even asked after the test when he could do it again. At one point he was belly laughing! That just doesn't happen with him on a test or a computer.” Id. “You personally relieved my personal concerns for utilizing the program with five-year old children! I'm on board now to support and encourage our teachers to give this a try . . . .” Id. “I am so relieved to say I really think we will love this new program! The Istation staff are all so very passionate and most of them are former teachers who used this in their own classrooms!” Id. “I feel very much at ease with it, and the other administrators I talked with were also very relieved and optimistic.” Id. “I have really been impressed with your company's responsiveness. I have dealt with support twice, and they were incredible.” Id. Kansas Kansas approved the entire program for Kansas Reading Success in 2015. A fifth grade teacher at Chanute Elementary School in Kansas shared how Istation has impacted her students’ learning, her teaching, and her classroom. Though Kansas differs from North Carolina because all schools use the curriculum, her comments about reports and simplicity for gathering data provide an important example of how teachers quickly gain insight into using data to inform instruction. “I love the reports! They give so much information on exactly what the students are getting and what they aren’t. For example, I had one Tier 3 student who was doing great on Istation. With each passing week, her scores were improving. When the monthly assessment was pushed in, however, she tanked it. I was able to drill in and see exactly which questions she had missed. After the analysis and a conversation with her, it was apparent she bombed it on purpose; she was nervous about moving up to a harder level. After much encouragement she is back on track and moving forward!” “I love that it is pushed in each calendar month. It helps keep the kids motivated and is so easy for the teacher. If a student is absent the day of the assessment, it’s just there the next time they log on . . . so easy on the teacher! Also, the reports generated from the assessments are amazingly data rich. I feel that it is meaningful data that drives both instruction and intervention. My Professional Learning Community (PLC) is considering using the Istation assessments as our common assessment next year to track student growth.” Lindsey Joyce, Year One With Istation: A Grade 5 Teacher Reflects, ISTATION BLOG (May 4, 2016), https://blog.istation.com/year-one-with-istation-a-grade-5-teacher-reflects. 18 3. Istation Feedback from North Carolina Eric Davis, Chairman of the State Board, told me that he discussed the State Board’s decision with education experts he trusts, and they told him they consider Istation one of the top diagnostic tools available (second only to another vendor that applied but did not meet all the criteria of the Read to Achieve law (not Amplify’s tool)). Multiple meetings of the State Board have occurred since the contract award to discuss Istation and its implementation in the 2019-2020 school year. The State Board has been supportive of the contract award and has not budged from its decision. When discussing the training schedule for Istation in a board meeting on June 28, 2019, State Board member Amy White said: “This is not a discussion about whether we’re going back to mCLASS.” T. Keung Hui, After outcry from educators, NC will delay use of computer-based reading test, THE NEWS & OBSERVER (June 29, 2019, updated June 29, 2019), https://www.newsobserver.com/news/politics-government/article232045842.html. At the same meeting, Amy White said: “I think having something new is always hard. Moving from one platform to another is always hard. But I think once they see it, touch it, feel it, they will realize that [Istation’s] going to be something very positive.” Board member Dr. Olivia Oxendine said: “This is more of an instructional feature advantage that I see in Istation. The other thing is that I go back to the year 2013, definitely 2014-15, and there was an incredible angst around the state among classroom teachers and curriculum specialists and superintendents about the amount of time taken away from sheer, pure instruction so that so much could be done with assessment, the assessment of students. So, we received during that period of time, probably about 18 months to two years, a barrage of emails from superintendents and teachers about not enough, not sufficient time given to the direct instruction of reading. So, I believe that we are going to have the opportunity to allow teachers to do what is called teaching reading and acknowledging and teaching to the science of reading, which is music to my ears.” Jack Hoke, the Executive Director of the NC School Superintendents Association, emailed me and told me that he attended a session in Asheville with Istation. He said it appeared to be an excellent tool. The number of positive responses from North Carolina educators continues to grow as more and more attend Istation trainings. Please see the current compilation of responses collected after educators completed face-to-face training with Istation. (See Exhibit G). 19 4. Amplify’s Claims are Incorrect a. Istation Meets the Requirements of Law Before this process, Istation was approved by the State Board as an assessment for proficiency in third grade reading under the Read to Achieve law and was used by others in North Carolina at their own expense. Istation meets all the criteria of the Read to Achieve law (G.S. § 115C-83.6): i. ii. iii. iv. Istation serves as a valid, reliable, formative, and diagnostic reading assessment for students in kindergarten, first, second, and third grades. Istation provides immediate data that can be used with the Education Value-Added Assessment System (EVAAS) to analyze student data to identify root causes for difficulty with reading development and to determine actions to address them. Istation provides formative and diagnostic assessments and resultant instructional supports and services to address oral language, phonological and phonemic awareness phonics, vocabulary, fluency, and comprehension using developmentally appropriate practices. Istation is administered by computer or some other electronic device as allowed by law. b. Istation Adequately Predicts Students Who Are at Risk of a Reading Difficulty The authors, Dr. Joseph Torgesen and Dr. Patricia Mathes took great care in constructing the ISIP ER item pool, basing the types and content on contemporary findings in early reading research. Istation’s ISIP ER measures are designed to be used for Universal Screening (benchmarks three times a year) and Progress Monitoring (monthly). The benchmarks and progress monitoring assessments are designed to test the same skills in the same format and with the same consistent and comparable results. While the questions will change for the individual student, the subtest, the format, and overall score composition will remain consistent between progress monitoring and benchmarks to ensure consistency in instructional design and instructional decisions between benchmark months. The advanced technology used in Istation’s adaptive format provides data to help teachers determine which students are at risk for reading difficulties and those who have missed critical foundational skills that are typically not assessed at later grade levels. For each student, the tasks adapt to lower or higher-level skills – meeting the student where he or she is on the student’s reading journey – and produces a reliable estimate of their ability. Reliability and validity are two important qualities of measurement data. Reliability can be thought of as consistency and validity can be thought of as accuracy. The ISIP Early Reading Technical Report (c 2011; 2016) describes a study conducted using ISIP Early Reading data collected from kindergarten through grade 3 in five elementary schools in north Texas. Regarding measures of reliability, data suggest consistently high levels of internal consistency in the subtest ability scores as well as the overall reading ability scores. ISIP ER also produces stable scores over time. Evidence of concurrent validity is found in the numerous strong, positive 20 relationships to external measures of reading constructs. Data suggest mostly large to very large criterion validity with scores from well-known measures of reading constructs, such as CTOPP, GORT-4, PPVT-III, TOWRE, WJ-II ACH, WLPB-R, and WIAT-II, as well as with TPRI and ITBS. In addition, classification accuracy and area under the curve estimates for third grade accountability measures meet generally accepted standards. Furthermore, predictive validity results show that ISIP Overall Reading is a stronger predictor than DIBELS ORF for state mandated Texas Assessment of Knowledge and Skills (TAKS) Reading, using scores from 1 to 5 months prior. Taken together, the evidence supports that ISIP ER produces reliable and valid data for measuring key areas of reading development such as phonemic awareness, alphabetic knowledge, vocabulary, and reading comprehension, as well as overall reading ability. Additionally, Campbell, L. O., Lambie, G. W., and Sutter, C. conducted a study to examine third grade ISIP-ER scores and Florida Standards Assessment English Language Arts (FSA-ELA) scale scores. Data were collected during the 2016-2017 school year from five public school districts in Florida. Researchers used a simple linear regression analysis to examine the correlation of Overall Reading Ability and Reading Comprehension to the FSA-ELA scores. Results of the study indicated ISIP ER scaled scores had a strong correlate to FSA-ELA scores and predictability bands were computed to identify cut scores to predict all achievement levels. This study provides evidence that Istation’s reading assessment can predict the FSA-ELA statewide examination scores for all achievement levels among third grade students. Campbell et. al., (2018). Measuring the Predictability of Istation Indicators of Progress Early Reading (ISIP-ER) Scores on Florida Standards Assessment Scores. University of Central Florida, www.ucf.edu/mirc. Recently, Campbell, Sutter, Lambie, and Tinstman (2019) conducted a study to measure the predictability of Istation’s Indicators of Progress (ISIP) Early Reading Scores on Renaissance STAR Reading. Researchers examined kindergarten, first, and second grade ISIP-ER scores (Overall Reading Ability and Reading Comprehension) in one county in the state of Florida. Data were collected during the 2017 – 2018 school year and a simple linear regression analysis was conducted to determine the correlation of the STAR scaled scores and the ISIP-ER overall reading scores. Results indicated ISIP-ER scaled scores had strong correlate to the STAR scores and predictability bands were computed to identify cut scores for accurate prediction for all achievement levels. This study provides evidence that ISIP Reading cut scores can predict the STAR statewide examination scores for all achievement levels among first and second grade students using computer-adaptive testing. Campbell et. al, (2019). Measuring the predictability of Istation Indicators of Progress Early Reading (ISIP-ER) scores on Renaissance STAR Reading scores. University of Central Florida, www.ucf.edu/mirc. 21 c. Istation Screens for Dyslexia and Meets the Statutory Requirements for Screening Dyslexia Istation is used in other states as a dyslexia screener for children. One of the authors of ISIP Early Reading (ISIP ER) was Dr. Joseph Torgesen, an Emeritus Professor of Psychology and Education at Florida State University. He is also the author, with Dr. Richard Wagner and Dr. Carol Rashotte, of two of the most widely used diagnostic tests for dyslexia, The Comprehensive Test of Phonological Processes (CTOPP) and the Test of Word Reading Efficiency (TOWRE). Dr. Torgesen was asked to establish the Florida Center for Reading Research (FCRR), which has been the premiere site for disseminating information about research-based practices related to literacy instruction and assessment for children since 2002. In the development of ISIP ER, several subtests were included that are often used to help in an initial screening for dyslexia. ISIP ER measures have high correlations with other measures that are used for dyslexia screening; therefore, ISIP ER measures can confidently be used to screen for dyslexia, particularly if they are given on a monthly basis. Additionally, Dr. Torgesen stated, “I consider this to be as good a screener as any that I know of.” Data related to student performance in the areas that can be predictive red flags of dyslexia and difficulties in other areas of reading (e.g., phonemic awareness, letter sound-symbol recognition, letter knowledge) are provided by ISIP ER to identify student skill weaknesses, evaluate student intervention plans, discuss student performance with administrators, and plan for parent-teacher conferences. From research, we know that early diagnosis of dyslexia is more likely when the problems occur in students who have strong abilities in other areas of language such as vocabulary. Istation measures listening comprehension of students in kindergarten as well as vocabulary K-3, thus allowing educators to gather assessment data for vocabulary, which has been an optional measure with mCLASS. Characteristics associated with reading difficulties are connected with spoken language. Difficulties in young children can be assessed through screenings of phonemic awareness and other phonological skills (Sousa, 2005). Additionally, Eden (2015) points out that “when appropriate intervention is applied early, it is not only more effective in younger children, but also increases the chances of sparing a child from the negative secondary consequences associated with reading failure. ISIP ER is a tool that can be used to screen for risk of dyslexia early in students’ academic careers.” Beginning with phonological processing, measures of phonological awareness tend to be the most predictive of success at early reading. As stated by Dr. Joseph Torgesen, “In young children, the most reliable indicators of risk for dyslexia are difficulties in the development of phonemic awareness and alphabetic knowledge and skills. The latter category includes lettersound knowledge and alphabetic reading skills (ability to “sound out” unknown words). If difficulties in these areas are not addressed by powerful instruction early in a child’s development (kindergarten and first grade), the student typically does not progress in the ability to read text fluently, and this is an extremely reliable indicator of dyslexia by the end of first grade. Since the ISIP-ER provides reliable, continuous assessment in each of these critical areas, it can be a powerful aid in identifying students with dyslexia early in their development.” 22 It is important to note that ISIP ER does include measures that can be used to screen students for dyslexia, and Dr. Joseph Torgesen has confirmed that requiring any additional measures is not necessary to accurately identify students who are at risk for dyslexia. However, Istation has worked with North Carolina to identify a process that has been used in Arkansas to further help and flag students for further evaluation. Using the model that has been successful in Arkansas, educators in North Carolina could use results from an ISIP screening that would be gathered at a time other than a benchmark, including measures that are not typically administered at that time/grade level, in order to identify students who are in need of additional advanced screening. The initial screening can take effect as follows: • • • • Kindergarten: Kindergarten students can be automatically screened upon login, at which point they can receive the additional subtest of alphabetic decoding. This subtest can be given later in the year in addition to the normal ISIP subtests for kindergarten: listening/language comprehension, phonological and phonemic awareness, letter knowledge, and vocabulary. 1st Grade: 1st graders will automatically receive the relevant subtests based on the initial screening. These subtests include phonological and phonemic awareness, letter knowledge, vocabulary, alphabetic decoding, reading comprehension, and spelling. 2nd Grade: 2nd grade students can be automatically screened upon login with the additional subtests of alphabetic decoding, letter knowledge, and phonological and phonemic awareness. These subtests would be in addition to the normal ISIP subtests for 2nd graders: vocabulary, reading comprehension, spelling, and text fluency. 3rd Grade: 3rd grade students can be automatically screened upon login with the additional subtests of alphabetic decoding, letter knowledge, and phonological and phonemic awareness. These subtests would be in addition to the normal ISIP subtests for 3rd graders: vocabulary, reading comprehension, spelling, and text fluency. For clarity, while the Department and I desire these functions for our Read to Achieve diagnostic tool, and Istation provides these functions, your claim that Session Law 2017-127 (HB 149) requires the Read to Achieve diagnostic tool to screen for dyslexia is not accurate. Session Law 2017-127 states that “it is the intent of the General Assembly that all students with specific learning disabilities, including dyslexia and dyscalculia, receive the necessary and appropriate screenings, assessments, and special education services to provide interventions for learning difficulties with language, reading, writing, and mathematics.” S.L. 2017-127, §1. The law goes on to state that “prior to the start of the 2017-2018 school year, local boards of education shall review the diagnostic tools and screening instruments used for dyslexia, dyscalculia, or other specific learning disabilities to ensure that they are age-appropriate and effective and shall determine if additional diagnostic and screening tools are needed.” Id., at §4. The law also requires the State Board to craft definitions related to dyslexia identification and to ensure the availability professional development opportunities related to identification of dyslexia. The law does not actually require the Read to Achieve tool to serve as a dyslexia screener for the state. Id., at §2-3. 23 Again, though, I agree that the Read to Achieve diagnostic tool serves as a good opportunity to provide local school districts with dyslexia screening and am pleased that Istation does so. d. Istation is Developmentally Appropriate One important piece of Istation is the modeling opportunity provided by the teacher for students before students interact with the assessment. The computer-adaptive nature of Istation ensures that each student, regardless of grade level, will be assessed in a way that identifies his or her individual proficiencies as well as challenges in reading. Data are immediately available through benchmarks and progress monitoring to differentiate instruction in order to address individual needs. Istation reports also provide for grouping of students by skill performance for small group instruction. Recommendations for individual and small group instruction, as well as interventions are provided through Istation to support the teacher in data-based decision making and instructional planning. Teachers have access to a repository of resources, including lesson plans and instructional activities specific to each student’s needs. Dr. Joseph Torgesen states, “A computer-adaptive assessment like ISIP ER is developmentally appropriate because it administers items that are individually selected to be at the right level of difficulty for each child. In addition, because of its lively and engaging format, it reduces many of the problems, such as distraction and lack of interest, that are sometimes encountered when young children are assessed.” With Istation, teachers will assess their own students and obtain valid, reliable, and authentic data using technology tools and skills. For mCLASS: Reading 3D, there is variance in the level of teacher training that often yields unreliable data, especially as it relates to the components with more difficult scoring rules, such as NWF and PSF, and TRC. There are considerable concerns related to the subjectivity associated with administration of mCLASS Reading 3D, especially TRC measures. This is evident in the fact that teachers do not assess their own students at MOY and EOY. Instead, teachers across the state have assessed students in another classroom on their grade level to increase the objectivity in administration using mCLASS Reading 3D and obtain scores for students that are nonbiased and accurate. Even though students interact via technology with Istation, students are expected to read text just as students read text on Flex or any other mCLASS measure, but ISIP measures will adjust based on the student’s performance. With Istation, students absolutely can produce the sounds and read the words out loud or silently. The teacher will have the flexibility to monitor students based on individual need as they apply their knowledge to complete tasks to demonstrate proficiency on the critical skills in reading. In addition, K-12 students have Information and Technology Essential Standards required by the state which include the following: Kindergarten: Use technology tools and skills to reinforce classroom concepts and activities. As documented in the ISIP Early Reading Technical Report: following a Computer Adaptive Testing (“CAT”) model, each item within the testing battery is assessed to determine how well it discriminates ability among students and how difficult it actually is through a process 24 called Item Response Theory (IRT) work. Once item parameters have been determined, the CAT algorithm can be programmed. Using this sophisticated algorithm, the computer selects items based on each student’s performance, selecting easier items if previous items are missed and harder items if the student answers correctly. Through this process of selecting items based on student performance, the computer generates “probes” that have higher reliability than those typically associated with alternate formats. This results in a better reflection of each student’s true ability. The process described above provides evidence that the true ability of students in K3 will be established using the computer-based assessment model of ISIP ER Assessment. As it relates to the developmental appropriateness of using a computer-adaptive assessment like ISIP ER for young children, Dr. Joseph Torgesen stated, “There is no research that I am aware of to indicate that the small increases in time spent in front of a computer screen using the ISIP-ER constitute a risk for young children. In fact, monthly assessments using the ISIP-ER very likely involve an extremely small proportion of the total time young children spend in front of computer, or TV, or IPAD screens in our modern world.” Lastly, while Istation does meet the criteria as described above, in an attempt to support Amplify’s approach, your protest letter added inappropriate context for “developmentally appropriate” attached to the description of this RFP specification around observation-based practices. The actual specification is: “Describe how the proposed solution directly assesses reading and pre-reading behaviors to support students’ learning development at the various grade levels to inform instruction, including any observation-based practices if applicable.” e. Istation Assesses Required Measures As noted in the ISIP Early Reading Technical Report (c 2011; 2016), ISIP Early Reading follows a continuum of learning that, research indicates, is predictive of later reading success. Skills build upon skills, and the sequence of subtests builds upon prior subtests. As skills of lower-level difficulty are achieved, they are eliminated from the test battery and more advanced skills are added. Importantly, there is no other continuous progress monitoring assessment tool that measures vocabulary and comprehension accomplished through short tests, or “probes,” administered at least monthly, that samples critical areas that predict later performance. Id. ISIP Early Reading provides growth information in the five critical domains of early reading: phonemic awareness, alphabetic knowledge and skills, fluency, vocabulary, and comprehension. It is designed to (a) identify children at risk for reading difficulties, (b) provide automatic and continuous progress monitoring of skills that are predictors of later reading success, and (c) provide immediate and automatic linkage of assessment data to student learning needs, which facilitates differentiated instruction. Below is a grade level breakdown of how Istation ISIP ER measures assess all required measures plus listening comprehension, vocabulary, and spelling that are not assessed with mCLASS: Reading 3D. Id. Kindergarten (Subtests - Listening Comprehension, Phonemic Awareness, Letter Knowledge, Vocabulary): Kindergarten students require assessment of their growth in phonemic awareness, alphabetic knowledge and skills, and vocabulary. Fluency in letter names and sounds facilitates 25 spelling, but these skills are usually not developed sufficiently to assess spelling ability. Their reading skills are also rarely sufficiently developed to usefully assess reading fluency and reading comprehension. In general, research has shown that phonological awareness and letter sound knowledge in Kindergarten are predictive of Grade 1 outcomes. For children at risk of reading difficulty due to poverty or language background, vocabulary is critical to reading success (Foorman, Anthony, Seals, & Maouzaki, (2016); Snow et al., 1998; Dickinson & Tabors, 2001). Vocabulary assessments for Kindergarten students are mostly “tier 1” words and items to assess children’s knowledge of prepositions and verbs of varying tense, since these classes of words are particularly difficult for young children. Id. Grade 1 (Subtests - Phonemic Awareness, Letter Knowledge, Vocabulary, Alphabetic Decoding, Comprehension, Spelling, Connected Text Fluency, Oral Reading Fluency): It is important to continue to monitor students’ development of phonemic awareness and alphabetic knowledge and skill, because struggling students may continue to have difficulty in these areas. The development of accurate and fluent decoding skills should be monitored, since these foundational skills for reading accuracy undergo major development. Word recognition at the beginning of Grade 1 has been found to be predictive of Grade 1 outcomes. Spelling has also been found to be a predictor of oral reading fluency. Vocabulary growth is important in the development of reading comprehension. As soon as students can demonstrate the ability to read connected text with reasonable accuracy and understanding, reading fluency (timed reading with meaning) should be monitored. Continued growth in vocabulary should be assessed, as well as reading comprehension. Id. Grade 2 (Vocabulary, Comprehension, Spelling, Connected Text Fluency, Oral Reading Fluency): Word reading continues to be a strong predictor of Grade 2 outcomes, with reading fluency and comprehension becoming increasing important predictors. Second graders need continued monitoring of their decoding abilities, because struggling students may still have difficulty in this area. Reading fluency is critical through second grade, since students must make strong growth in this skill to maintain grade level reading proficiency. The development of reading comprehension is dependent on fluency and vocabulary. Sight vocabulary must grow rapidly in second grade to keep pace with expected reading outcomes. Thus, continued growth in Spelling, Vocabulary and Reading Comprehension should be measured. Id. Grade 3 (Vocabulary, Comprehension, Spelling, Connected Text Fluency, Oral Reading Fluency): Reading fluency and comprehension are strong predictors of Grade 3 outcomes. The primary dimensions of reading growth that should be measured in third grade are reading fluency, reading comprehension, spelling and vocabulary. Id. f. Istation is Effective at Measuring a Student’s Progress ISIP ER measures are designed to be used for Universal Screening (benchmarks three times a year) and Progress Monitoring (monthly). The benchmarks and progress monitoring 26 assessments are designed to test the same skills in the same format and with the same consistent and comparable results. While the questions will change for the individual student, the subtest, the format, and overall score composition will remain consistent between progress monitoring and benchmarks to ensure consistency in instructional design and instructional decisions between benchmark months. As stated in the ISIP ER Technical Manual (c 2011; 2016), “With recent advances in Computer Adaptive Testing (“CAT”) and computer technology, it is now possible to create assessments that adjust to the actual ability of each child. Thus, CAT replaces the need to create parallel forms. Assessments built on CAT are sometimes referred to as “tailored tests” because the computer selects items for students based on their performance, thus tailoring the assessment to match the performance abilities of the students. This also means that students who are achieving significantly above or below grade expectations can be assessed to more accurately reflect their true abilities.” Furthermore, the Technical Manual (c 2011; 2016) indicates, “There are many advantages of using a CAT model rather than a more traditional parallel forms model, as is used in DIBELS. First, it is virtually impossible to create alternate forms of any assessment that are truly parallel. Thus, reliability from form to form will always be somewhat compromised. However, when using a CAT model, it is not necessary that each assessment be of identical difficulty to the previous and future assessments. Following a CAT model, each item within the testing battery is assessed to determine how well it discriminates ability among students and how difficult it actually is through a process called Item Response Theory (IRT) work. Once item parameters have been determined, the CAT algorithm can be programmed. The use of CAT algorithms also creates efficiencies in test administration. The adaptive item algorithm allows the computer to adjust item difficulty while the child is taking the test, quickly zeroing in on ability level. Thus, the use of CAT algorithms reduces the amount of time necessary to accurately determine student ability.” The entire assessment battery for any assessment period requires 40 minutes or less. It is feasible to administer ISIP Early Reading assessments to an entire classroom, an entire school, and even an entire district in a single day, given adequate computer resources. Classroom and individual student results are immediately available to teachers, illustrating each student’s past and present performance and skill growth. Teachers are alerted when a particular student is not making adequate progress so that the instructional program can be modified before a pattern of failure becomes established. With the advanced computer technology, data are provided in real time to help teachers determine which students are at risk for reading difficulties and those who have missed critical foundational skills that are typically not assessed at later grade levels. For each student, the computer adapts to lower or to higher level skills and produces a reliable estimate of their ability. 27 g. Istation Provides Opportunities for Students to Read Aloud to and Interact with their Teachers Students can read to and interact with teachers with Istation in place. The teacher can and very much should be highly involved in the reading and learning of a child. Teachers are encouraged to sit with the students as they read aloud and participate in learning activities. For example, in Istation’s Oral Reading Fluency subtest, the teacher or reading specialist is asked to listen to the child read aloud and identify key reading behaviors and skills. With Istation, students absolutely can produce the sounds and read the words out loud to teachers. The teacher will have the flexibility to monitor students based on individual needs as they apply their knowledge to complete tasks to demonstrate proficiency on the critical skills in reading. The purpose of K-3 formative assessments is to provide data pertaining to a student’s performance level, not to substitute for teacher interaction. Teacher interaction is critical during instructional time, as research consistently demonstrates evidence-based instruction informed by meaningful data produces changes in student outcomes. Teachers who monitor their student’s progress and use the data to inform instruction and decision making have higher student achievement. In fact, results from a study conducted by SERVE (2015) indicated teachers reported using many different measures to assess students prior to the introduction of mCLASS: Reading 3D and 96% of teachers indicated that they used formative assessment to inform instruction before mCLASS: Reading 3D. Since a CAT like the ISIP-ER is efficient, and items are individually selected each time a student takes the test, it can be given frequently enough to produce reliable estimates of student growth from month to month. This is, of course, very important information for the teacher, who can adjust teaching strategies, or teaching intensity, based on whether or not each individual student is growing in the critical reading skills assessed by the ISIP-ER. Valid, timely, and reliable assessment data gathered in an effective and efficient way provides more time for the teacher to interact with students as he or she engages in effective small group, targeted reading instruction. Gains in instructional time allow teachers to use small group instructional sessions to determine where individual children fall on the continuum of reading skills and provide systematic, explicit instruction to promote reading proficiency. Flex assessments in mCLASS are also computer-based and gather data through a device. Students are asked to read and stop to select a correct answer. Teachers have reported an appreciation for the option to use Flex due to the reduced time spent assessing their whole class. Istation takes this flexibility to another level in a way that still allows for audible phonemics to be sounded out by students, while saving more instructional time for both students and teachers. ISIP-ER can provide an accurate and efficient assessment of student ability because it uses advanced computer technology to select items that are “just right,” in terms of their level of difficulty, for each individual student. Thus, students do not waste time responding to test items that are “way too hard” or “way too easy.” The computer can also estimate when the student has 28 responded to enough items of the appropriate level of difficulty in order to produce a reliable estimate of their ability. As the Putnam (2016) study demonstrates, with the relationship between technology and literacy being at the forefront of many educational decisions, “the teacher’s role in the classroom is strengthened by the introduction of new technologies,” and “technology supplements, not supplants the teacher in the classroom.” h. North Carolina is Already Engaged in a Successful Implementation of Istation The Department and Istation have developed a statewide implementation schedule to have initial training for all educators complete before the start of school. With the ease of administering Istation in combination with the level of support offered in online modules, faceto-face training, webinars, and on demand links, successful implementation of Istation is well underway. The training and implementation plan include the following steps: • • • • • • Begin using Istation at the start of the 2019-2020 school year (with delay in metrics described below) Gather data during the fall to start becoming familiar with the assessments Start using data to inform instruction Delay the use of data to measure growth for EVAAS until MOY benchmark; Use first official benchmark in the winter (MOY) and end of year benchmark (EOY) for EVAAS purposes Train all teachers by start of school with a schedule including: − In-person workshops (face-to-face regional-based half day train the trainer model for one campus leader per school and district leader/RtA Contact) − On-demand webinars (monthly technical and educator live webinar in Q&A environment recorded and available for on demand viewing) − Learning modules (monthly timely topics in released online modules/podcasts for educators to view brief course, complete quiz, get certificate of completion) − Technical assistance − Ongoing support from the K-3 Literacy team 29 Below is the “Getting Started with Istation Roll Out Plan,” which has the support of local superintendents and the State Board. Summer 2019 (June, July, August) • Istation enrollment and deployment activated in schools • Live webinars hosted • In-person regional trainings hosted • On-going implementation support • Districts can begin using Istation as early as July, but it will not count in metrics Fall 2019 (September, October, November) • Students take Istation’s ISIP assessment to begin to learn about the program • Additional in-person regional trainings hosted • Additional live webinars hosted • Fall is a “Getting Started” learning opportunity. Data will not feed into EVAAS Winter 2019-2020 (December, January, February) • Ongoing progress monitoring continues • Ongoing training continues • January: The first benchmark window for 2019-2020 opens for EVAAS purposes (MOY)* Spring 2020 (March, April, May) • Ongoing progress monitoring continues • Ongoing training continues 30 • May: End-of-year benchmark window for 2019-2020 opens for EVAAS purposes (EOY)* (*Traditional School Calendar) As of July 25, 2019, over 2,735 North Carolina educators and districts leaders have engaged in webinars and in-person trainings. Educators are already modeling ISIP measures with students in year-round schools. • 989 total attendees in tech webinars • 1,241 total attendees in live educational webinars • 505 total attendees in person face to face workshops On a survey following training, participants were asked to, “Rate how helpful you feel Istation’s ISIP assessment will be for you and your campus on a scale of 1-5 stars, with a 1 being not very helpful, and a 5 being very helpful.” To date, results indicate: 32% = 5 stars, 38% = 4 stars, 23% = 3 stars, 6% = 2 stars, 2% = 1 star Additionally, technical integration of Istation and the Department has been a smooth process. Integration with PowerSchool has been seamless. Rostering occurs automatically with nightly updates so that any changes are applied instantaneously. Integration with NCEdCloud has been successful, and we already have teachers logging in successfully to Istation via NCEdCloud. • 114 districts/charters signed up through registration • 450 campuses rostered • 105,000 students rostered • 629 teacher logins i. Istation Satisfied the Requirements of the Procurement As discussed in this response, Istation satisfied all the requirements of the procurement. j. Istation’s ISIP-ORF was not Considered in the Contract Award This feature described by Amplify was not a factor in the contract award. 31 Attachment E Amplify In your protest letter, you state, “The agency must also consider factors including whether the vendor complies with industry standards, the vendor’s past performance with the state, and the probability of the vendor providing the required service on time. 09 NCAC 06B.0302(1)(f)(iii).” In responding to your letter with these discussions of Amplify’s tool, mCLASS, and Istation, I would like to take this opportunity to share concerns with Amplify’s product. a. Teachers’ Concerns with mCLASS A study was conducted by the SERVE Center at UNC Greensboro who contracted with the Department to report on initial Read to Achieve implementation. One major finding for the 2014 Year One Report was, “implementation challenges identified by respondents that need attention, mostly related to administering benchmarking and progress monitoring assessments and a perception that reading assessments are taking too much time from instruction.” Given the results of 2014 report, the Department again contracted with the SERVE Center to conduct the current (2015) study to further examine teacher perceptions and assessment processes of mCLASS: Reading 3D in classrooms due to the questions from educators and the State Board as to how long the assessments in mCLASS: Reading 3D were taking, whether students lost too much instructional time or time spent on other educational activities, and whether the results of the formative assessments were being used to modify and tailor individual instruction. Researchers analyzed data from teacher logs, observations, and interviews of 100 teachers from 37 randomly selected schools across the state and 6 schools that the Department recommended as exemplars, as well as results from 749 teachers’ responses to a survey in order to gain an understanding of the assessment process. Results indicated an overwhelming concern with the time involved with mCLASS: Reading 3D that is taken from providing quality instruction to students that improves reading skills. Nearly half (48%) of teachers indicated concern about what students were doing while others were being assessed and reported they felt their students were not receiving high quality instruction. While teachers reported they like the data, they did not feel it was worth the time spent assessing and away from instruction. All K-3 teachers who were surveyed or interviewed indicated concern for time spent doing benchmark assessment and progress monitoring. More specifically, K-2 teachers who were interviewed stated they spent 24 hours on average assessing during benchmark periods, while K2 teachers who were surveyed indicated 26 hours. Third grade teachers’ survey results indicated 28 hours. Among the teachers surveyed, 74% indicated progress monitoring takes up too much instructional time. Teachers were concerned with the frequency required to assess students at “yellow” level (every 20 days) and “red” level (every 10 days) because the frequency did not allow sufficient time to teach deficit skills to make improvement between monitoring. Participants reported they would prefer to spend the time currently devoted to mCLASS: 32 Reading 3D assessments for guided reading and/or small group instruction (42%) and more or higher quality instruction (38%). Overall, 50% of third grade teachers surveyed were “slightly” or “not at all satisfied” with the overall utility of Amplify in providing data to identify and intervene with students experiencing reading difficulty. Teachers reported insufficient time to use benchmarking and progress monitoring data, 52% and 46%, respectively. While 72% agreed or strongly agree that it is a meaningful use of time to analyze benchmarking data, 89% of teachers surveyed thought that the Amplify assessment instrument took too much time away from instruction and 64% thought the information gained through Amplify assessment was not worth the loss in instructional time. Anderson, J., Hutchins, B., Howse, R., Amwake, L., Dufford-Melendez, K., & Wynn, L., et al (2015). Read to Achieve Evaluation Year Two: An Evaluation of the Implementation of Read to Achieve Assessment and Reding Camp. SERVE Center at the University of North Carolina at Greensboro. The complete (2014) Year One Report can be accessed at: http://www.serve.org/uploads/files/RtA%20_report_SERVE_9.5.14.pdf. b. Amplify’s Accuracy on Universal Screening You note in your letter that a core component of the Read to Achieve program is universal screening. Recent research conducted by REL Southeast (2019) examined longitudinal data from mCLASS: Reading 3D to predict outcomes on grade 3 EOG. In the study, data were analyzed using classification and regression tree (CART) analysis to identify decision rules for identifying students in grades K-3 as at risk or not at risk for reading problems at the end of grade 3. The following administration points of the mCLASS 3D reading were used as predictors of grade 3 EOG: kindergarten middle of the year (MOY), grade 1 beginning of the year (BOY) and MOY, grade 2 BOY, and grade 3 BOY. The study used a longitudinal statewide sample of 91,855 students entering kindergarten in the 2014-2015 school year and completing grade 3 in 2017-2018. Results revealed a large number of students were misidentified as likely to be non-proficient in grade 3 in all of the models, with the exception of a model in grade 3. While prediction models beginning with grade 1 MOY were acceptable for identifying students who are likely to be proficient by the end of grade 3, more information was needed in order to accurately identify students who are at risk of not being proficient at the end of grade 3. Less than 80 percent of students were accurately identified as at risk of poor performance (that is, the sensitivity rate was found to be below a recommended minimum standard of .80). Grade 3 BOY with North Carolina’s grade 3 beginning-of-grade assessment as an additional predictor was the only model that correctly classified over 80 percent of students as at risk or not at risk for reading problems at the end of grade 3 (that is, both the sensitivity and specificity rates, respectively, were found to be greater than .80). Findings of the study are limited to the cohort and assessments included in the analysis and may differ with additional cohorts and/or assessments. However, the study findings point to the need to supplement the mCLASS student scores with informal, curriculum-based assessments when setting instructional objectives and differentiating classroom instruction. The REL Southeast is in the process of preparing a full report with the study findings. 33 c. Amplify is Only One of Many Opportunities for Teacher-Student Engagement You claim in your letter that Amplify’s “direct engagement with students’ thinking during the assessment also provides more reliable results.” But teachers should be listening to their students reading daily, regardless of the benchmark assessment used three times yearly. While different, both Istation and Amplify offer opportunities for teachers to monitor progress, determine areas of need, and work closely with students. Additionally, with implementation of Amplify’s mCLASS, teachers have not been allowed to assess their own students for MOY or EOY due to fidelity issues. Therefore, the student is assessed by another teacher with whom the student may not have rapport, which can certainly impact the reliability of results. On the other hand, with Istation, students can be assessed by their own teachers in their own classrooms. d. Decreasing Reading Scores in North Carolina In your letter, you also state that “mCLASS’s benefits have not only been seen in North Carolina schools, but nationwide. Amplify has partnered with schools across the country and has a demonstrated history of successful implementations.” While we do not dispute that Amplify’s mCLASS has some strengths, reading scores have gone down in North Carolina since Amplify was fully implemented across the state. (See Exhibit H). e. Amplify is Open to Inappropriate Actions by Teachers Despite years of training on the assessment, school districts continue to report issues with inappropriate actions (such as sharing secure test materials) by teachers using mCLASS, resulting in testing irregularities with mCLASS. In 2018-19 alone, 23 districts reached out to the Department for guidance in handling these issues. While formal investigation results are still coming in, at least 14 testing irregularities have been formally reported to the Department’s accountability division for 2018-19, with retesting being the usual remedy. f. Amplify Technical Issues Present Challenges for North Carolina Educators Educators across the state, tech directors at school districts, the Department’s IT division, and K-3 Literacy personnel experienced significant challenges in 2018 due to issues with Amplify incorrectly assigning students to class rosters. Amplify failed to timely address the situation, making matters worse. There were 250 mCLASS related issues created in our DPI-RtA Jira ticketing system and October of 2018 during BOY (Note: In September and October, the issues were imported into Jira from the classing Department spreadsheet in Sharepoint, so a resolution is not reported, but data are included). Because of the magnitude of mCLASS related issues in October, the Department had to hire a temporary technical specialist to manage and remedy the volume of Amplify issues. 34 Below is a breakdown of the time Amplify took to resolve issues reported each month from September 2018 to July 2019, with tables for both normal priority and high/critical priority issues. On average, it took 21 days for Amplify to resolve normal priority issues, ranging from 0-46 days for resolution. However, a requirement from the existing contract with Amplify (BAFO 3 Contract) for resolving a normal priority issue states the following: “The Contractor will provide a preliminary status update and, to the extent practical, preliminary remedy to the State not later than the next business day after the problem is initially logged with technical support.” (See Exhibit I, pg. 20). Furthermore, the average resolution time for Amplify for high priority/critical issues was 9 days, ranging from 0-43 days for resolution. Please note the requirement from the contract with Amplify in effect in 2018-19 (BAFO 3 Contract) for resolving a high to critical priority issue states the following: “The Contractor will provide a preliminary update and, to the extent practical, a preliminary remedy to the State within one (1) hour after the High Priority problem is initially logged with technical support.” (See Exhibit I, pg. 20). 35 Amplify’s past performance with the State and untimely provision of services leave much to be desired. Amplify’s unreliable reading diagnostic tool threatens to delay the identification of poorly performing students, which ultimately delays providing these students with the additional help they need. g. No Viable Reading Camp Solution An important component of the Read to Achieve legislation is the opportunity for students to attend summer reading camps to improve their skills. Amplify does not provide a viable solution for these reading camps because it only has an offering for BOY, MOY, and EOY periods in a given school year. Amplify’s common practice since the state of North Carolina began using mCLASS has been to roll the entire state into the next school year and then invalidate and delete all TRC data following the conclusion of a reading camp and prior to the start of BOY. Yes, delete data. This process was challenged this year when it was discovered and Amplify justified deleting data because they had always done it this way. It became a major issue for the 2018-2019 school year and beyond due to new requirements to report 1st and 2nd grade data, which was not required in previous years. In previous years, 3rd grade data were reported, then invalidated and deleted once PowerSchool rostering data file was reactivated at the beginning of the year. This was not possible for 1st and 2nd grade, however, because they would continue to use mCLASS in subsequent years. When pressed to provide an alternative solution for reading camps that provides an “isolated environment,” it was discovered that Amplify “shares their environment” with all clients making it difficult for Amplify to create an isolated instance to meet the needs for collecting and reporting data for reading camps. It should be noted 36 that Istation offered a solution within 48 hours of the request that will also flag the 3/4 transition students for continued monitoring and never delete data. h. Lexile Level Issues Amplify identifies texts of their Atlas Kits with Lexile levels. Based on the levels provided by Amplify, TRC level Q was identified as the text level for third grade students to demonstrate grade level proficiency. Based on the Lexile level of 770, level Q most closely aligned with NC End of Grade Lexile of 725. Department and local district staff have raised concerns about inaccuracies in the levels of the books provided by Amplify. This “mis-leveling” means that students’ reading proficiency was tested using books that were too hard, or too easy, for them. As a result, teachers and administrators are getting inaccurate information on student reading abilities and making instructional and promotional decisions for students based on that information. Examples of concerns include: • • • Nearly half of Amplify test books do not have a Lexile Level identified on the common search engine Lexile.com. MetaMetrics (Lexile.com headquarters) is well known among educators as identifying Lexile Levels of texts. A book that Amplify identified as Level Q (Lexile 770) is listed as Level M (Lexile 570) on Lexile.com, potentially allowing students to pass third grade without reaching a high enough Lexile level to read at grade level. A book that Amplify identified as Level C was actually Level E. This may have affected kindergartners who are supposed to reach at least a Level C to be considered proficient. On Lexile.com, the Lexiles are very different within the same band of letters in the Atlas TRC Inventory. Many of the texts, within the Atlas Kits, cannot be found using a common search engine, and the majority of the identifiable books are mis-leveled. As of April 16, 2019, 32 of the 74 books from levels A-Z in the Amplify Atlas Kits had no measured listing at Lexile.com. 37 EXHIBITS Exhibit A Evaluation Committee Member’s Confidentiality Agreement Exhibit B Evaluation Committee Member’s Statement Regarding Conflict of Interest Exhibit C January 8, 2019 Text Message Chain – Breach of Confidentiality Exhibit D November 19, 2018 Evaluation Consensus Meeting Notes (pp. 1-2) Exhibit E Read to Achieve 2018 Weighted Criteria Chart Exhibit F June 7, 2019 Contract Award Recommendation Exhibit G Positive Responses from North Carolina Educators After Istation Training Exhibit H North Carolina Reading Scores Exhibit I 40-IT00107-16 / BAFO 3 (pp. 20-21) Exhibit A EVALUATION COMMITTEE MEMBER'S CONFIDENTIALITY AGREEMENT For RFP # 40-RQ20680730 - Read to Achieve Diagnostics - Software as a Service (RtAD­ SaaS) Pursuant to North Carolina's Administrative Code 09 NCAC 06B.0103, all information and documentation (verbal and written) relative to development of a contractual document is deemed ''confidential" and shall remain confidential until successful completion of the procurement process. Therefore, Evaluation Committee Members (both voting and contributing advisors) are required to keep all comments, discussions, and documentation confidential until a notification of award has been made by the Issuing Agency for this solicitation. By participating in this Evaluation Committee, you agree to not divulge any information to an unauthorized person in advance of the time prescribed for its authorized release to the public. This includes co­ workers, supervisors, family, friends, etc. If it is discovered that there has been a breach of confidentiality by a member of this Committee, he/she will be immediately excused by the Committee Chair until further notice. The solicitation may be cancelled and a new solicitation may be issued with a new Evaluation Committee. In addition, the issue will be referred to the employee's department director or agency head. Department directors or the heads of autonomous agencies shall be responsible for the preliminary examination and investigation of reports from employees of any violations which compromise the procurement process. If, following a preliminary examination and investigation, the department director or agency head finds evidence of a violation or finds that further investigation is warranted, a report shall be submitted to the respective Human Resources Office for potential disciplinary action. By signing below, I certify that, as a member of this Evaluation Committee, I will keep all comments and discussions, preliminary/ working evaluation notes, and all other information (verbal and written) regarding the above referenced solicitation, confidential until after a notification of award has been made by the Issuing Agency. Signature Date Exhibit B EVALUATION COMMITTEE MEMBER'S STATEMENT REGARDING CONFLICT OF INTEREST AND DISCLOSURE For RFP # 40-RQ20680730- Read to Achieve Diagnostics - Software as a Service (RtAD-SaaS) The following organizations have submitted a bid proposal and response to the above solicitation: 1) Amplify Education Inc. 2) Curriculum Associates (i-Readv) 3) lmaqination Station Inc. (!Station) 4) NWEA Each member involved in the evaluation process must verify that he I she has no personal, financial, business or other conflicts of interest, with regard to this procurement and his/ her official duties as an evaluator. North Carolina General Statute § 143-58.1 prohibits unauthorized use of public purchase(s) or contract procedure for private benefit. Therefore, by signing this statement, you certify that neither you nor members of your immediate family currently have or expect to gain, any personal, financial, business or other benefit, from the potential contract awarded to any of the competing, bidding-vendors listed above; and that, neither you nor m�mbers of your immediate family have any potential conflicts of interest in the organization(s} listed above, including any subcontractor referenced in their respective proposals, that could influence, or be reasonably perceived as influencing, your evaluation or recommendations for this solicitation. If it appears as potential conflict of interest between your official duties as an evaluator and your personal interest, you will be excused from participation by the Evaluation Committee Chair. Please return this form unsigned and a replacement evaluator will be assigned. You need not disclose the relationship or conflict. In addition the issue will be referred to the employee's Department Director or Agency Head. Department Directors or the Heads of autonomous agencies shall be responsible for the preliminary examination and investigation of reports from employees of any violations which compromise the procurement process. If, following a preliminary examination and investigation, the Department Director or Agency Head finds evidence of a violation or finds that further investigation is warranted, a report shall be submitted to the respective Human Resources Office for potential disciplinary action. By signing below, I certify that I do not have, nor does any member of my immediate family have, any personal, financial, business, or other conflicts of interest in the bidding-vendors listed above. Signature Date Exhibit C lo - I agree ...no matter how wt try to soften it You're eicactly right!! Geez 1 What ,s going on ? 0\4G! I know they were shocked! So meone, • should've recorded it on her phone! Oh yeah that would be tough...who else on the team was ,n the room? Have they r.amed a replacement for� Dnd and and and one of Mark's staff voted for helping teac�crs She sa1e1 he ta:a:ed aocut hetp,n� teichers and never once mentioned ch.ldten ana !.av1ng the tedchers · t•me She and and vc,ted for children. Ass The sad th,ng is, he may wm his next race because he will talk about how he helped teachers! Exactly! Achieve 2018 Evaluation Consensus Meeting notes Consensus Meetin to rank the ro State Board Room Education Buildi November 19 2018' 8:30 AM 5:00 PM November 20 Linda Lowe and Srirekha Viswanathan vendors Rale' h. 2018 8:30 2:30 PM Exhibit Next Meeti TBD Voting Member Participants Abbey Whitford Amy Jablonski Chloe Gossage Dewey Kristi Day Lynne Loeser Matt Hoskins Pam Shue Rebecca Belcastro Susan Laney Thakur Karkee Non-Voting Member Participants Courtney Moates Constance Bridges Deborah Wilkes Erika Berry Giancarlo Anselmo Gin Hodge Julien AlHour KC Hunt K.C. Elander Linda Lowe Meera Phaltankar Mia Johnson Paola Pilonieta Shaunda Cooper Srirekha Viswanathan Tonia Parrish Tymica Dunn Agenda Items The agenda for this meeting was to discuss the evaluation notes from the independent reviews by voting and non- voting members, reach consensus to rank the proposals and determine the next steps in this procurement. This meeting summaiy includes notes from the meeting on 11-19-2018 and 11-20-2018. Meeting Summary (11-19-2018) 1. Sri kicked off the meeting by thanking the participants for their thorough review and participation at the consensus meeting. 2. The intent of the meeting and the approach to evaluate all the criteria were discussed - including being objective, impartial, unbiased and fair in all aspects of the evaluation process and arrive at a consensus. All proposals should be ranked consistently. Consensus means general agreement and not unanimity. 3. The six evaluation criteria in proposal were reiterated: Substantial Conformity to Solicitation Specifications RFP Desired Specification Proof of Concept/Demonstration Vendor Cost Proposal Vendor Relevant Experience and Reference Checks Vendor Financial Stability ?nance 4. All responsive vendors were evaluated on all six evaluation criteria. 5. To evaluate substantial conformity to specifications the team unanimously agreed to take the following approach: a. Review the legislatively mandated specifications for all responsive vendors. b. Vendors who were deemed substantially conforming to statutory requirements to be further evaluated for all RFP specifications to ideally reach a group agreement and further rank the vendors. RtAD-Consensus Meeting_11192018 11202018 Page 1 of 31 nda Summary c. Those vendors that were not substantially conforming to statutory requirements were ranked lower by the team for this evaluation criteria. following ranking was used for each specification a. ?Yes? implies conforms to specifications. b. ?No? implies does not conform to specifications. c. ?MayBe? implies that the team is unsure about conformity. Discussion during the consensus meeting is summarized below. The voting members were issued three colored cards Green to show compliance to specification, Pink to show that the specification was not complied with and Yellow to show Maybe there was compliance. In the case of Maybe responses, further clarifications may occur during negotiation prior to Best and Final Offer (BAFO) submission and Award. Negotiation questions matter for Vendors in the competitive range that are selected for further consideration. The voting members discussed each mandatory requirement in full and arrived at a consensus by showing the appropriate cards. Outcomes from consensus meeting for the various specifications are provided in a separate spreadsheet for each bidder with appropriate and weaknesses. The proposals were taken up in an alphabetical order for ranking. 1. Substantial Conformity to Specifications Review of Legislated Specifications Amplify Education Inc.: Business Specification 1: Describe how the proposed solution directly assesses reading and pre-reading behaviors to support student?s learning development at the various grade levels to inform instruction, including any observation-based practices if applicable: a. oral language (expressive and receptive) b. phonological and phonemic awareness c. phonics d. vocabulary e. fluency f. comprehension Consensus Ranking: The voting members were unanimous in their agreement that Amplify complied with this specification. Two of the voting members mentioned that while online versions are available for students with appropriate self-regulation and computer skills; teachers continue to have the option to directly assess/observe students. The voting members were 11-0 Yes on Amplify?s ability to comply with this specification. Business Specification 3: Describe the validity and reliability of the assessment in the following areas: a. oral language b. phonological and phonemic awareness c. phonics d. vocabulary e. fluency f. comprehension RtAD-Consensus Meeting_11192018 11202018 Page 2 of 31 Exhibit Team Consensus Ranking after the Cunsensus Meeting on 11-19-:a13 and 11v2?-1?13 ef Read to Achieve RFP Review 1. 1. DukedSp-ed?ca?uns 1-. Vendor Eastman-ma! Sherwin-f Refetms Rank 5. vm mar-mummy Phase lfahlabed a on Rank a Vendor: Rank Rad: mum Wekmd am Rank 0 W?ar? Hall: Wekmd ?ed: Rank 0 Rank mmhr. (unimlurn lane-cite! hm am (103 am 0.Ema-Hun nit-era 3 Weih'l: 1.3'Jbsien1uJ 1-: 5-: li-L'Iui'ian 2. EME- Elsi-red Spectra-1415311 CunmL-?Elemnsimliun - Esp-ms?: de? 5 Fran-use! 5.51.1111!? Mean-LE Final-:35] Exhibit F Roy Cooper Governor Eric Boyette Secretary of Information Technology State Chief Information Officer Contract Award Recommendation To: Andrea Pacyna Deputy Chief IT Procurement Officer Department of Information Technology From: Tymica Dunn Procurement Chief Department of Public Instruction Date: June 7, 2019 Subject: Contract Award Recommendation Read to Achieve Diagnostics - Requisition # - RQ20680730, DIT File #300042 Reference #: Request for Negotiations 40-RQ20680730A, DIT File #300042 Enclosed for your review and approval is the award recommendation for Requisition # RQ20680730. Bids received pursuant to RFN #40-RQ20680730A have been reviewed and an Evaluation Committee hereby requests the Statewide IT Procurement Office to award the contract, as follows: Description: Recommended Vendor: Read to Achieve Diagnostics – Software as a Service Imagination Station Inc., dba, Istation 1 of 14 v.2017-06-06 Cost: $8,405,820 for 3 years Contract Term: Two (2) years plus 1 (one) year optional renewals at the discretion of the State Project Name and Number: Read to Achieve Diagnostics - 2018 DIT file # 300042 Thank you for your assistance. If additional information is required, please do not hesitate to contact me. cc: Evaluation Committee Patti Bowers, DSCIO Glenn Poplawski, DSCIO Kathy Bromead, PMA Table of Contents Section 1: Introduction ............................................................................................................................... 4 Section 2: Evaluation Committee ............................................................................................................... 4 Section 3: Evaluation Criteria / Methodology ............................................................................................ 5 Section 4: Timeline...................................................................................................................................... 6 Section 5: Evaluation of Bid Submission ..................................................................................................... 6 Section 6: Vendors .................................................................................................................................... 7 A. Evaluation Criteria ................................................................................................................ 7 B. Cost ........................................................................................................................... 7 C. Vendor Financial Stability ........................................................................................ 9 D. Formative and Diagnostic Assessment ..................................................................... 9 E. Personalized Learning ............................................................................................. 11 2 of 14 v.2017-06-06 Section 7: Finalist Vendor(s) ................................................................................................................... 13 Section 8: Award Recommendation ......................................................................................................... 14 Section 9: Supporting Documentation ..................................................................................................... 14 3 of 14 v.2017-06-06 Section 1: Introduction The North Carolina Department of Public Instruction posted Request for Proposal number 40-RQ20680730A to the North Carolina Interactive Purchasing System on September 6, 2018. A total of four (4) bids were received; however, the evaluation committee could not reach a consensus and deemed it most advantageous to the State to cancel and negotiate with sources of supply. NCDPI requested and received approval from the DIT DSCIO/Chief Procurement Officer to negotiate. Request for Negotiations were sent to Amplify and !station on March 28, 2019 and negotiation meetings were conducted on April 11, 2019 with both vendors at North Carolina Department of Public Instruction. DSCIO/Chief Procurement Officer The purpose of this award recommendation and the resulting contract award is to identify a vendor best qualified to offer services for Read to Achieve Diagnostic Software as a Service solution (RtAD) to meet NCDPl's obligations under state law, N.C.G.S. llSC83.1, et. seq. North Carolina state law requires kindergarten through third grade students to be assessed with valid, reliable, formative and diagnostic reading assessments. NCDPI is obligated to adopt and provide these developmentally appropriate assessments. The solution must assess student progress, diagnose difficulties, inform instruction and remediation, and yield data that can be used with the Education Value-Added Assessment System (EVAAS). Section 2: Evaluation Committee Name Title/Agency Participation Level Berry, Erika Craver, Nathan Senior Policy Advisor, NCDPI Digital Teaching and Learning Consultant, NCDPI Decision Maker Decision Maker Karkee, Thakur Shue, Pam AIHour, Julien Psychometrician, NCDPI Deputy Superintendent of Early Education, NCDPI Decision Maker Decision Maker SME Dunn, Tymica Director - Architecture, Integration, & Quality Assurance, NCDPI Purchasing Section Chief, NCDPI 4 of 14 Procurement Officer v.2017-06-06 Gossage, Chloe Strong, Melissa Chief Strategy Officer, NCDPI State Board of Education Attorney SME SME Project Manager, NCDPI Project Manager Decision Maker: Key business stakeholders evaluating the bid responses. Voting Project Manager: Overall responsibility includes successful initiation, planning, design, execution, implementation, and closure of a project. Non-Voting Subject Matter Expert (SME) Person who is an authority in a particular technical area pertaining to the procurement Non-Voting Viswanathan, Srirekha Role Definitions: Section 3: Evaluation Criteria / Methodology The selection process was conducted using the "best value" methodology authorized by N.C.G.S. §§143-135.9 and 143B-1350(h). The evaluation committee met as a group and evaluated the responsive proposals. The evaluation criteria listed below is in the order of importance: Evaluation Criteria Cost Vendor Financial Stability Formative and Diagnostic Assessment Personalized Learning 5 of 14 v.2017-06-06 Section 4: Timeline Date Milestone March 21, 2019 RFP Cancellation Notifications sent to vendors, Request to Negotiate Review Period March 27, 2019 RFP proposals were extended to June 29, 2019 - Clarification 1 April 11, 2019 Negotiation Meeting with vendors April 17, 2019 Clarification issued to vendors - Clarification 2 April 23, 2019 Clarification response received and shared with evaluation team April 25, 2019 Evaluation Committee meeting and discussion of proposal strengths and weakness May 3, 2019 Clarification issued to vendor - Clarification 3 May 15, 2019 Clarification issued to vendor - Clarification 4 June 4, 2019 Best and Final Offer (BAFO) June 6, 2019 Award Recommendation Clarification response received and shared with evaluation team Clarification response received and shared with evaluation team Section 5: Evaluation of Bid Submission Proposal response from the following two vendors were considered for further negotiations: 1. 2. mplify Education Inc. Imagination Station dba, !station 55 Washington Street, Suite 800, Brooklyn, NY 11201 8150 North Central Expressway, Suite 2000, Dallas, TX 75206 6 of 14 v.2017-06-06 Section 6: Vendors Listed below is a synopsis of each proposal submitted based on the criteria defined in Section 3. A. Evaluation Criteria "Best Value" procurement method authorized by N.C.G.S. §§143-135.9 and 143B-1350(h) has been used for this evaluation. A one step source selection was used. The proposals were objectively evaluated using the evaluation criteria described below. The evaluation team members did their due diligence and issued clarifications for each proposal before meeting the vendors on April 11 , 2019. Strengths and weaknesses were discussed during the evaluation meeting on April 25,2019. The following evaluation criteria was used to determine strengths and weakness 1. Cost 2. Vendor Financial Stability 3. Formative and Diagnostic Assessment 4. Personalized Learning B. Cost The strengths and weaknesses identified by the Evaluation team for the responsive vendors are summarized in the tables below. Vendor Amplify Strengths Cost 1. No strengths noted. Weakness Amplify submitted two cost offers - one for assessment only at $4,312,210 (Year 1), $3,895,210 (Year 2), $3,883,760 (Year 3) totaling $12,102,096.08 another 7 of 14 v.2017-06-06 2. 3. 4. 5. IstatIion 1. Istation submitted two cost offers one for the assessment component only and one for the both the assessment and curriculum components. The cost for the assessment was $2,751,940 (Year 1) $2,751,940 (Year 2) $2,751,940 (Year 3) totaling $8,255,820. For both the assessment and curriculum was $9,934,813 (Year 1), $9,934,813 (Year 2), $9,934813 (Year 3) totaling $29,804,438. 2. The assessment cost of $5.70 per student is less expensive than Amplify and includes more features such as 3,000 teacher directed lessons, remote student and parent access to Istation's iPractice. one for personalized and blended approach to learning at $11,948,912.75 (Year 1), $10,934,412.75 (Year 2) and $10,922,962.75 (Year 3) totaling $33,806,288.25. The assessment only cost which was considered for this proposal review is significantly higher than Istation’s assessment only tool. The assessment cost of $8.00 per student is higher than that of Istation and does not include online assessments nor remote student or parent access. This cost does not include teacher lessons. The assessment is not automated and requires teacher intervention by reading the tests aloud and takes away significant classroom time from teaching. Professional Development cost for year 1 is 556,650; however, is limited to training Master Literacy Trainers and NCDPI Consultants. The proposal response did not adequately include strategies for ensuring consistent scoring to evaluate training effectiveness. 1. Solution is not compatible with screen readers or keyboards and will cost extra to ensure compatibility. 8 of 14 v.2017-06-06 3. $76,103 for professional development offers 22 onsite trainings, 14 recorded live webinars and 10 virtual teacher trainings annually, in addition the vendor will provide up to 5 additional onsite and 10 recorded webinars annual at no additional cost. 4. Vendor will provide additional professional development beyond these allowances at a rate of $5,800.00 per day of professional development and $550.00 per webinar. 5. The cost for Professional Development also covers the logistics which includes securing learning facilities, paying the cost to host the training, coordinating training dates, communication to participants etc. C. Vendor Financial Stability The strengths and weaknesses identified by the Evaluation team for the responsive vendors are summarized in the tables below. Vendor Amplify !station Vendor Financial Stability Strengths NCDPI Financial Director finds no going concern. NCDPI Financial Director finds no going concern. None Weakness None D. Formative and Diagnostic Assessment The strengths and weaknesses identified by the Evaluation team for the responsive vendors are summarized in the tables below. 9 of 14 v.2017-06-06 Amplify Vendor Formative and Diagnostic Assessment Strengths 1. Assessment covers all five areas of early literacy which is mandated by law. The service has the capability to appropriately assess K-3 students. 2. Amplify Service has enough item pool for 20 assessments (i.e., number of items that are aligned to NC standards which will be enough for 20 tests). It is also to be noted that Schools have three tests per grade level for this age group. 3. The reports are easily understandable. Weakness 1. Benchmarking and progress monitoring per student per grade level consumes a lot of time and requires excessive teacher involvement to manually administer and enter test results. The fixed form manual test takes more time testing to find where the students are at. This takes away significant instructional time. 2. The $8 option is not adaptive i.e., it does not measure student's exact level of achievement. It was difficult to gauge from the proposal response how the service adapts when students gain mastery. 3. The fixed form tests don't always provide feedback on the student's exact level of achievement which brings to question the effectiveness of the data driven instructional support. Home Connect Letters for parents is clear. There are multiple reports for teachers about instruction and areas that need intervention. !station 1. Adaptive assessment (also known as Computer Adaptive Assessment) allows students to reach their full potential. This assessment measures student's mastery with the minimal amount of teacher time. 2. The aggregate reports for teachers are easy to read and interpret. 3. !station has enough item pool for 10 None assessments (i.e., number of items that are aligned to NC standards which will be 10 of 14 v.2017-06-06 enough for 10 tests). It is also to be noted that Schools have three tests per grade level for this age group. E. Personalized Learning The strengths and weaknesses identified by the Evaluation team for the responsive vendors are summarized in the tables below. Amplify Vendor 1. Personalized Learning Strengths Personalized Learning was only offered in the Alternate Cost proposal which came with increased pricing. 2. Progress Monitoring when a student is identified as at risk for achievement, is at individual skills level. 3. Amplify offers a dyslexia component. 11 of 14 Weakness 1. The basic cost proposal offered does not have all aspects of personalized learning and is not computer adaptive. 2. Progress Monitoring for students at risk requiring intervention takes up a lot of time for teachers. The basic assessment solution option is not computer based it is takes away significant instruction time from teachers and the reliability and validity of results vary significantly. 3. Home Reading is not included in the bid offering. This limits the ability for students to have access to resources outside of school which limits their learning and the participation from parents. v.2017-06-06 Istation 1. The assessment is computer adaptive and caters 1. Although Istation stated that their assessment can be to the individual student's need. used to screen for dyslexia, the vendor does not have a 2. The time for assessment offered by Istation is 40 separate dyslexia component at this time. minutes/student and is fully online (i.e., teacher can work with other students in class while a group of students are taking the assessment) . Amplify’s assessment is 45 minutes/student on the low end and requires teachers to spend time with the students while they are being assessed. The reduced assessment time and the fact that the teacher does not have to be with students who are being assessed (using the computerized model) allows teachers more time to support student’s individual needs. 3. Istation allows students see their own academic need and take responsibility for their learning by providing feedback after each subtest. This feedback is available to students, parents and teachers. Further students are allowed access outside of school. They can personalize their learning by choosing games and activities to further enhance their learning. 12 of 14 v.2017-06-06 Section 7: Finalist Vendor(s) NCDPI entered into negotiations with both vendors. Each vendor was given the opportunity to present their assessment solution and how it would best meet the needs of the department. Clarification 1 was issued to both vendors extending their RFP bid submission as the proposal response was used in the negotiation process. Clarification 2 was issued to both vendors prior to the negotiation meeting. The question provided in this request were focal points during the meeting. This clarification request also gave the Evaluation Team some guidance and understanding with both vendor offering. After the negotiation meeting held on April 11, 2019 the team unanimously agree to continue further negotiation efforts with Istation. Clarification 3 Istation was asked by NCDPI to provide the cost of both the assessment and curriculum. This request was to compare the Alternative Cost proposal 2 submitted by Amplify which included the curriculum portion. After reviewing Istation’s submission the team agreed to go with only the assessment portion which is required in legislation. While there was in interest in the curriculum offering it is not required in the law. Clarification 4 was issued to negotiation on the Terms of Use and Privacy policy that Istation has in place. NCDPI’s legal team negotiated the language that was provided by Istation. Istation was in agreement and signed the clarification giving the department permission to incorporate in in the final contract offering. While Amplify was able to submit an offer to satisfy the agencies needs it was not cost effective. As the incumbent the progress made by students in reading is not significant. The effectiveness of the data driven instructional support is questionable. The current test scores does not support the inflated cost offered by Amplify. Istation provided a solution that was robust, cost effective, offered additional enhancements that were required, and met the business needs of NCDPI. While Istation’s dyslexia component may be missing key measures, the service substantially conforms to the requirements specified under N.C.G.S. 115C-83.1, which is the primary obligation of this procurement. Negotiations were issued to Istation and memorialized in the BAFO # 40-20680730A dated June 4, 2019 in which Istation agreed to the following change in specifications: ADA Compliance high contrast reports, Voice Recognition Software, Onsite Training and 13 of 14 v.2017-06-06 Recorded Webinars, Growth Calculation, Summer Reading Camps, Customizations and Enhancements, BAFO Cost, as well as modifications to the Istation Terms of Use and Privacy Policy which comprise the License grant and agreement for the State’s use of the Istation Resources. IStation also completed the Vendor Security Assessment Guide (VRAR) that was reviewed and approved by NCDPI and DIT technical teams. Section 8: Award Recommendation The Evaluation Committee has determined that Istation’s bid substantially conforms to the specifications and requirements of the law and therefore, recommends award RFP No. 40-RQ20680730A to Imagination Station Inc. (Istation) in the amount of $8,405,820 (Year 1 - $2,751,940, Year 2 - $2,751,940, Year 3 - $2,751,940) for 2 years with the option of one (1) additional one (1) year renewals. Section 9: Supporting Documentation The following supporting documents that reflect the vendor selection are included: 1. 2. 3. 4. Bid Response Clarification documents – Signed BAFO document Hosting Exception and Privacy and Threshold Analysis (approved by DIT) 14 of 14 v.2017-06-06 Exhibit G On a scale of 1-5 stars, with a 1 being not very helpful, and a 5 being very helpful, rate how helpful you feel Istation's ISIP assessment will be for you and your campus: What is a favorite take-away from today's workshop? Please indicate why you made this rating. Thank you! Response What district are you with? Open-Ended Response Please explain why you chose your selection: Once we get familiar with the assessment, I think it will be great. It is the steep learning curve in the transition process with tight window to receive training the 5 week before teachers report back to work. There is a national team of folks supporting iStation and the company has rolled out state implementation. The numerous resources in the TOOLBOX to support teachers. The training was engaging where new information was presented in small sections and then time to process with iStation coach to answer questions and address implementation questions. Our district was represented by 13 teacher and 2 district office staff left the training with our questions addressed and feeling VERY comfortable about the product. We just need time to now play in the system to practice and become Lincoln County Schools familiar with the product. Sarah Cude and her team was very energetic and passionate about ISIP and 5 her enthusiasm is contagious :) I think the Teacher Resources will be wellreceived :) Great presentation. Gave takeaways to 5 get us started. Teacher Resources 5 the model Charlotte Mecklenburg Schools 5 The resources in the teacher toolkit Charlotte Mecklenburg Schools Gaston County Schools I think this will be an upgrade from Mclass knowledgable, with a good understanding 5 of the varied experiences in the room. good reminder of the resources available. 5 Lots of resources for teachers. 5 5 Open-Ended Response Charter School MGSD Learning about all of the components. It is a lot of information to take in at one time. MGSD Toolbox Thank you Sarah for all of your energy. You made the 4 hours fly by. Casa Esperanza Montessori Charter School An overview of the program. I just have to get in there and use it for real to feel comfortable with it. I think once we get through a few months, we will all feel Cleveland County better about it. Lots of helpful information but a lot of new I valued the positive presenters and 5 information at one time. understanding a bit more the subtests. Charlotte-Mecklenburg Examples, time to generate questions and examples of the subtest- skills assessed reflect on subtests then discuss with and the resources to support and adjust 5 Istation coach instruction Thank you for your support guidance and help ease the transition of to Istation. Kannapolis City School 5 Very candid and helpful better understanding great workshop Kannapolis City Schools Seems like a good, reliable assessment with good reports to help inform 5 instruction. How the assessments work. Union County It was very informative and helpful in 5 learning this new assessment tool. learning the new Istation assessment tool Lincoln County Schools I really can see the benefits in using this. One of the biggest struggles is making use of common language across staff, 5 parents, and students. I would like more detailed PD on how to read more in detail The ability for there to be a Parent Portal!!! reporting. Lincoln Charter School 5 great data info teacher resources and data reports Lincoln County I feel it will be very informative and I love that lessons are easily found in the 5 teacher resources. Love the intervention lessons to help my students grow!!! I believe this will be a very informative piece to help me be a better teacher. Lincoln County It has a huge database of resources and many different reports to aid in determining what students need to focus 5 on. The demo and looking at the assessment and data. 5 Lots of good information. Union County Public Schools 5 the detail look into the assessments Charter It was a lot of information. I need to navigate through it and may have questions later. Istation seems very user friendly and I am excited that my teaching staff will be able 5 to spend more time in their classroom. the recording of ORF. Sugar Creek Charter School Gaston Co Charter School 5 I would prefer more guided instruction to walk me through the importance of action-taking after formative reports, versus having to self-discover what they all mean and assessment - continuous data cycle tell me. Queen's Grant Community School I am looking at change in a positive light, 5 we are here to help the children. Teacher Toolbox Lincoln County 5 all of the knowledge and hopefully the easy of transition from mClass UCPS 5 Understanding what Istation is. Avery 5 More time given back to teacher's to instruct!!! And I am so glad I brought my jacket....the room is freezing!! Gaston It will be a big improvement to be able to have students tested at once rather than 5 one on one. Computer scoring for student read aloud. NA The specialist (Julie) was very helpful. Engaging and informative presenters. Thank you! Charlotte-Mecklenburg Schools Valid testing will provide formative data to better instruct. This way teachers can 5 focus more on instruction than assessing. ORF!! Any new program needs repeated facilitating. Gaston County Schools Presenters have used Istation in their own Enthusiasm of how to use the Program in 5 classrooms. the best way to drive instuction Using reports, resources Mountain Island Day Very clear with small group and hands on Very clear with small group and hands on 5 time time Stanly County Schools 5 coaching Charlotte Mecklenburg Schools 5 The data and resources available. Mountain Island Day Community Charter I think the ISIP will help with not losing 5 instructional time and help with fidelity. The video training and the view cards teachers can do and pass. 5 Great information in a hands on format. Union County Public Schools 5 - Training was very helpful due to not 5 knowing much about Istation. There's a lot of resources available for teachers to help students grow. Union County Public Schools 5 Teaching time freed up for students due to shorter time needed for testing. great presenters and support throughout the training session Union This will be a big time saver for teachers and allow them to focus more time on 5 instruction. Teacher Resources Iredell-Statesville The reports will be helpful and the site is easy to follow. There are many resources which will be great when organizing All of the resources and the top two things 5 groups and monitoring student growth, needed when training teachers. Very excited about IStation Concord Lake STEAM Academy It is a lot of information,but I feel that there are many resources I can use if I need help moving forward. Movement Charter School I feel that it will be helpful to see very 5 targeted data for each student. I enjoyed learning about the Teacher toolbox and other resources available to us. I feel that I have a good initial understanding of the overall program and will learn more as I get familiar with iStation. Gaston County Schools 5 This will provide very granular data that teachers can review. The intervention resources will provide a great tool for our intervention and progress monitoring. I was able to get answers to questions I had. Today provided a great deal of clarification to move forward. Mountain Island Charter School It is very similar to MClass so a lot of the information is the same. This made it easier to learn. 5 Pacing and overall climate of information The intervention resources I can go back and get teachers started 5 with ISIP. Teacher Resources Iredell-Statesville Schools 5 Orf assessment A.c.e academy charter 5 Graet training with great resources. Most of it is adaptive and can be tailored to Because I did learn a lot and feel more prepared to support student needs. teachers with Istation. Charlotte Mecklenburg Schools The presentation of the 3 segments was well planned and presented. The information provided will help us get started. The istation website was easy to maneuver. Support and resources have 5 been provided. Support and resources are available. Movement Charter Union County 5 great information Thanks so much! 5 Resources Gaston County Union County Public Schools The presenters are very knowledgable 5 and helpful. I really like the tier 1, 2, and 3 activities for each skill Cleveland County Schools 5 I like how Istation is quick. I love the listening and speaking portion of the assessment. This will help our English learners. Charlotte Mecklenburg Schools Everything was very detail and thought 5 out. The steps to use to get my teachers started Ace academy The training was broken down in sessions 5 so we could go back and train teachers. The teacher resources. Stanly County 5 I think this assessment is great ACE Academy Charter 5 Teacher Resources and recorded oral reading. Thank you! Gaston County 5 The amount of resources available This seems like a very useful resource. Gaston County 5 I love models to practice and demonstrate how to use the program. Stanly County This will help our teachers adjust 5 instruction for their students. I was concerned about the oral reading fluency but feel better that there have been measures taken to make sure all students I have a more positive outlook towards implementing this are monitored accurately. program. Lake Norman Charter 5 The positive energy of the presenters. Hickory Public Schools 5 Lots of resources Wilkes Once we are comfortable with all the features and are able to change the mindset surrounding the way assessing The cape! I am also so glad that you all has been done for the past several years, shared some ideas for getting teachers 5 it will be great. and students excited about Istation. I know that I will need to spend much more time exploring myself before I will be able to share with teachers. Wilkes County Schools 5 Looks to be a very thorough program! The teacher toolbox The scale of 1-5 is visible as a U with an 5 S. My opinion is a 5 Practice in small groups not just whole group modeling. I learned many important details today that I did not learn with our Spanish implementation. Union 5 Student intervention history Great ideas for teachers! Bertie 5 The reports are very comprehensive.. data tracking, limiting assessment time, clarifying the ORF and how that will work to put teachers at ease, hearing from 5 teachers as our trainers. reports, teacher resources, and intervention tracking - tied for first! Caldwell County Schools Richmond County this was very helpful. It was a lot of information but I feel once I can get into the site and play around more hands on, I will be ready to train our teachers and get them comfortable. Thank you. Union Academy Presenters were well informed and although all details were not quite worked out...this is what normally happens so it 5 was just par for the course. Online resources. I learn best by doing...PowerSchool was down so I was unable to practice with the online ancillaries. Guilford County Schools 5 Session was very informative The program will allow more instructional time for teachers. All needed questions were addressed. Vance County 5 options. we love options Once I have a chance to think, I may have some questions. Rutherford County teacher led lessons and tools I do feel better about this after this training. I was not excited about another new program. Guilford County Schools 5 Over view of Istation I would like to have a complete presentation to share with teachers. Rutherford County 5 Easy, intuitive program for teachers and students. 5 Swain The visuals and help button will be a huge Being able to see what it's about really 5 asset. help ease my anxiety. It learned a lot and it was really helpful. Montgomery They answered all our questions and 5 eased or fears. They were very explicit in explaining directions. Montgomery County That you can get priority reports. There are lots of resources for teachers, I station will give teachers more time to I think the information on student Tiers will 5 each instead of assessing. be helpful with the MTSS process. The resources given and the presentation Recording and self grading fluency 5 will be very helpful. assessment. Montgomery County I feel excited to share with teachers how useful and efficient these tests will be to benefit teachers and students. Montgomery County So many questions answered. Anxiety 5 about the change decreased greatly. Padlet to deliver district PD. Montgomery County I think the ISIP assessment will help us address interventions and how to group 5 students. I love both of the reports that were shared today. The summary and priority report will help us provide better instruction for I learned a lot and all my questions about the program were our students. answered. Montgomery County I was very concerned that our students would be sitting in front of a screen completing lessons and that there would be less teacher directed instruction. I am This will be another great tool to help us relieved that NC will utilize IStation for analyze individual student data in order to assessments/progress monitoring and 5 provide instruction to meet those needs. teacher directed instruction only. 5 The amount of data is great! I feel much more at ease because of the way the information was presented today and that we will have access to podcasts and webinars all along the way. The simplicity of the platform and the amount of data. Sorry, that's two takeaways. Montgomery County Chatham County Schools I came in WANTING to love this, and I'm excited about it's launch. I had a lot of regarding this online program coming in, but I'm ready and informed! Trainers were knowledgeable and 5 respectful. The Oral Reading Fluency Assessment! I'm excited about this! 5 The people :) ABSS Provides comprehensive information related to the screening of foundational 5 reading skills Positive energy and the padlet resources Craven I think it's going to be a great tool that will allow for more teaching time while still getting the important information that 5 leads instruction. adaptive testing!! love the fact that It seems like an amazing program overall! Looking forward to readings are recorded and takes away the being able to play around with the program and see what's it's subjectivity from scoring all about. Durham Public Schools 5 A better understanding of how teachers can use Istation as a formative assessment tool Warren County Schools Durham Public Schools I think the presenters did a great job introducing the material. They seemed limited by specifics that our state will need to figure out quickly and handled the hot There is a support network that seems 5 seat amazingly well with professionalism. very accessible I see a great deal of promise with the program that I didn't initially - I see less room for teacher error and more transparency with many aspects of the assessment. Durham Public Schools Exhibit H Passed 2017-18 3rd Grade Reading EOG Passed EOG 68,397 Promoted from 3rd Grade 68,397 18,901 Retained Total 2017-18 3rd Grade Reading Passed Good Cause Alternative Exemption Total Promoted 17,991 13,198 99,586 90% 80% 70% 118,487 Total Spring membership (de 121,567 (Same Data in % of 3rd Graders) 60% 50% Passed 2017-18 3rd Grade Reading EOG Passed EOG Promoted from 3rd Grade 56% 56% Retained 16% Passed Good Cause Alternative Exemption Total Promoted 15% 11% 82% 40% 30% 20% 10% Passed EOG Third Grade Reading Proficiency Trends 100% ■ Passed EOG 90% Retained Promoted from 3rd Grade ■ Passed Alternative ■ Good cause Exemption 80% 70% 2017-18 3rd Grade Reading 60% 50% 120,000 40% 30% 100,000 20% 10% 80,000 2015 2014 2016 2017 ■ Students Who Entered 3rd Grade Reading at 3rd Grade level ■ Students Who Completed 3rd Grade Reading at 3rd Grade level 60,000 40,000 Based on BOG and EOG Reading Scores - Percent Proficient 20,000 0 Passed EOG Early Grade Reading Proficiency ■ Pa.ssed EOG 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Kindergarten 1st ■ 2015 ■ 2016 ■ 2017 2nd Promoted from 3rd Grade ■ Passed Alternative Retained ■ Good Cause Exemption Exhibit I REQUEST FOR BEST AND FINAL OFFER NO. 40-IT00107-16 / BAFO 3 STATE OF NORTH CAROLINA DEPARTMENT OF PUBLIC INSTRUCTION (NCDPI) Offers will be received until: July 21, 2016 Issue Date: July 19, 2016 Refer ALL inquiries regarding this BAFO to: Commodity Number: 920 Mike Beaver Michael.beaver@dpi.nc.gov 919-807-3683 Description: Read to Achieve Diagnostic Software as a Service Using Agency: Department of Public Instruction See page 2 for mailing instructions. Requisition No.: RQ19195897 NOTICE TO VENDOR Offers, subject to the conditions made a part hereof, will be received at this office, 301 N. Wilmington Street, Raleigh, NC 27601 until 3PM ET on the day of opening and then opened, for furnishing and delivering the goods and services as described herein. Refer to page 2 for proper mailing instructions. Bids submitted via facsimile (fax) machine in response to this Best and Final Offer (BAFO 3) will not be accepted. Bids are subject to rejection unless submitted on this form. EXECUTION In compliance with this Request for Best and Final Offer (BAFO 3), and subject to all the conditions herein, the undersigned offers and agrees to furnish any or all goods and services which are offered, at the prices agreed upon and within the time specified herein. Pursuant to N.C.G.S. §143B-1354 and under penalty of perjury, the undersigned Vendor certifies that this offer has not been arrived at collusively or otherwise in violation of Federal or North Carolina law and this offer is made without prior understanding, agreement, or connection with any firm, corporation, or person submitting an offer for the same commodity, and is in all respects fair and without collusion of fraud. Failure to execute/si n offer rior to submittal shall render offer invalid. Late offers are not acce table. OFFEROR: STREET ADDRESS: P.O. BOX: ZIP: TELEPHONE NUMBER: TOLL FREE TEL. NO 212-796-2200 FAX NUMBER: Education Inc. DATE: Jul 20,2016 E-MAIL: 11201 800-823-1969 212-796-2311 bids@am li .com ACCEPTANCE OF OFFER If the State accepts any or all parts of this offer, an authorized representative of the Department of Public Instruction shall affix his/her signature to the Vendor's response to this Request for BAFO. The acceptance shall include the response to this BAFO, BAFO 1 and BAFO 2;, any provisions and requirements of the original RFP which have not been superseded by this BAFO, BAFO 1, and BAFO 2; and the Department of Information Technology Terms and Conditions. These documents shall then constitute the written agreement between the parties. A copy of this acceptance will be forwarded to the successful vendor(s). FOR NC DEPARTMENT OF PUBLIC INSTRUCTION USE ONLY Offer accepted and contract awarded this_ day of _______� 20_, as indicated on attached cerlification, by (Authorized representative of NC Deparlment of Public Instruction) State Superintendent of Public Instruction by _________________ (Authorized representative of NC Deparlment of Public Instruction) Chief Financial Officer BAFO Number: 40-IT00107-16 / BAFO 3 Instructions: Sealed offers, subject to the conditions made a part hereof, will be received at the address below, for furnishing and delivering the goods, software, and/or services as described herein. DELIVER TO: RFP 40-IT00107-16 BAFO 3 Department of Public Instruction Attn: Mike Beaver 301 N. Wilmington Street, Room B04 Raleigh, NC 27601 It is the responsibility of the Vendor to deliver the offer in this office by the specified time and date of opening, regardless of the method of delivery. Address envelope and include RFP number as shown above. Vendors are cautioned that offers sent via U.S. Mail, including Express, Certified, Priority, Overnight, etc., may not be delivered in time to meet the deadline. Deliver one (1) signed original executed offer, one (1) copy of the executed offer response, and one (1) signed, executed electronic copy of its offer on a USB Flash Drive or read-only CD/DVD(s). The files must not be password-protected and must be capable of being copied to other media. Offers submitted via facsimile (FAX) machine, telephone or electronically in response to this will not be accepted. SOLICITATION REQUEST FOR BEST AND FINAL OFFER (BAFO) This request is to acquire a best and final offer from Vendor for Read to Achieve Diagnostic Software as a Service. The offer should integrate the previous response to the RFP, BAFO 1, and BAFO 2 and any changes listed below. Any individual vendor can receive a different number of requests for BAFOs that other offerors. NOTE: This bid is still in the evaluation period. During this period and prior to award, possession of the BAFO, original bid response and accompanying information is limited to personnel of the Department of Information Technology (DIT) IT Strategic Sourcing Office, and to agencies responsible for participating in the evaluation. Bidders who attempt to gain this privileged information, or to influence the evaluation process (i.e. assist in evaluation) will be in violation of purchasing rules and their offer will not be further evaluated or considered. 1. The State offers the following with regards to Section VII of the RFP: Negotiated Terms and Conditions The Parties agree that Section VII. of the RFP, Department of Information Technology Terms and Conditions, is replaced in its entirety with the following negotiated terms and conditions and these terms replace any proposed Vendor Terms and Conditions: VII. Department of Information Technology Terms and Conditions 1. DEFINITIONS: “Agency” means the agency identified in this solicitation as the Purchasing Agency or Award Authority. a) Page 2 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 “Data” means recorded information, regardless of form or method of recording. “User” means user(s) from the Agency along with all current and future North Carolina Local Education Agencies (LEAs), Charter Schools, Regional Schools, Educational Services for the Deaf and Blind Schools (ESDB), and other public schools as the need may arise (individually or collectively). d) “Services” means the duties and tasks undertaken by the Vendor to fulfill the requirements and specifications of this solicitation, including access to and use of the Vendor’s Web-base Software Application. e) “State” shall mean the State of North Carolina, the Department of Information Technology as an agency, or the Agency. f) “Support” includes updates, upgrades, maintenance and support Services conducted by the Vendor on its own Web-based Software, and consulting, training and other support Services provided by or through Vendor for the State. g) “Web-based Software Application” shall mean the Vendor software applications residing or provided on the Vendor's system and accessed and used by authorized State and Agency End Users through a web browser and shall include Vendor-hosted storage, databases, related documentation, and other functionalities or Services provided by Vendor to facilitate the use of the Vendor’s Services. b) c) 2. ACCESS AND USE OF SERVICES: Vendor grants the State a personal non-transferable and non-exclusive right to use and access, during the term(s) of this Agreement and paid subscription, for the number of Users agreed or as may be agreed, all Services furnished or accessible under this Agreement. The State may access and use the Services as agreed herein. The State is authorized to access Agency Data and Vendor provided data as specified herein and to transmit revisions, updates, deletions, enhancements, or modifications to the State Data. This shall include the right of the State to, and access to, Support from Vendor without the Vendor requiring a separate maintenance or support agreement. The State may use the Services with any computer, computer system, server, or desktop workstation owned or utilized by the State; provided that the State acknowledges that devices recommended by the Vendor are set forth in D.2.1, Table A: Business Specifications of Vendor’s response to the RFP. User access to the Services shall be routinely provided by the Vendor or as may be further specified in a Service Level Agreement (SLA) agreed to in writing by the parties. The State shall notify the Vendor immediately of any unauthorized use of any password or account, or any other known or suspected breach of security access. The State also agrees to refrain from taking any steps, such as reverse engineering, reverse assembly or reverse compilation to derive a source code equivalent to Services or any portion thereof. Use of the Services to perform services for commercial third parties (so-called “service bureau” uses) is not permitted, but the State may utilize the Services to perform its usual and governmental functions. If the Service fees are based upon the number of Users and/or hosted instances, the number of Users/hosted instances available may be adjusted at any time by mutual agreement and State Procurement approval. All Services and information designated as “confidential” or “proprietary” shall be kept in confidence except as may be required by the North Carolina Public Records Act: NCGS § 132-1, et seq. b) The State’s license for the Services neither transfers, vests, nor infers any title or other ownership right in any intellectual property rights of the Vendor or any third party, nor does this license transfer, vest, or infer any title or other ownership right in any source code associated with the Services unless otherwise agreed to by the parties. The provisions of this section will not be construed as a sale of any ownership rights in the Services unless, and only to the extent, Custom Services are being developed as a Work for Hire in response to the State’s solicitation documents. Any Services or technical and business information owned by Vendor or its suppliers or licensors made accessible or furnished to the State shall be and remain the property of the Vendor or other party, respectively. Vendor has a limited, non-exclusive license to access and use the a) Page 3 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 State Data as provided to Vendor, but solely for performing its obligations under this Agreement and in confidence as provided herein. c) Vendor or its suppliers shall at minimum, and except as otherwise agreed, provide telephone assistance to the State for all Services provided hereunder during the State’s normal business hours unless otherwise specified herein. Vendor warrants that its support and customer service and assistance will be performed in accordance with generally accepted industry standards. The State has the right to receive the benefit of upgrades, updates, maintenance releases or other enhancements or modifications made generally available to Vendor’s customers. Vendor’s right to a new use agreement for new version releases of the Services shall not be abridged by the foregoing. Vendor may, at no additional charge, modify the Services to improve operation and reliability or to meet legal requirements. d) Successful access by the State’s users to the Vendor’s Services is required. The Agency shall notify the Vendor in writing within ten days that the Agency’s Users cannot successfully access the implemented Services. Final acceptance is expressly conditioned upon such successful access and other requirements as specified herein. e) Services provided pursuant to this Solicitation may, in some circumstances, be accompanied by a clickwrap agreement. The term clickwrap agreement refers to an agreement that requires the end user to manifest his or her assent to terms and conditions by clicking an “ok” or “agree” button on a dialog box or pop-up window as part of the process of access to the Services. All terms and conditions of any clickwrap agreement provided with any Services solicited herein shall have no force and effect and shall be non-binding on the State, its employees, agents, and other authorized users of the Services. f) The Vendor may utilize partners and/or subcontractors to assist in the provision of the Services, so long as the State Data is not removed from the United States and the security provisions referenced herein are complied with. The Vendor shall identify all of its strategic business partners related to Services provided under this contract, including but not limited to, all subcontractors or other entities or individuals who may be a party to a joint venture or similar agreement with the Vendor, who will be involved in any application development and/or operations. g) Vendor warrants that all Services will be performed with professional care and skill, in a workmanlike manner and in accordance with industry standards. An SLA or other agreed writing shall contain provisions for scalability of Services and any variation in fees or costs as a result of any such scaling. h) Professional services provided by the Vendor at the request by the State in writing in addition to agreed Services shall be at the then-existing Vendor hourly rates when provided, unless otherwise agreed in writing by the parties. 3. WARRANTY OF NON-INFRINGEMENT; REMEDIES. a) RESERVED 4. ACCESS AVAILABILITY; REMEDIES: a) The Vendor warrants that the Services will be in good working order, and operating in conformance with Vendor’s standard specifications and functions as well as any other specifications agreed to by the parties in writing, and shall remain accessible 24/7, with the exception of scheduled outages for maintenance and of other service level provisions agreed to in the SLA. Vendor does not warrant that the operation of the Services will be completely uninterrupted or error free, or that the Services functions will meet all the State’s requirements, unless developed as Customized Services. b) The State shall notify the Vendor if the Services are not in good working order or inaccessible during the term of the Agreement. Vendor shall, at its option, either repair, replace or reperform any Services reported or discovered as not being in good working order and accessible during the applicable contract term Page 4 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 without cost to the State. If the Services monthly availability averages less than the agreed upon percentages (excluding agreed-upon maintenance downtime) set forth in the SLA for three (3) or more months in a rolling twelve-month period, the State may terminate the Agreement for material breach in accordance with the Default provisions herein below. 5. EXCLUSIONS: a) Except as stated in Paragraph 4, Vendor and its parent, subsidiaries and affiliates, subcontractors and suppliers make no warranties, express or implied, as to the Services, and specifically disclaim warranties of merchantability or fitness for a particular purpose as provided by N.C.G.S. §§25-2-316, 25-2-313 and 25-2315; and as may be amended. b) The warranties provided in Paragraph 4 do not cover repair for damages, malfunctions or service failures caused by: i) Actions of non-Vendor personnel; ii) Failure to follow Vendor’s operation or other written instructions relating to the Services provided to the State; iii) Force Majeure conditions set forth hereinbelow; iv) The State’s sole misuse of, or its own inability to us, the Services. 6. PERFORMANCE REVIEW AND ACCOUNTABILITY: N.C.G.S. § 143B-1318(f) and 09 NCAC 06B.1207 require provisions for performance review and accountability in State IT contracts. For this procurement, these shall include: a) The availability percentages, response time metrics, credits, and other applicable service level provisions set forth in the SLA; and b) The project management specifications set forth in Attachment A to the RFP and Attachment B, Table D to the RFP, as such specifications are amended by BAFO 1. For clarity and notwithstanding the proposed Project Timeline and Project Plan set forth in Attachment A to the RFP, the Parties acknowledge that the final timeline and project schedule shall be agreed between the parties promptly upon execution of this Agreement. The Services herein will be provided consistent with and under the foregoing Services performance guarantees. 7. INTELLECTUAL PROPERTY INDEMNITY a) The Vendor, at its own expense, shall defend any action brought against the State to the extent that such action is based upon a claim that the Services supplied by the Vendor, their use or operation infringes on a patent, copyright, trademark or violates a trade secret in the United States. The Vendor shall pay those costs and damages finally awarded or agreed in settlement against the State in any such action; In any event damages shall be limited as provided in N.C.G.S. 143B-1350(h1). Such defense and payment shall be conditioned on the following: i) That the Vendor shall be notified within a reasonable time in writing by the State of any such claim; and, ii) That the Vendor shall have the sole control of the defense of any action on such claim and all negotiations for its settlement or compromise provided, however, that the State shall have the option to participate in such action at its own expense. b) If any Services supplied by Vendor become the subject of a claim of infringement of a patent, copyright, Trademark or a trade secret in the United States, the Vendor, shall at Page 5 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 its option and expense, either procure for the State the right to continue using the Services, or to replace or modify the same to become noninfringing. If neither of these options can reasonably be taken in Vendor's judgment, or if further use shall be prevented by injunction, the Vendor agrees to cease provision of any affected Services, and refund any sums the State has paid Vendor for Services and the State shall then have the option of terminating the Agreement, or applicable portions thereof, without penalty or termination charge. c) Vendor will not be required to defend or indemnify the State if any claim by a third party against the State for infringement or misappropriation results from the State's alteration of any Vendor-branded Services, or from the continued use of the Services after receiving notice they infringe on an intellectual property right of a third party. 8) LIMITATION OF LIABILITY a) For purposes of the exclusive remedies and limitations of liability set forth in this Paragraph, Vendor shall be deemed to include the Vendor and its employees, agents, representatives, subcontractors, and suppliers and damages shall be deemed to refer collectively to all injuries, damages, losses, liabilities, expenses or costs incurred. b) The Vendor’s liability for damages to the State arising under the Agreement shall be limited to the value of the Agreement during the then current term. In any event, damages shall be further limited as provided in N.C.G.S. 143B-1350(h1). c) The foregoing limitation of liability shall not apply to claims covered by other specific provisions including but not limited to Service Level Agreement or Warranty compliance, or to claims for injury to persons or damage to tangible personal property caused by Vendor’s gross negligence or willful or wanton conduct. This limitation of liability does not apply to contributions among joint tortfeasors under N.C.G.S. 1B-1 et seq., the receipt of court costs or attorney’s fees that might be awarded by a court in addition to damages after litigation based on this Contract. d) For delays in the first provision of the Services, Vendor shall have no liability unless the agreed provision date date is delayed by more than thirty (30) days by causes not attributable either to the State or to Force Majeure conditions, in which case the State shall have the right, as its remedies: i) To recover direct costs attributable to Vendor's delay, and ii) To cancel the order without incurring cancellation charges. Vendor shall have no liability unless the default in delivery of Services is occasioned by causes not attributable either to the State or to Force Majeure conditions 9. SUPPORT AND MAINTENANCE: Vendor will provide to the State the same Support Services for updating, maintaining and continuing optimal performance for the Services as provided to other similarly situated users of the Services, but minimally as provided for and specified herein. Unless otherwise agreed in writing, Support will also be provided for any other (e.g., third–party) software provided by the Vendor in connection with the Vendor’s solution herein. The technical and professional activities required for establishing, managing, and maintaining the Services environment are the responsibilities of the Vendor. Any training specified in this Agreement will be provided by the Vendor to certain State users for the fees or costs as set forth in this Agreement or in an SLA. 10. MODIFICATION OF SERVICES: If Vendor modifies or replaces the Services provided to the State and other customers, and if the State has paid all applicable fees, the State shall be entitled to receive, at no additional charge, access to a newer version of the Services that supports substantially the same functionality as the then accessible version of the Services. Newer versions of the Services containing substantially increased functionality may be made available to the State for an additional subscription fee. In the event of Page 6 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 either of such modifications, the then accessible version of the Services shall remain fully available to the State until the newer version is provided to the State and accepted. If a modification materially affects the functionality of the Services as used by the State, the State, at its sole option, may defer such modification. 11. TRANSITION PERIOD: a) For ninety (90) days, either prior to the expiration date of this Agreement, or upon notice of termination of this Agreement, Vendor shall assist the State, upon written request, in extracting and/or transitioning all Data in the format determined by the State (“Transition Period”). b) The Transition Period may be modified in an SLA or as agreed upon in writing by the parties in a contract amendment. c) During the Transition Period, Services access shall continue to be made available to the State without alteration. d) Vendor agrees to compensate the State for damages or losses the State incurs as a result of Vendor’s failure to comply with this Transition Period section in accordance with the Exclusive Remedies and Limitation of Liability provisions above. e) Upon termination, and unless otherwise stated in an SLA, and after providing the State Data to the State as indicated above in this section with acknowledged receipt by the State in writing, the Vendor shall permanently destroy or render inaccessible any portion of the State Data in Vendor’s and/or subcontractor’s possession or control following the completion and expiration of all obligations in this section. Within thirty (30) days, Vendor shall issue a written statement to the State confirming the destruction or inaccessibility of the State’s Data. f) The State at its option, may purchase additional Transition services as may be agreed upon in a supplemental agreement. 12. TRANSPORTATION: Transportation charges for any Deliverable sent to the State other than electronically or by download, shall be FOB Destination unless delivered by internet or file-transfer as agreed by the State, or otherwise specified in the solicitation document or purchase order. 13. TRAVEL EXPENSES: All travel expenses should be included in the Vendor’s proposed costs. Separately stated travel expenses will not be reimbursed. In the event that the Vendor may be eligible to be reimbursed for travel expenses specifically agreed to in writing and arising under the performance of this Agreement, reimbursement will be at the out-of-state rates set forth in G.S. §138-6; as amended from time to time. Vendor agrees to use the lowest available airfare not requiring a weekend stay and to use the lowest available rate for rental vehicles. All Vendor incurred travel expenses shall be billed on a monthly basis, shall be supported by receipt and shall be paid by the State within thirty (30) days after invoice approval. Travel expenses exceeding the foregoing rates shall not be paid by the State. The State will reimburse travel allowances only for days on which the Vendor is required to be in North Carolina performing Services under this Agreement. 14. PROHIBITION AGAINST CONTINGENT FEES AND GRATUITIES: Vendor warrants that it has not paid, and agrees not to pay, any bonus, commission, fee, or gratuity to any employee or official of the State for the purpose of obtaining any contract or award issued by the State. Subsequent discovery by the State of non-compliance with these provisions shall constitute sufficient cause for immediate termination of all outstanding Agreements. Violations of this provision may result in debarment of the Vendor(s) or Vendor(s) as permitted by 9 NCAC 06B.1030, or other provision of law. 15. AVAILABILITY OF FUNDS: Any and all payments by the State are expressly contingent upon and subject to the appropriation, allocation and availability of funds to the Agency for the purposes set forth in Page 7 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 this Agreement. If this Agreement or any Purchase Order issued hereunder is funded in whole or in part by federal funds, the Agency’s performance and payment shall be subject to and contingent upon the continuing availability of said federal funds for the purposes of the Agreement or Purchase Order. If the term of this Agreement extends into fiscal years subsequent to that in which it is approved such continuation of the Agreement is expressly contingent upon the appropriation, allocation, and availability of funds by the N.C. Legislature for the purposes set forth in the Agreement. If funds to effect payment are not available, the Agency will provide written notification to Vendor. If the Agreement is terminated under this paragraph, Vendor agrees to terminate any Services provided to the Agency under this Agreement and relieve the Agency of any further obligation thereof. The State shall remit payment for Services accepted prior to the date of the aforesaid notice in conformance with the payment terms. 16. PAYMENT TERMS: Payment terms are Net 30 days after receipt of correct invoice or acceptance of the Services, whichever is later. The Purchasing State Agency is responsible for all payments under the Agreement. No additional charges to the Agency will be permitted based upon, or arising from, the Agency’s use of a Business Procurement Card. The State may exercise any and all rights of Set Off as permitted in Chapter 105A-1 et seq. of the N.C. General Statutes and applicable Administrative Rules. a) Upon Vendor’s written request of not less than 30 days and approval by the State, the State may: (1) Forward the Vendor’s payment check(s) directly to any person or entity designated by the Vendor, or (2) Include any person or entity designated in writing by Vendor as a joint payee on the Vendor’s payment check(s), however (3) In no event shall such approval and action obligate the State to anyone other than the Vendor and the Vendor shall remain responsible for fulfillment of all Agreement obligations. b) For any third party software licensed by Vendor or its subcontractors for use by the State, a copy of the software license including terms acceptable to the State, an assignment acceptable to the State, and documentation of license fees paid by the Vendor shall be provided to the State before any related license fees or costs may be billed to the State. c) An undisputed invoice is an invoice for which the State and/or Agency has not disputed the invoice in writing sent to the Vendor on the grounds of an invoice error within thirty (30) days from the invoice date, unless the State and/or Agency requests more time for review of the invoice. Upon Vendor’s receipt of notice that the State and/or Purchasing State Agency disputes an invoice, Vendor will work to correct the applicable invoice error, provided that such dispute notice shall not relieve the State or Agency from its payment obligations for the undisputed items on the invoice or for any disputed items that are ultimately corrected. The Agency is not required to pay the Vendor for any goods and/or services provided without a written purchase order from the appropriate Agency. In addition, all Services provided must meet all terms, conditions, and specifications of the Agreement and purchase order and be accepted as satisfactory by the Agency before payment will be issued. 17. ACCEPTANCE CRITERIA: Acceptance testing is required for all Vendor Services unless provided otherwise in the solicitation documents. The State may define such processes and procedures as may be necessary or proper, in its opinion and discretion, to ensure compliance with the State’s specifications and Vendor’s technical representations. In the event acceptance of Services is not described in additional contract documents, the State shall have the obligation to notify Vendor, in writing and within ten (10) days following provision of any Service provided by Vendor if it is not acceptable. The notice shall specify in reasonable detail the reason(s) the Service is unacceptable. Acceptance by the State shall not be unreasonably withheld; but may be conditioned or delayed as required for testing of Services. If the Service is not acceptable to the Page 8 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 State, the State may terminate this Agreement for convenience as provided in Paragraph 30 c) ii) without liability for any costs or expenses associated with the Service. 18. CONFIDENTIALITY: The State may maintain the confidentiality of certain types of information described in N.C.G.S. §132-1 et seq. Such information may include trade secrets defined by N.C.G.S. §66152 and other information exempted from the Public Records Act pursuant to N.C.G.S. §132-1.2. Vendor may designate information, Services, or appropriate portions of its response as confidential, consistent with and to the extent permitted under the Statutes and Rules set forth above, by marking the top and bottom of pages containing confidential information with a legend in boldface type “CONFIDENTIAL.” By so marking any page, the Vendor warrants that it has formed a good faith opinion, having received such necessary or proper review by counsel and other knowledgeable advisors that the portions marked confidential meet the requirements of the Rules and Statutes set forth above. However, under no circumstances shall price information be designated as confidential. The State agrees to promptly notify the Vendor in writing of any action seeking to compel the disclosure of Vendor’s confidential information. If an action is brought pursuant to N.C.G.S. §132-9 to compel the State to disclose information marked confidential, the Vendor agrees that it will intervene in the action through its counsel and participate in defending the State, including any public official(s) or public employee(s). The Vendor agrees that it shall hold the State and any official(s) and individual(s) harmless from any and all damages, costs, and attorneys’ fees awarded against the State in the action. The State shall have the right, at its option and expense, to participate in the defense of the action through its counsel. The State shall have no liability to Vendor with respect to the disclosure of Vendor’s confidential information ordered by a court of competent jurisdiction pursuant to N.C.G.S. §132-9 or other applicable law. 19. SECURITY OF STATE DATA: a) All materials, including software, Data, information and documentation provided by the State to the Vendor during the performance or provision of Services pursuant to this Agreement is the property of the State of North Carolina and must be kept secure and returned to the State. The Vendor will protect State Data in its hands from unauthorized disclosure, loss, damage, destruction by natural event, or other eventuality. Proprietary Vendor materials shall be identified to the State by Vendor prior to use or provision of Services hereunder and shall remain the property of the Vendor. Derivative works of any Vendor proprietary materials prepared or created during the performance of provision of Services hereunder shall remain the property of the Vendor and shall be provided to the State as part of the Services. b) The Vendor shall protect the confidentiality of all information, Data, instruments, studies, reports, records and other materials provided to it by the State or maintained or created in accordance with this Agreement. No such information, Data, instruments, studies, reports, records and other materials in the possession of Vendor shall be disclosed in any form without the prior written consent of the State. The Vendor will have written policies governing access to and duplication and dissemination of all such information, Data, instruments, studies, reports, records and other materials. c) Protection of Personal Identifying Information. Vendor acknowledges its responsibility for securing personal identifying information collected by the State and stored in any Vendor site or other Vendor housing systems, including but not limited to computer systems, networks, servers, or databases, maintained by Vendor or its agents or subcontractors in connection with the Services. Vendor warrants, at its sole cost and expense, that it shall implement processes and maintain security of personal identifying information; provide reasonable care and efforts to detect fraudulent activity involving personal identifying information; and promptly notify the Agency of any breaches of security involving personal identifying information. Page 9 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 d) Family Educational Rights and Privacy Act: Student educational records are subject to 20 U.S.C. 1232g, Family Educational Rights and Privacy Act (FERPA). The Vendor shall ensure that the Vendor and Vendor’s Services fully complies with FERPA and every employee responsible for carrying out the terms of this contract is aware of the confidentiality requirements of federal law. In addition, every such employee must sign a confidentiality acknowledgement that indicates that he or she understands the legal requirements for confidentiality. The Vendor is responsible for the actions of its employee and must take all precautions necessary to ensure that no violations occur. Finally, access to personally identifiable student education information shall be limited to those employees who must have access to it in order to perform their responsibilities pursuant to this Agreement. e) Vendor will provide and maintain secure backup of the State Data. Vendor shall implement and maintain secure passwords for its online system providing the Services, as well as all appropriate administrative, physical, technical and procedural safeguards at all times during the term of this Agreement to secure such Data from Data Breach, protect the Data and the Services from loss, corruption, unauthorized disclosure, and from hacks, introduction of viruses, disabling devices, malware and other forms of malicious or inadvertent acts that can disrupt the State’s access to its Data and the Services. If requested by the State, the Vendor will allow periodic back-up of State Data by the State to the State’s infrastructure as the State requires or as may be provided by law. f) Vendor shall certify to the State: i) The sufficiency of its security standards, tools, technologies and procedures in providing Services under this Agreement; ii) That the Services will comply with the following: (1) Any DIT security policy regarding Cloud Computing, and the DIT Statewide Information Security Policy Manual; (2) Privacy provisions of the Federal Privacy Act of 1974; (3) The North Carolina Identity Theft Protection Act, N.C.G.S. Chapter 75, Article 2A (e.g., N.C.G.S. § 75-65 and -66); (4) The North Carolina Public Records Act, N.C.G.S. Chapter 132; and (5) Applicable industry standards and guidelines, including but not limited to, relevant security provisions of the Payment Card Industry (PCI) Data Security Standard (PCIDSS) including the PCIDSS Cloud Computing Guidelines. (6) Any requirements implemented by the State under N.C.G.S. §§ 143B-1341 and -1342. g) Security Breach. “Security Breach” under the NC Identity Theft Protection Act (N.C.G.S. § 75-60ff) means (1) any circumstance pursuant to which applicable Law requires notification of such breach to be given to affected parties or other activity in response to such circumstance (e.g., N.C.G.S. § 7565); or (2) any actual, attempted, suspected, threatened, or reasonably foreseeable circumstance that compromises, or could reasonably be expected to compromise, either Physical Security or Systems Security (as such terms are defined below) in a fashion that either does or could reasonably be expected to permit unauthorized Processing (as defined below), use, disclosure or acquisition of or access to any the State Data or state confidential information. “Physical Security” means physical security at any site or other location housing systems maintained by Vendor or its agents or subcontractors in connection with the Services. “Systems Security” means security of computer, electronic or telecommunications systems of any variety (including data bases, hardware, software, storage, switching and interconnection devices and mechanisms), and networks of which such systems are a part or communicate with, used directly or indirectly by Vendor or its agents or subcontractors in connection with the Services. “Processing” means any operation or set of operations performed upon the State Data or State confidential information, whether or not by automatic means, such as creating, collecting, procuring, obtaining, accessing, recording, organizing, storing, adapting, altering, retrieving, consulting, using, disclosing or destroying. Page 10 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 h) Breach Notification. In the event Vendor becomes aware of any Security Breach due to Vendor acts or omissions other than in accordance with the terms of this Agreement, Vendor shall, at its own expense, (1) immediately notify the State’s Contract Administrator of such Security Breach and perform a root cause analysis thereon, (2) investigate such Security Breach, (3) provide a remediation plan, acceptable to the State, to address the Security Breach and prevent any further incidents, (4) conduct a forensic investigation to determine what systems, data and information have been affected by such event; and (5) cooperate with the State, and any law enforcement or regulatory officials, credit reporting companies, and credit card associations investigating such Security Breach. The State shall make the final decision on notifying the State’s persons, entities, employees, service providers and/or the general public of such Security Breach, and the implementation of the remediation plan. If a notification to a customer is required under any Law or pursuant to any of the State’s privacy or security policies, then notifications to all persons and entities who are affected by the same event (as reasonably determined by the State) shall be considered legally required. i) Notification Related Costs. Vendor shall reimburse the State for all Notification Related Costs incurred by the State arising out of or in connection with any such Security Breach due to Vendor acts or omissions other than in accordance with the terms of this Agreement resulting in a requirement for legally required notifications. “Notification Related Costs” shall include the State’s internal and external costs associated with addressing and responding to the Security Breach, including but not limited to: (1) preparation and mailing or other transmission of legally required notifications; (2) preparation and mailing or other transmission of such other communications to customers, agents or others as the State deems reasonably appropriate; (3) establishment of a call center or other communications procedures in response to such Security Breach (e.g., customer service FAQs, talking points and training); (4) public relations and other similar crisis management services; (5) legal and accounting fees and expenses associated with the State’s investigation of and response to such event; and (6) costs for credit reporting services that are associated with legally required notifications or are advisable, in the State’s opinion, under the circumstances. In the event that Vendor becomes aware of any Security Breach which is not due to Vendor acts or omissions other than in accordance with the terms of this Agreement, Vendor shall immediately notify the State of such Security Breach, and the parties shall reasonably cooperate regarding which of the foregoing or other activities may be appropriate under the circumstances, including any applicable Charges for the same. j) Vendor shall allow the State reasonable access to Services security logs, latency statistics, and other related Services security data that affect this Agreement and the State’s Data, at no cost to the State. k) In the course of normal operations, it may become necessary for Vendor to copy or move Data to another storage destination on its online system, and delete the Data found in the original location. In any such event, the Vendor shall preserve and maintain the content and integrity of the Data, except by prior written notice to, and prior written approval by, the State. l) Remote access to Data from outside the continental United States, including, without limitation, remote access to Data by authorized Services support staff in identified support centers, is prohibited unless approved in advance by the State Chief Information Officer or the Using Agency. m) In the event of temporary loss of access to Services, Vendor shall promptly restore continuity of Services, restore Data in accordance with this Agreement and as may be set forth in an SLA, restore accessibility of Data and the Services to meet the performance requirements stated herein or in an SLA. As a result, Service Level remedies will become available to the State as provided herein, in the SLA or other agreed and relevant documents. Failure to promptly remedy any such temporary loss of access may result in the State exercising its options for assessing damages under this Agreement. n) In the event of disaster or catastrophic failure that results in significant State Data loss or extended loss of access to Data or Services, Vendor shall notify the State by the fastest means available and Page 11 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 also in writing, with additional notification provided to the State Chief Information Officer or designee of the contracting agency. Vendor shall provide such notification within twenty-four (24) hours after Vendor reasonably believes there has been such a disaster or catastrophic failure. In the notification, Vendor shall inform the State of: 1) The scale and quantity of the State Data loss; 2) What Vendor has done or will do to recover the State Data from backups and mitigate any deleterious effect of the State Data and Services loss; and 3) What corrective action Vendor has taken or will take to prevent future State Data and Services loss. 4) If Vendor fails to respond immediately and remedy the failure, the State may exercise its options for assessing damages or other remedies under this Agreement. Vendor shall conduct an investigation of the disaster or catastrophic failure and shall share the report of the investigation with the State. The State and/or its authorized agents shall have the right to lead (if required by law) or participate in the investigation. Vendor shall cooperate fully with the State, its agents and law enforcement. o) In the event of termination of this contract, cessation of business by the Vendor or other event preventing Vendor from continuing to provide the Services, Vendor shall not withhold the State Data or any other State confidential information or refuse for any reason, to promptly return to the State the State Data and any other State confidential information (including copies thereof) if requested to do so on such media as reasonably requested by the State, even if the State is then or is alleged to be in breach of the Agreement. As a part of Vendor’s obligation to provide the State Data pursuant to this Paragraph 19) o), Vendor will also provide the State any data maps, documentation, software, or other materials necessary, including, without limitation, handwritten notes, materials, working papers or documentation, for the State to use, translate, interpret, extract and convert the State Data. p) The State requires cloud service providers (CSPs) to provide, on an annual basis, a security/risk assessment report. CSPs will utilize a Third-Party Assessment Agency/Organization to perform the assessment of the cloud service provider’s security controls to determine the extent to which security controls are implemented correctly, operate as intended, and is in compliance with the Statewide Information Security Manual. Assessments will be conducted using industry best practice certification, e.g. SOC 2 Type 2 or NIST Risk Management Framework (RMF). Assessment reports provided, must be with 1 year of completion date in order to satisfy this requirement. Re-assessments will occur annually or when a significant change to the system’s security posture is made. The State will ensure that supported agency’s security staff is available for consultation during these processes. Identified gaps between required Security Control Baselines and the contractor's implementation as documented in the Security Assessment Report shall be tracked by the Vendor for mitigation in a Plan of Action and Milestones (POA&M) document. Vendor will remediate within an agreed-upon timeline any material weaknesses in the Vendor’s security controls identified in such report that are identified as a reason for the auditor to issue such report as “qualified” or “adverse”, and Vendor will use commercially reasonable efforts to remediate any other material weaknesses identified in such report. 20. ACCESS TO PERSONS AND RECORDS: Pursuant to N.C. General Statute 147-64.7, the State, the State Auditor, appropriate federal officials, and their respective authorized employees or agents are authorized to examine all books, records, and accounts of the Vendor insofar as they relate to transactions with any department, board, officer, commission, institution, or other agency of the State of North Carolina pursuant to the performance of this Agreement or to costs charged to this Agreement. The Vendor shall retain any such books, records, and accounts for a minimum of three (3) years after the completion of this Agreement. Additional audit or reporting requirements may be required by the State, if in the State’s Page 12 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 opinion, such requirement is imposed by federal or state law or regulation. 21. ASSIGNMENT: Vendor may not assign this Agreement or its obligations hereunder except as permitted by 09 NCAC 06B.1003 and this Paragraph. A consolidation, acquisition, or merger that does not include an assignment of this Agreement or its obligations hereunder (for purposes of this Agreement, a “Permitted Transaction”) is permitted. Vendor shall provide reasonable notice of not less than thirty (30) days after any Permitted Transaction that occurs more than ninety (90) days prior to the expiration of this Agreement. Vendor shall provide reasonable notice promptly after any Permitted Transaction that occurs less than ninety (90) days prior to the expiration of this Agreement. Any assignee shall affirm this Agreement attorning to the terms and conditions agreed, and that Vendor shall affirm that the assignee is fully capable of performing all obligations of Vendor under this Agreement. An assignment may be made, if at all, in writing by the Vendor, Assignee and the State setting forth the foregoing obligation of Vendor and Assignee. 22. NOTICES: Any notices required under this Agreement should be delivered to the Contract Administrator for each party. Unless otherwise specified in the Solicitation Documents, any notices shall be delivered in writing by U.S. Mail, Commercial Courier, facsimile or by hand. 23. TITLES AND HEADINGS: Titles and Headings in this Agreement are used for convenience only and do not define, limit or proscribe the language of terms identified by such Titles and Headings. 24. AMENDMENT: This Agreement may not be amended orally or by performance. Any amendment must be made in written form and signed by duly authorized representatives of the State and Vendor. 25. TAXES: The State of North Carolina is exempt from Federal excise taxes and no payment will be made for any personal property taxes levied on the Vendor or for any taxes levied on employee wages. Agencies of the State may have additional exemptions or exclusions for federal or state taxes. Evidence of such additional exemptions or exclusions may be provided to Vendor by Agencies, as applicable, during the term of this Agreement. Applicable State or local sales taxes shall be invoiced as a separate item. 26. GOVERNING LAWS, JURISDICTION, AND VENUE: This Agreement is made under and shall be governed and construed in accordance with the laws of the State of North Carolina. The place of this Agreement or purchase order, its situs and forum, shall be Wake County, North Carolina, where all matters, whether sounding in contract or in tort, relating to its validity, construction, interpretation and enforcement shall be determined. Vendor agrees and submits, solely for matters relating to this Agreement, to the jurisdiction of the courts of the State of North Carolina, and stipulates that Wake County shall be the proper venue for all matters. 27. DEFAULT: a) In the event any Service provided by the Vendor fails to conform to any material requirement of the specifications (other than the performance guarantees described in Paragraph 4 and Paragraph 6, which the Parties acknowledge are subject to the remedies set forth in Paragraph 4, Paragraph 6, and the SLA), notice of the failure is provided by the State and the failure is not cured within ten (10) days, or Vendor fails to meet the SLA requirements, the State may terminate as set forth in Section 30. Default may be cause for debarment as provided in 09 NCAC 06B.1206. The rights and remedies of the State provided above shall not be exclusive and are in addition to any other rights and remedies provided by law or under the Agreement. b) If Vendor fails to deliver Services within the times required by this Agreement’s SLA, the State may Page 13 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 provide written notice of said failure to Vendor, and by such notice require payment of a penalty. N.C.G.S. § 143B-1340(f); 09 NCAC 06B.1207. c) Should the State fail to perform any of its obligations upon which Vendor’s performance is conditioned, Vendor shall not be in default for any delay, cost increase or other consequences due to the State’s failure. Vendor will use reasonable efforts to mitigate delays, costs or expenses arising from assumptions in the Vendor’s bid documents that prove erroneous or are otherwise invalid. Any deadline that is affected by any such failure in assumptions or performance by the State shall be extended by an amount of time reasonably necessary to compensate for the effect of such failure. 28. FORCE MAJEURE: Neither party shall be deemed to be in default of its obligations hereunder if and so long as it is prevented from performing such obligations as a result of events beyond its reasonable control, including without limitation, fire, power failures, any act of war, hostile foreign action, nuclear explosion, riot, strikes or failures or refusals to perform under subcontracts, civil insurrection, earthquake, hurricane, tornado, or other catastrophic natural event or act of God. 29. COMPLIANCE WITH LAWS: The Vendor shall comply with all laws, ordinances, codes, rules, regulations, and licensing requirements that are applicable to the conduct of its business, including those of federal, state, and local agencies having jurisdiction and/or authority. 30. TERMINATION: Any notice or termination made under this Agreement shall be transmitted via US Mail, Certified Return Receipt Requested. The period of notice for termination shall begin on the day the return receipt is signed and dated. a) The parties may mutually terminate this Agreement by written agreement at any time. b) The State may terminate this Contract, in whole or in part, pursuant to Paragraph 27), or pursuant to the Special Terms and Conditions in the Solicitation Documents, if any, or for any of the following i) Termination for Cause: In the event the Services furnished by the Vendor during performance fails to conform to any material specification or requirement of the Agreement (other than the performance guarantees described in Paragraph 4 and Paragraph 6, which the Parties acknowledge are subject to the remedies set forth in Paragraph, Paragraph 6, and the SLA), and the failure is not cured within the specified time after providing written notice thereof to Vendor, the State may cancel and procure the articles or Services from other sources; holding Vendor liable for 1) refunding any pre-paid fees (on a pro-rated basis) for the remaining term of the Contract, 2) refunding any fees paid for the Contract period during which such failure occurred, and 3) performing the Transition services set forth in Paragraph 11 at no cost to the State. The rights and remedies of the State provided above shall not be exclusive and are in addition to any other rights and remedies provided by law or under the Agreement. Vendor shall not be relieved of liability to the State for damages sustained by the State arising from Vendor’s breach of this Agreement; and the State may, in its discretion, withhold any payment due as a setoff until such time as the damages are finally determined or as agreed by the parties. Voluntary or involuntary Bankruptcy or receivership by Vendor shall be cause for termination. ii) Termination for Convenience Without Cause: The State may terminate service and indefinite quantity contracts, in whole or in part by giving thirty (30) days prior notice in writing to the Vendor. Vendor shall be entitled to sums due as compensation for Services provided and performed in conformance with the Agreement. In the event the Contract is terminated for the convenience of the State the Agency will pay for all Services and work provided and performed in conformance with the Agreement up to the date of termination. 31. DISPUTE RESOLUTION: The parties agree that it is in their mutual interest to resolve disputes Page 14 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 informally. A claim by the State shall be submitted in writing to the Vendor’s Contract Administrator for decision. The parties shall negotiate in good faith and use all reasonable efforts to resolve such dispute(s). During the time the parties are attempting to resolve any dispute, each shall proceed diligently to perform their respective duties and responsibilities under this Agreement. If a dispute cannot be resolved between the parties within thirty (30) days after delivery of notice, either party may elect to exercise any other remedies available under this Agreement, or at law. This term shall not constitute an agreement by either party to mediate or arbitrate any dispute. 32. SEVERABILITY: In the event that a court of competent jurisdiction holds that a provision or requirement of this Agreement violates any applicable law, each such provision or requirement shall be enforced only to the extent it is not in violation of law or is not otherwise unenforceable and all other provisions and requirements of this Agreement shall remain in full force and effect. All promises, requirement, terms, conditions, provisions, representations, guarantees and warranties contained herein shall survive the expiration or termination date unless specifically provided otherwise herein, or unless superseded by applicable federal or State statute, including statutes of repose or limitation. 33. INDEPENDENT CONTRACTORS. Vendor and its employees, officers and executives, and subcontractors, if any, shall be independent Vendors and not employees or agents of the State. This Agreement shall not operate as a joint venture, partnership, trust, agency or any other business relationship. 34. FEDERAL INTELLECTUAL PROPERTY BANKRUPTCY PROTECTION ACT: The Parties agree that the Agency shall be entitled to all rights and benefits of the Federal Intellectual Property Bankruptcy Protection Act, Public Law 100-506, codified at 11 U.S.C. 365(n), and any amendments thereto. 35. ELECTRONIC PROCUREMENT: (Applies to all contracts that include E-Procurement and are identified as such in the body of the solicitation document): Purchasing shall be conducted through the Statewide E-Procurement Service. The State’s third party agent shall serve as the Supplier Manager for this E-Procurement Service. The Vendor shall register for the Statewide E-Procurement Service within two (2) business days of notification of award in order to receive an electronic purchase order resulting from award of this contract. Subparagraphs a) and b) may apply only to any goods procured herein, but not to Services. a) Reserved. b) Reserved. c) The Supplier Manager will capture the order from the State approved user, including the shipping and payment information, and submit the order in accordance with the E-Procurement Service. Subsequently, the Supplier Manager will send those orders to the appropriate Vendor on State Contract. The State or State approved user, not the Supplier Manager, shall be responsible for the solicitation, bids received, evaluation of bids received, award of contract, and the payment for goods delivered. d) Vendor agrees at all times to maintain the confidentiality of its user name and password for the Statewide E-Procurement Services. If a Vendor is a corporation, partnership or other legal entity, then the Vendor may authorize its employees to use its password. Vendor shall be responsible for all activity and all charges for such employees. Vendor agrees not to permit a third party to use the Statewide EProcurement Services through its account. If there is a breach of security through the Vendor’s account, Vendor shall immediately change its password and notify the Supplier Manager of the security breach by e-mail. Vendor shall cooperate with the state and the Supplier Manager to mitigate and correct any security breach. Page 15 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 2. The State offers the following with regards to the Vendor’s Customer Terms and Conditions attached to Vendor’s response to this RFP: The Parties agree that the Vendor’s Customer Terms and Conditions, attached to Vendor’s response to this RFP, is deleted in its entirety and replaced with the following: CUSTOMER TERMS & CONDITIONS 1. Scope. This Customer Terms and Conditions document (“Agreement”) is between Amplify Education, Inc. (“Amplify”) and the North Carolina Department of Public Instruction (“Customer”). It is an attachment to BAFO Number 1 to RFP No. 40-IT00107-16 (the “RFP”) and is governed by the terms and conditions agreed by the parties pursuant to the RFP, as modified by BAFO Number 1. Amplify will deliver one or more of the following products or services, as specified in the RFP: assessment software and content, digital and print curriculum and instructional materials, hardware devices, and professional development materials and services (collectively, the “Products”). 2. License. Subject to the terms and conditions of the RFP and this Agreement, Amplify grants to Customer a non-exclusive, non-transferable, non-sublicenseable license to access and use, and permit Authorized Users to access and use the Products solely in the U.S. during the Term for the number of Authorized Users agreed between Amplify and Customer pursuant to the RFP and for whom Customer has paid the applicable fees to Amplify. “Authorized User” means an individual teacher or other personnel employed by Customer or End Users (as defined in the RFP), or an individual student registered for instruction at Customer’s or an End User’s school, whom Customer permits to access and use the Products subject to the terms and conditions of the RFP and this Agreement, and solely while such individual is so employed or so registered. Each Authorized User’s access and use of the Products shall be subject to the terms and conditions of the RFP, in addition to the terms and conditions of this Agreement, and violations of such terms may result in suspension or termination of the applicable account. 3. Restrictions. Customer shall access and use the Products solely for non-commercial instructional and administrative purposes of Customer’s school. Further, Customer shall not, except as expressly authorized or directed by Amplify: (a) copy, modify, translate, distribute, disclose or create derivative works based on the contents of, or sell, the Products, or any part thereof; (b) decompile, disassemble or otherwise reverse engineer the Products or otherwise use the Products to develop functionally similar products or services; (c) modify, alter or delete any of the copyright, trademark, or other proprietary notices in or on the Products; (d) rent, lease or lend the Products or use the Products for the benefit of any third party; (e) avoid, circumvent or disable any security or digital rights management device, procedure, protocol or mechanism in the Products; or (f) permit any Authorized User or third party to do any of the foregoing. Customer also agrees that any works created in violation of this section are derivative works, and, as such, Customer agrees to assign, and hereby assigns, all right, title and interest therein to Amplify. The Products and derivatives thereof may be subject to export laws and regulations of the U.S. and other jurisdictions. Customer may not export any Product outside of the U.S. Further, Customer will not permit Authorized Users to access or use any Product in a U.S.- embargoed country or otherwise in violation of any U.S. export law or regulation. The software and associated documentation portions of the Products are “commercial items” (as defined at 48 CFR 2.101), comprising “commercial computer software” and “commercial computer software documentation,” as those terms are used in 48 CFR 12.212. Accordingly, if Customer is the U.S. Government or its contractor, Customer will receive only those rights set forth in this Agreement in Page 16 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 accordance with 48 CFR 227.7201-227.7204 (for Department of Defense and their contractors) or 48 CFR 12.212 (for other U.S. Government licensees and their contractors). 4. Reservation of Rights. SUBSCRIPTION PRODUCTS ARE LICENSED, NOT SOLD. Subject to the limited rights expressly granted hereunder, all rights, title and interest in and to all Products, including all related IP Rights, are and shall remain the sole and exclusive property of Amplify or its third-party licensors. “IP Rights” means, collectively, rights under patent, trademark, copyright and trade secret laws, and any other intellectual property or proprietary rights recognized in any country or jurisdiction worldwide. Customer shall notify Amplify of any violation of Amplify’s IP Rights in the Products of which Customer has actual knowledge, and shall reasonably assist Amplify as necessary to remedy any such violation. Amplify Products are protected by patents (see http://www.amplify.com/virtual-patent-marking). 5. Payments. Intentionally Omitted. 6. Account Information. For subscription Products, the authentication of Authorized Users is based in part upon information supplied by Customer or Authorized Users, as applicable. Customer will and will cause its Authorized Users to (a) provide accurate information to Amplify or a third-party authentication service as applicable, and promptly report any changes to such information, (b) not share or allow others to use their account, (c) maintain the confidentiality and security of their account information, and (d) use the Products solely via such authorized accounts. Customer agrees to notify Amplify immediately of any unauthorized use of its or its Authorized Users’ accounts or related authentication information of which Customer has actual knowledge. Amplify will not be responsible for any losses arising out of the unauthorized use of accounts created by or for Customer and its Authorized Users. 7. Confidentiality. Intentionally Omitted. 8. Student Data. The parties acknowledge and agree that Customer is subject to federal and local laws relating to the protection of personally identifiable information of students (“PII”), including the Family Educational Rights and Privacy Act (“FERPA”), and that Amplify is obtaining such PII for the purpose of providing the Products hereunder. Subject to the terms and conditions of the RFP and this Agreement, Amplify will not take any action to cause Customer to be out of compliance with FERPA or other applicable North Carolina or Federal laws relating to PII. The terms and conditions of the RFP will govern collection, use and disclosure of information collected or stored on behalf of Customer under this Agreement, provided that such collection, use, and disclosure is not otherwise prohibited by North Carolina or Federal law. 9. Customer Materials. Customer represents, warrants and covenants that it has all the necessary rights, including consents and IP Rights, in connection with any data, information, content and other materials provided to or collected by Amplify on behalf of Customer or its Authorized Users using the Products or otherwise in connection with this Agreement (“Customer Materials”), and that Amplify has the right to use such Customer Materials as contemplated hereunder or for any other purposes required by Customer. Customer is solely responsible for the accuracy, integrity, completeness, quality, legality and safety of such Customer Materials. Customer is responsible for meeting hardware, software, telecommunications and other requirements listed at http://www.amplify.com/customer-requirements. 10. Warranty Disclaimer. Intentionally Omitted. 11. Limitation of Liability. Intentionally Omitted. 12. Term; Termination. Intentionally Omitted. 13. Miscellaneous. This Agreement may not be modified except in writing signed by both parties. All defined terms in this Agreement shall apply to their singular and plural forms, as applicable. The word “including” means “including without limitation.” This Agreement will be binding upon and inure to the benefit of the parties and their respective successors and assigns. The parties expressly understand and agree that their relationship is that of independent contractors. Nothing in this Agreement shall constitute Page 17 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 one party as an employee, agent, joint venture partner or servant of another. Each party is solely responsible for all of its employees and agents and its labor costs and expenses arising in connection herewith. If one or more of the provisions contained in this Agreement shall for any reason be held to be unenforceable at law, such provisions shall be construed by the appropriate judicial body to limit or reduce such provision or provisions so as to be enforceable to the maximum extent compatible with applicable law. 3. The State offers the following with regards to the Vendor’s Service Level Agreement attached to Vendor’s response to the RFP: The Parties agree that the Vendor’s Service Level Agreement, attached to Vendor’s response to this RFP, is deleted in its entirety and replaced with the following. For clarity, the Parties agree that the following Service Level Agreement replaces and supersedes both the Vendor’s Service Level Agreement attached to Vendor’s response to the RFP and the Vendor’s Service Level Agreement attached to Vendor’s response to BAFO Number 2 to this RFP. RtAD CONTRACTOR SERVICE LEVEL AGREEMENT This Service Level Agreement (SLA) describes the terms of customer and technical support to be provided under that certain agreement (the “Agreement”) between North Carolina DPI (the “State”), and Amplify Education, Inc. (“Contractor”), and the Statement of Work (or “SOW”) to which this Service Level Agreement is attached, if any. Contractor Platform Availability: The Contractor RtAD Platform will be available for use by the State and End Users (as “End Users” is defined in RFP No. 40-IT00107-16) at a minimum as defined in the following two (2) Service Windows for monitoring and reporting the SLA metrics of availability uptime percentage and calculating service credits for each calendar month that this SLA and the Agreement are in effect. Service Window #1: 7:00 AM to 6:00 PM EST with a 99.9% RtAD Platform availability Monday through Friday excluding any Maintenance Windows pre-approved by NCDPI. Service Window #2: 6:00:01 PM to 6:59:59 AM EST Monday –Thursday and 6:00:01 PM Friday to 6:59:59 AM EST Monday with a 99.5% RtAD Platform availability for this Service Window excluding any Maintenance Windows pre-approved by NCDPI. NCDPI approved scheduled recurring Maintenance Windows shall only take place during Service Window #2. Unless otherwise agreed by the parties, such scheduled recurring Maintenance Windows shall be 9-11 pm Tuesdays and Fridays. Unscheduled Downtime: Is defined as any event that causes the RtAD Platform to be unavailable for use by the State and its authorized users for less than the following uptime specification listed in the Service Window tables below. Service Window #1 SYSTEM UPTIME CREDIT APPROXIMATE MONTHLY UNSCHEDULED DOWNTIME PERCENTAGE (MINUTES) Page 18 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 ≥ 99.9 % 0% 14 minutes 21 Seconds 99.9% > and ≥ 99.8% 2.5% 14 min 22 sec – 28 min 42 sec 99.8% > and 99.7% 5% 28 min 43 sec – 43 Minutes 3 sec 99.7% > and ≥ 99.5% 8% 43 min 4 sec – 1 hour 11 min 45 sec 99.5% > 10% ≥ 1hour 11 min 46 sec Service Window #2 SYSTEM UPTIME CREDIT PERCENTAGE APPROXIMATE MONTHLY UNSCHEDULED DOWNTIME (MINUTES) ≥ 99.5 % 0% 2 hours 27minutes 24 seconds 99.5% > and ≥ 99.4% 2.5% >2 hrs. 27 min 25 sec – 2 hours 56 min 53 sec 99.4% > and ≥ 99.1% 3.5% >2 hours 56 min 54 sec – 4 hours 25 min 19 sec 99.1% > 5% ≥ 4 hours 25 min 20 sec Account Credit for RtAD Platform Unscheduled Downtime: In the event that the monthly RtAD System uptimes identified in the Service Window tables above is not met for any calendar month during the term of the Agreement, the State shall be entitled to an account credit in accordance with the tables set forth immediately above. All account credits shall be calculated by multiplying the RtAD Monthly Subscription Fee by the credit percentage that corresponds to the calculated Unscheduled Downtime. Example: For Service Window 1, if the Unscheduled Downtime for Month “X” is 29 minutes and the RtAD Monthly subscription Fee is $527,714 ($6,332,573/12mths), then the NCDPI account credit to the State for Month “X” is $26,386. ($527,714*.05). Maximum Credit: The State acknowledges and agrees that the account credits represented above are liquidated damages and represent the State’s sole remedy for the Contractors failure to meet the service levels set forth herein; and the aggregate amount of account credits, for all instances where the Contractor, in any given month does not meet the services levels set forth in Service Windows 1 and 2 above shall not exceed 15% of the Contractors monthly subscription Fee in total. Contractor RtAD Unscheduled Outage Reporting: The Contractor shall provide access to performance monitoring reports and upon request, will provide a report reflecting any unscheduled outages of the RtAD subscription service within 10 business days at the conclusion of the previous month. The report shall Page 19 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 include the Outage Date and Time, outage description, duration, and remedy. Excluded Events: There shall be no excluded events from the Contractor RtAD platform with the exception of events beyond the Contractor and Contractor’s subcontractors’ and representatives’ reasonable control, including without limitation, fire, power failures, any act of war, hostile foreign action, nuclear explosion, riot, strikes, civil insurrection, earthquake, hurricane, tornado, or other catastrophic natural event or act of God. The Contractor shall establish industry standard RtAD system redundancy to ensure maximum RtAD platform reliability and minimalize Force Majeure conditions. Force Majeure reported Contractor outages shall be accompanied by validated evidence to support the event. The Contractor shall be bonded and insured for severe weather related outage events. The Contractor shall make all reasonable efforts to protect RtAD State data and return the RtAD systems to operational status promptly in response to a Force Majeure event. Problem Identification and Resolution Response Times: Problem Resolution In general, during the term of the Agreement, Contractor will use commercially reasonable efforts to provide problem resolution and technical support on any and all questions that the State has regarding the Services or Deliverables that Contractor is obligated to provide under the Agreement. As used herein, a “High Priority” or “Priority 1” problem is any systemic and material failure of the Contractor Platform to operate in accordance with its documentation that substantially impedes assessment administration, scoring or reporting. As used herein, a “Normal Priority” problem is any failure of the Contractor Platform to operate substantially as described in its documentation. Contractor shall provide resolution to all problems identified to the Support Number as follows: Normal Priority problems: The Contractor will provide a preliminary status update and, to the extent practical, a preliminary remedy to the State not later than the next business day after the problem is initially logged with technical support. Contractor will use commercially reasonable efforts to work diligently to correct (or provide a reasonable work-around for) any Normal Priority problems that are not immediately resolvable as soon as reasonably possible in light of their severity and overall impact on the State’s use of the Contractor Platform. High Priority problems: The Contractor will provide a preliminary update and, to the extent practical, a preliminary remedy to the State within one (1) hour after the High Priority problem is initially logged with technical support. Contractor will promptly assign appropriate r e s o u r c e s to correct (or provide a reasonable work-around for) any High Priority problems that are not immediately resolvable in a highly expedited manner. Without limiting the generality of the foregoing, for all problems not resolved within designated time frames, Contractor will use commercially reasonable efforts to provide the State with additional resources as necessary for expedient problem resolution and timely follow-up. Contractor will be responsible for delays, High Priority, Normal Priority or other problems or failures that arise with the exception of Excluded Events (as defined above). The State also Page 20 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 acknowledges and agrees that Contactor’s ability to resolve any problems hereunder on a timely basis is dependent upon the State’s cooperation in assisting with identifying, reporting, and replicating functionality problems hereunder, and providing other cooperation as may reasonably be necessary. Additionally, the State agrees that the Contractor and the State shall work together in good faith to undertake a root cause analysis to ascertain the source of the issue, and further work together in good faith and offer reasonable cooperation in an attempt to resolve any such issues or disputes relating thereto. 4. The North Carolina Department of Public Instruction has an existing contract with the Vendor that terminates on August 30, 2016. In the event that the State awards this RFP to Vendor, the term of the resulting contract will overlap with the term of the existing contract. The State therefore requests that the Vendor provide the State with a credit for each calendar day that the term of the contract resulting from this RFP overlaps with the term of the existing contract. This per day credit will be multiplied by the number of days remaining on the existing contract. The total amount of the credits shall then be deducted from the first invoice submitted from the Vendor to the State during the term of the contract awarded pursuant to this RFP, if any. Please provide the amount of the credit below: 22,691.80 $______________________ per calendar day. 5. The State encourages the vendor to supply more competitive prices. Vendor should submit its most competitive prices starting on page [22] of this request for BAFO 3. PLEASE NOTE THE FOLLOWING: 1) NCDPI is requesting a cost proposal without the DIBELS Deep. Please incorporate this information, when completing the Read to Achieve Diagnostic Software as a Service Cost Template below. 2) Exclude optional terms from the overall contract cost and provide them separately. 3) Clarify the final price is inclusive of the official links to guide teachers in instructions based on the MSV analysis (as agreed to during BAFO 01).4) Lastly, provide a payment schedule using the following as a guide. Refer to section IV (2) of the proposal for further details: 1. Recurring Quarterly Operational Cost 2. Deliverable/Milestones based payment schedule Section IV. Cost Proposal 1) The Vendor must submit costs (inserting as many rows as necessary) in the following Cost Response Form: Page 21 of 23 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 Cost Offer (450,000 or more students) Cost Component One-Time Costs Describe Cost Basis Project management $0.00 Included in subscription fee below. Customization required or proposed addressing specification $0.00 Included in subscription fee below. Technical and user documentation $0.00 Included in subscription fee below. Installation/conversion/integration/ $0.00 transition costs Included in subscription fee below. Training including training materials $0.00 Included in subscription fee below. Other (describe) $0.00 Total One-Time Costs $0.00 Cost Component Recurring Costs Year 1 Year 2 Year 3 Describe Cost Basis Subscription fees or costs $6,332,573 $6,332,573 $6,332,573 $13 annual per student license fee for 450,000 students or more. The summary cost in each year assumes purchase of 487,121 licenses in each year. Final cost will be based on actual number of students. Enhancement costs $ $ $ Technical support-customer service $ $ $ Other (describe) $ $ $ Total Recurring Costs $6,332,573 $6,332,573 $6,332,573 Contract Total Page 22 of 27 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 Grand Total – Entire Contract (including Optional Years) $18,997,719 *Provide Costs separately for any optional/desired service features if not already included in the offered costs above. $1,100,000** -Supplemental kit price. See notes below. Notes **This cost reflects the purchase of additional texts for the existing kits that are already used across the State, as described in Section D.2.3.1 (p. 195) of the proposal. Please note that this extension of the book set was based on feedback that we have received from North Carolina educators and administrators. If NCDPI wishes to modify this plan (such as reduce the number of b o ok s per set or adjust the total number of sets purchased for the districts) or remove the extension, we would be happy to adjust the cost accordingly or remove it entirely. This supplement includes:   Additional nonfiction texts at select levels; and Texts at upper levels that are not covered at all by the existing kits (Note that, since the upper level texts will be required less frequently, the texts will be allocated to Grade 3 classrooms and may be shared with teachers at lower grade levels in the school). Payment Plan Proposal Year 1 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Amount $2,691,344 $1,213,743 $1,213,743 $1,213,743 Payment Due Date 9/30/2016 12/30/2016 3/30/2017 6/30/2017 Amount $2,691,344 $1,213,743 $1,213,743 $1,213,743 Payment Due Date 9/30/2017 12/30/2017 3/30/2018 6/30/2018 Amount $2,691,344 $1,213,743 $1,213,743 $1,213,743 Payment Due Date 9/30/2018 12/30/2018 3/30/2019 6/30/2019 Year 2 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Year 3 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Page 23 of 27 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 Alternative Cost Response 1 (250,000 to 449,999 students) One-Time Costs Cost Component Describe Cost Basis Project management $0.00 Included in subscription fee below. Customization required or proposed addressing specification $0.00 Included in subscription fee below. Technical and user documentation $0.00 Included in subscription fee below. Installation/conversion/integration/ transition costs $0.00 Included in subscription fee below. Training including training materials $0.00 Included in subscription fee below. Other (describe) $0.00 Total One-Time Costs $0.00 Cost Component Recurring Costs Year 1 Describe Cost Basis Year 2 Year 3 Subscription fees or costs $6,637,485.25 $6,637,485.25 $6,637,485.25 $14.75 annual per student license fee for up to 449,999 students. The summary cost in each year assumes purchase of 449,999 licenses in each year. Final cost will be based on actual number of students. Enhancement costs $ $ $ Technical support-customer service $ $ $ Other (describe) $ $ $ Total Recurring Costs $6,637,485.25 $6,637,485.25 $6,637,485.25 Contract Total Page 24 of 27 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 Grand Total – Entire Contract (including Optional Years) $19,912,455.75 *Provide Costs separately for any optional/desired service features if not already included in the offered costs above. $1,015,884** - Supplemental kit price. See notes below. Notes **This cost reflects the purchase of additional texts for the existing kits that are already used across the State, as described in Section D.2.3.1 (p. 195) of the proposal. Please note that this extension of the book set was based on feedback that we have received from North Carolina educators and administrators. If NCDPI wishes to modify this plan (such as reduce the number of b o ok s per set or adjust the total number of sets purchased for the districts) or remove the extension, we would be happy to adjust the cost accordingly or remove it entirely. This supplement includes:   Additional nonfiction texts at select levels; and Texts at upper levels that are not covered at all by the existing kits (Note that, since the upper level texts will be required less frequently, the texts will be allocated to Grade 3 classrooms and may be shared with teachers at lower grade levels in the school). Payment Plan Proposal Year 1 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Amount $2,820,931 $1,272,185 $1,272,185 $1,272,185 Payment Due Date 9/30/2016 12/30/2016 3/30/2017 6/30/2017 Amount $2,820,931 $1,272,185 $1,272,185 $1,272,185 Payment Due Date 9/30/2017 12/30/2017 3/30/2018 6/30/2018 Amount $2,820,931 $1,272,185 $1,272,185 $1,272,185 Payment Due Date 9/30/2018 12/30/2018 3/30/2019 6/30/2019 Year 2 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Year 3 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Page 25 of 27 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 Alternative Cost Response #2 (1 to 249,999 students) One-Time Costs Cost Component Describe Cost Basis Project management $0.00 Included in subscription fee below. Customization required or proposed addressing specification $0.00 Included in subscription fee below. Technical and user documentation $0.00 Included in subscription fee below. Installation/conversion/integration / transition costs $0.00 Included in subscription fee below. Training including training materials $0.00 Included in subscription fee below. Other (describe) $0.00 Total One-Time Costs $0.00 Cost Component Recurring Costs Year 1 Year 2 Year 3 Subscription fees or costs $3,937,484.25 $3,937,484.25 $3,937,484.25 Enhancement costs $ $ $ Technical support-customer service $ $ $ Other (describe) $ $ $ $3,937,484.25 $3,937,484.25 Total Recurring Costs $3,937,484.25 Page 26 of 27 Describe Cost Basis $15.75 annual per student license fee for up to 249,999 students. The summary cost in each year assumes purchase of 249,999 licenses in each year. Final cost will be based on actual number of students. 03/07/2016 BAFO Number: 40-IT00107-16 / BAFO 3 Contract Total Grand Total – Entire Contract (including Optional Years) $11,812,452.75 *Provide Costs separately for any optional/desired service features if not already included in the offered costs above. $564,379** - Supplemental kit price. See notes below. Notes **This cost reflects the purchase of additional texts for the existing kits that are already used across the State, as described in Section D.2.3.1 (p. 195) of the proposal. Please note that this extension of the book set was based on feedback that we have received from North Carolina educators and administrators. If NCDPI wishes to modify this plan (such as reduce the number of b o ok s per set or adjust the total number of sets purchased for the districts) or remove the extension, we would be happy to adjust the cost accordingly or remove it entirely. This supplement includes:   Additional nonfiction texts at select levels; and Texts at upper levels that are not covered at all by the existing kits (Note that, since the upper level texts will be required less frequently, the texts will be allocated to Grade 3 classrooms and may be shared with teachers at lower grade levels in the school). Payment Plan Proposal Year 1 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Amount $1,673,431 $754,684 $754,684 $754,684 Payment Due Date 9/30/2016 12/30/2016 3/30/2017 6/30/2017 Amount $1,673,431 $754,684 $754,684 $754,684 Payment Due Date 9/30/2017 12/30/2017 3/30/2018 6/30/2018 Amount $1,673,431 $754,684 $754,684 $754,684 Payment Due Date 9/30/2018 12/30/2018 3/30/2019 6/30/2019 Year 2 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Year 3 Quarterly Payment Payment 1 Payment 2 Payment 3 Payment 4 Page 27 of 27 03/07/2016