Exhibits to Amplify’s July 24, 2019 letter to DPI A. Evaluation Consensus votes summary (presented at protest meeting, derived from July 12 public records production) B. DIT cancellation approval memo (presented at protest meeting, from DIT records production) C. March 21 memo from DPI to DIT requesting cancellation (copy provided by DPI at protest meeting) D. Evaluation Criteria changes (presented at protest meting derived from July 12 public records production) E. May 23, 2019 Draft DPI Award memo (p. 5 notes DPI only used “evaluation factors deemed appropriate”, this text was removed from final memo) F. SOC 2 Type II Issues Evaluation Consensus  EXHIBIT A   Requisition No. RQ20680730 (Meeting Notes, November 19-20, 2018)    Committee Votes  Amplify  Istation  Y E S V O T E S   Y E S V O T E S   R E Q U I R E M E N T S  Dyslexia Screening (BS9)  11 • ••••••••••  Developmentally Appropriate (BS8)  11 • ••••••••••  Information Security Audit SOC2 (TS35)  11 • ••••••••••  Reading Assess To Inform Instruction (BS1)  11 • ••••••••••  Standards Alignment (BS17)  7 • ••••••  Progress Monitoring (BS6 )  8 • •••••••  1 •   Diagnostic Assessment (B S12)  10 • •••••••••  2 • •  Project Management /WBS (P MS1)  11 • ••••••••••  2 • •  Educators/Students Use Devices (BS10)  11 • ••••••••••  3 • ••  Minimize Time (BS15)  7 • ••••••  3 • ••  Parent Communications (RS4)  7 • ••••••  3 • ••  Formative Assessments ( BS11)  9 • ••••••••  7 • ••••••  Assessment Validity & Reliability (BS3)  10 • •••••••••  9 • ••••••••  Benchmarking (BS24)  11 • ••••••••••  10 • •••••••••  Identifying Intervention / Enrichment (BS5)  11 • ••••••••••  11 • ••••••••••  EVAAS (BS19)  11 • ••••••••••  11 • ••••••••••  Reporting Capabilities (RS3)  11 • ••••••••••  11 • ••••••••••  NC Security Manual (TS6)  0 ZERO YES VOTES  11 • ••••••••••  0 ZERO YES VOTES  0 ZERO YES VOTES  0 ZERO YES VOTES  0 ZERO YES VOTES  0 ZERO YES VOTES      Notes:     The tallies above are reported in pages 1-31 of the Read to Achieve (RtAD) Evaluation Consensus Meeting Notes from November 19  and 20, 2018. The stated purpose of the meeting is to rank the proposal vendors.     The meeting summary notes that the team unanimously agreed to review the “legislatively mandated specifications” and rank the  vendors, as follows: “Yes” implies conforms to specifications; “No” implies does not conform to specifications; “Maybe” implies that the  team is unsure about conformity.    The chart above shows the “legislated specifications” reviewed at the Consensus Meeting in order of “Yes” votes for Istation. “BS”  stands for “Business Specification”; “TS” stands for “Technical Specification”; “PMS” stands for “Project Management Specification”;  “RS” stands for “Reporting Specification.”    DocuSign Envelope ID: 6DE03F0B-9F25-4589-9F45-C8C87533B6A9 Statewide IT Procurement Office Purchase Transaction Approval (PTA) Form For Cancellation Agency: Department of Public Instruction Agency Requisition No.: RQ2068073 Tracking Number: Contract Specialist: EXHIBIT B 300042-003 Andrea Pacyna REQUEST APPROVAL FOR CANCELLATION Description of Reason for Cancellation: DPI is requesting to cancel an RFP for the procurement of Read to Achieve Diagnostics Software. Four (4) offers were received on or before the due date specified. All were responsive. DPI subsequently began their evaluation and discovered that two (2) of the four (4) Vendors had significant weaknesses as compared to the specifications issued in the RFP. Neither Vendor (Curriculum Associates and NWEA) appeared ready to execute a large, state wide effort, both had significant technical limitations and compliance with SOC 2 Type II was either absent or scheduled to be completed later. The two other (2) Vendors that responded appear capable of satisfying our Agency’s business needs but are at a disparate price range (so unlike that there is no basis for comparison). Specifically, the prices are: Amplify Education: $12,102,096.08 Istation: $3,098,606.17 As a result, DPI believes that continuing with the RFP would serve no valuable purpose and they would like to cancel. Pursuant to the definition of “NEGOTIATION” in 9 NCAC6B.0316, DPI intends to conduct negotiations with sources of supply to maximize the State’s ability to obtain best value based on the evaluation factors set forth in the solicitation. Contract Specialist Certification: I certify that the above described addendum/amendment was transacted in accordance with NC General Statutes, the NC Administrative Code and Policies & Procedures of the Department of Information Technology. Andrea Pacyna 3/25/19 Contract Specialist – Andrea Pacyna Date Certification: Statewide IT Procurement Management or Designees: I certify I have reviewed the above described amendment/addendum and to my knowledge was transacted in accordance with NC General Statutes, the NC Administrative Code and Policies & Procedures of the Department of Information Technology. 3/25/2019 3:44 PM EDT Statewide IT Procurement Manager – Leroy Kodak Date 3/27/2019 10:24 AM EDT Chief IT Procurement Officer – Patti Bowers Date Rev. 2018/02/26 INSTRUCTIONS: Adjust the highlighted text with the appropriate information. Finalize the document prior to submitting it to the Statewide IT Procurement Office by deleting this box, the directions in red type, and any highlighted areas. Please update the Table of Contents before submitting. Note: Per 09 NCAC 06B.0314, completion of specific sections of the template are required. These sections are highlighted in blue. EXHIBIT C Roy Cooper Governor Eric Boyette Secretary of Information Technology State Chief Information Officer Request for RFP Cancellation 40-IT00115-18 To: Patricia Bowers, DSCIO Statewide IT Procurement Office From: Tymica Dunn Date: March 21, 2019 Subject: Request for RFP Cancellation Read to Achieve Diagnostics – Software as a Service Reference #: DIT No. 40-IT00115-18 – DPI RFP No. 40-RQ20680730 Enclosed for your review and approval is a Request for RFP Cancellation for RFP 40-RQ20680730. In accordance with 09 NCAC 06B.0316 (Negotiation) and 09 NCAC 06B.0901 (Conditions for Limited or Waived Competition), the Department of Public Instruction (DPI) requests approval to conduct negotiations with the following two (2) Vendors: Amplify Education, Istation, Curriculum Associates, and NWEA. In response to RFP No 40-RQ20680730 DPI received bids from four (4) Vendors prior to the submission deadline. Discussions regarding vendor evaluations determined that the voting members found the following significant weaknesses with the vendors’ product: Curriculum Associates • • • • Fluency measure is not ready and will not be available until the 2021 School Year. There was skepticism about how the foundational questions were assessed just by measuring comprehension skills without fluency. The measures for oral language were also very limited. Progress Monitoring appeared to be a shortened version of the assessment and this brings up a question about the reliability of the assessment. This service requires a lot of reporting customization. It also requires manual intervention for student transfers. • • The company plans to complete SOC 2 Type II audit in the summer of 2019. The implementation plan proposed is not for statewide implementation and appeared time consuming which brought to question the ability to deploy for the 2019 School Year. This service will require a lot of customization for a statewide implementation. NWEA • • • • Service does not directly assess oral language. Progress Monitoring is not in place and implementing this solution will be a problem for the 2019 School year. Since this service does not have Progress Monitoring in place in the interim the growth checklist is used. This tool is also not a good diagnostic and formative assessment screener. Their project implementation timeline is not for a statewide implementation and there were serious doubts how the statewide implementation can be handled. There were concerns raised about teachers viewing other teacher’s students. Time must be accounted for customizations to be FERPA compliant. The vendor has indicated negotiating state security standards and has not given a clear response for SOC 2 Type II audit requirement which will delay approval. There are, however, two (2) Vendors that appear capable of satisfying our Agency’s business needs. Specifically, they are: Amplify Education: $12,102,096.08 Istation: $3,098,606.17 Soliciting formal offers via IPS again would serve no valuable purpose. Pursuant to the definition of “Negotiation” in 9 NCAC6B.0102, DPI intends to conduct negotiations to maximize the State’s ability to obtain best value based on the evaluation factors set form in the solicitation. We are proceeding in this manner. DPI would like to formally request that the project remain open in touchdown until a contract is established. If you have any concerns, please advise. Thank you for your assistance. If additional information is required, please do not hesitate to contact us. cc: Evaluation Committee Leroy Kodak, Contract Manager Andrea Pacyna, Contract Manager Glenn Poplawski, Chief Solutions Officer Kathy Bromead, PMA EXHIBIT D Evaluation Criteria: Read to Achieve Diagnostics, Requisition No. RQ20680730 Fall 2018 Evaluation PRIORITY EVALUATION CRITERIA Amplify Istation RANK RANK Spring 2019 Evaluation PRIORITY EVALUATION CRITERIA 1 Substantial Conformity To Specification 1 2 1 Vendor Cost Proposal 2 RtAD Desired Specifications 1 2 2 Vendor Financial Stability 3 Proof Of Concept / Demonstration 1 2 3 Formative & Diagnostic Assessment 4 Vendor Cost Proposal 4 1 4 Personalized Learning 5 Strength Of References 1 1 6 Vendor Financial Stability 4 2 Overall Rank 1 2 “Personalized Learning” Criterion Listed As “Desirable But Not Required” Source: Fall 2018 Evaluation: Read to Achieve -2018, December 4, 2018 presentation; Spring 2019 Evaluation: Contract Award Recommendation, June 7, 2019. Note: Evaluation criteria were listed in order of importance in both evaluations. Request for Negotiation EXHIBIT E Roy Cooper Governor Eric Boyette Secretary of Information Technology State Chief Information Officer Contract Award Recommendation To: Andrea Pacyna From: Tymica Dunn Procurement Chief Department of Public Instruction Date: May 23, 2019 Subject: Contract Award Recommendation Read to Achieve Diagnostics - Requisition # - RQ20680730, DIT File #300042 Reference #: Request for Negotiations 40-RQ20680730, DIT File #300042 Deputy Chief IT Procurement Officer Department of Information Technology Enclosed for your review and approval is the award recommendation for Requisition number Bids received pursuant to a RFN #40-RQ20680730A have been reviewed and an Evaluation Committee hereby requests the Statewide IT Procurement Office to award the contract, as follows: Description: Recommended Vendor: Read to Achieve Diagnostics – Software as a Service Imagination Station Inc., dba, Istation Cost: $8,255,818.50 for 3 years Contract Term: Two (2) years plus 1 (one) year optional renewals at the discretion of the State Project Name and Number: Read to Achieve Diagnostics - 2018 DIT file # 300042 Thank you for your assistance. If additional information is required, please do not hesitate to contact me. cc: Evaluation Committee Patti Bowers, DSCIO Glenn Poplawski, DSCIO Kathy Bromead, PMA 1 of 10 v.2017-06-06 Request for Negotiation Table of Contents Section 1: Introduction.....................................................................................................................3 Section 2: Evaluation Committee ....................................................................................................3 Section 3: Evaluation Criteria / Methodology .................................................................................4 Section 4: Timeline ..........................................................................................................................4 Section 5: Evaluation of Bid Submission ........................................................................................5 Section 6: Vendors...........................................................................................................................5 A. Evaluation Criteria .....................................................................................................5 B. Cost ........................................................................................................................... 6 C. Vendor Financial Stability ........................................................................................ 7 D. Formative and Diagnostic Assessment ..................................................................... 8 E. Personalized Learning............................................................................................... 8 Section 7: Finalist Vendor(s) ........................................................................................................10 Section 8: Award Recommendation ..............................................................................................10 Section 9: Supporting Documentation ...........................................................................................10 2 of 10 v.2017-06-06 Request for Negotiation Section 1: Introduction The purpose of this award recommendation and the resulting contract award is to identify a vendor best qualified to offer services for Read to Achieve Diagnostic Software as a Service solution (RtAD) to meet NCDPI’s obligations under state law, N.C.G.S. 115C-83.1, et. seq. North Carolina state law requires kindergarten through third grade students to be assessed with valid, reliable, formative and diagnostic reading assessments. NCDPI is obligated to adopt and provide these developmentally appropriate assessments. The solution must assess student progress, diagnose difficulties, inform instruction and remediation, and yield data that can be used with the Education Value-Added Assessment System (EVAAS). The Evaluation Committee, consisting of four (4) Decision Makers, used “best value” methodology authorized by N.C.G.S. §§143-135.9 and 143B-1350(h) during the evaluation process. Section 2: Evaluation Committee Name Title/Agency Participation Level Berry, Erika Craver, Nathan Karkee, Thakur Shue, Pam AlHour, Julien Decision Maker Decision Maker Decision Maker Decision Maker SME Dunn, Tymica Senior Policy Advisor, NCDPI Digital Teaching and Learning Consultant, NCDPI Psychometrician, NCDPI Deputy Superintendent of Early Education, NCDPI Director - Architecture, Integration, & Quality Assurance, NCDPI Purchasing Section Chief, NCDPI Gossage, Chloe Strong, Melissa Viswanathan, Srirekha Chief Strategy Officer, NCDPI NCDPI Attorney Project Manager, NCDPI Procurement Officer SME SME Project Manager 3 of 10 v.2017-06-06 Request for Negotiation Role Definitions: Decision Maker: Key business stakeholders evaluating the bid responses. Voting Project Manager: Overall responsibility includes successful initiation, planning, design, execution, implementation, and closure of a project. Non-Voting Subject Matter Expert (SME) Person who is an authority in a particular technical area pertaining to the procurement Non-Voting Section 3: Evaluation Criteria / Methodology The selection process was conducted using the “best value” methodology authorized by N.C.G.S. §§143-135.9 and 143B-1350(h). The evaluation committee met as a group and evaluated the responsive proposals. The evaluation criteria listed below is in the order of importance: Evaluation Criteria Cost Vendor Financial Stability Formative and Diagnostic Assessment Personalized Learning Section 4: Timeline Date Milestone March 21, 2019 RFP Cancellation Notifications sent to vendors, Request to Negotiate Review Period RFP proposals were extended to June 29, 2019 – Clarification 1 Negotiation Meeting with vendors Clarification issued to vendors – Clarification 2 Clarification response received and shared with evaluation team Evaluation Committee meeting and discussion of proposal strengths and weakness Clarification issued to vendor – Clarification 3 Clarification response received and shared with evaluation team Clarification issued to vendor – Clarification 4 March 27, 2019 April 11, 2019 April 17, 2019 April 23, 2019 April 25, 2019 May 3, 2019 May 15, 2019 4 of 10 v.2017-06-06 Request for Negotiation May 23, 2019 May 23, 2019 Clarification response received and shared with evaluation team Draft Best and Final Offer (BAFO) Draft Award Recommendation Section 5: Evaluation of Bid Submission In response to DIT No. 40-IT00115-18 – DPI RFP No. 40-RQ20680730 four (4) bid proposals were received. The RFP cancellation was issued on March 21, 2019 since NCDPI deemed soliciting formal offers via IPS again would serve no valuable purpose. Pursuant to the definition of “Negotiation” in 9 NCAC6B.0102, DPI conducted negotiations with two (2) of the four (4) vendors to maximize the State’s ability to obtain best value based on the evaluation factors determined appropriate. Proposal response from the following two vendors were deemed responsive and considered for further negotiations Vendor Responsive / Non-Responsive Amplify Education Inc. Imagination Station Inc. Responsive Responsive Section 6: Vendors Listed below is a synopsis of each proposal submitted based on the criteria defined in Section 3. A. Evaluation Criteria "Best Value" procurement method authorized by N.C.G.S. §§143-135.9 and 143B-1350(h) has been used for this evaluation. A one step source selection was used. The proposals were objectively evaluated using the evaluation criteria described below. The evaluation team members did their due diligence and issued clarifications for each proposal before meeting the vendors on April 11, 2019. Strengths and weaknesses were discussed during the evaluation meeting on April 25,2019. The following evaluation criteria was used to determine strengths and weakness 1. Cost 2. Vendor Financial Stability 3. Formative and Diagnostic Assessment 4. Personalized Learning 5 of 10 v.2017-06-06 Request for Negotiation B. Cost The strengths and weaknesses identified by the Evaluation team for the responsive vendors are summarized in the tables below. Cost Vendor Amplify Strengths Amplify’s clarification response was considered for cost evaluation. None Weakness 1. The assessment service at $8 per student is expensive. 2. This assessment does not include lessons. 3. The assessment requires teacher intervention and is not fully online. 4. Professional Development costs $556,650 for the first year and is very expensive. 5. The expectation from NCDPI Consultants is enormous for Training and Professional Development. Istation Istation’s clarification response was considered for cost evaluation. None 1. The $5.70 per student option comes with 3000 teacher directed lessons, access to Istation's iPractice to learn and work from home 2. Students are allowed access from home to all the resources. 3. $76,103 for professional development offers 22 onsite training, 14 recorded live webinars and 10 virtual teacher training. 4. Additionally, 5 onsite and 10 recorded webinars will be provided at no additional cost. 6 of 10 v.2017-06-06 Request for Negotiation Cost Vendor Strengths Weakness 5. The cost for Professional Development also covers the logistics and the vendor has taken responsibility for logistics. After discussing the overall strengths, weaknesses of the cost proposal, the Evaluation Team moved on to the review of financial stability. C. Vendor Financial Stability The strengths and weaknesses identified by the Evaluation team for the responsive vendors are summarized in the tables below. Vendor Financial Stability Vendor Strengths Weakness Amplify 1. There is no going concern. 1. In a thorough review of the Financial Statement, DPI's Financial Director found that the losses have gone up from $42,684,000 to $54,459,000. 2. The financial statement presented was still unaudited. No independent auditor's report is available. Istation 1. There is no going concern. 2. This vendor has good liquidity position. None After discussing the overall strengths, weaknesses of the vendor financial stability, the Evaluation Team moved on to review formative and diagnostic assessment capabilities of the vendors. 7 of 10 v.2017-06-06 Request for Negotiation D. Formative and Diagnostic Assessment The strengths and weaknesses identified by the Evaluation team for the responsive vendors are summarized in the tables below. Vendor Amplify Formative and Diagnostic Assessment Strengths 1. Assessment hits all the mark. 2. Enough Item pool for 20 assessments. 3. Good Reporting capabilities. Istation 1. Adaptive assessment allows children to reach their full potential. 2. Excellent reports for teachers to interpret the outcome of the assessment. 3. Enough item pool for 10 assessments Weakness 1. Assessment is time consuming 2. Requires excessive teacher involvement in the lower cost option 3. The $8 option is not adaptive 4. NC has been using Amplify suite for six years now. The progress made by students in reading is concerning. This brings to question the effectiveness of the data driven instructional support. 1. Does not currently support screen readers or keyboard. After discussing the overall strengths, weaknesses of the formative and diagnostic assessment capabilities, the Evaluation Team moved on to review personalized learning capabilities of the vendors. E. Personalized Learning The strengths and weaknesses identified by the Evaluation team for the responsive vendors are summarized in the tables below. 8 of 10 v.2017-06-06 Request for Negotiation Vendor Formative and Diagnostic Assessment Strengths Amplify 1. Personalized Learning is good in the Higher option. 2. Progress Monitoring is available at a subskill level. Istation 1. The assessment is adaptive and is online and caters to the individual student's need. 2. The time for assessment is less. This allows teachers more time with the students to support their individual needs. 3. Students are allowed to personalize their learning by choosing games and activities while working outside of school Weakness 1. The lower cost option does not have all aspects of personalized learning and is not adaptive. 2. Progress Monitoring at the subskill level takes up a lot of time for teachers. 3. Home Reading is not part of the lower cost option. None 9 of 10 v.2017-06-06 Request for Negotiation Based on the review of all evaluation criteria, Istation was ranked first (1st) and Amplify was ranked second (2nd). The evaluation team unanimously decided to proceed for BAFO with Istation as the only finalist based on the overall strengths on the above four evaluation criteria. Section 7: Finalist Vendor(s) Based on the consensus decision of the Evaluation Team, Istation was issued the Best And Final Offer (BAFO) clarifications. Best and Final Offer Negotiations The procurement officer of DPI sent the BAFO questions to IStation. In the BAFO document the following were clarified and confirmed : IStation also completed the Vendor Security Assessment Guide (VRAR) that was reviewed by DPI and DIT technical teams. Section 8: Award Recommendation The Evaluation Committee has determined that Istation’s bid substantially conforms to the requirements of the law, therefore, recommends awarding RFP No. to Imagination Station Inc. (IStation) in the amount of $$8,255,818.50 for 3 years with the option of one (1) additional one (1) year renewals. Section 9: Supporting Documentation The following supporting documents that reflect the vendor selection are included: Bid Response Clarification documents – Signed BAFO document Hosting Exception and Privacy and Threshold Analysis (approved by DIT) 40-IT00109-17 Rationale for OER Cancellation 1. 2. 3. 4. 10 of 10 v.2017-06-06 EXHIBIT F SOC 2 Type II Issues DPI's request for cancellation also omits the fact that IStation has not yet completed a third party security attestation, while highlighting an analogous shortcoming in the proposals from Curriculum Associates and NWEA (vendors that were excluded from the negotiation process). Notably, public records show that the Evaluation Committee was aware that a failure to comply with the third party attestation requirement would make it challenging to obtain DIT's approval for an award. In DPI's Request for RFP Cancellation to DIT, dated March 21, 2019 (Exhibit C), DPI stated that "Neither Vendor (Curriculum Associates and NWEA) appeared ready to execute a large, state wide effort, both had significant technical limitations and compliance with SOC 2 Type II was either absent or scheduled to be completed later. In the case of Curriculum Associates, the memo states that the company plans to complete SOC 2 Type II audit in the summer of 2019. DIT's own cancellation approval then restates this issue identified by DPI in justifying excluding the vendors from the negotiation, stating that both Curriculum Associates and NWEA "had significant technical limitations and compliance with SOC 2 Type II was either absent or scheduled to be completed later." (Exhibit B). However, public records show that IStation also has not completed its SOC 2 Type II attestation process. The Consensus Meeting notes from November 19, 2018 state that Istation “agreed to complete the SOC 2 Type II by Spring 2019” in order to meet the requirement to provide a 3rd party attestation (Technical Specification 35). The notes go on to say that "DIT will not permit agencies to issue new contracts for moderate level solutions that have not been received an acceptable 3rd party attestation. Obtaining approval from DIT to award a contract with a vendor that does not already meet this requirement will be challenging." (PDF p. 61 of 166.) Public records produced by DIT include a May 22, 2019 email from DPI to DIT, sharing a draft BAFO for IStation that includes a roadmap for SOC 2 Type II assessment for the service that shows that only the initial readiness assessment will be complete by June 2019, and SOC 2 Type II evaluation complete in December 2019, with report available in February 2020. In its initial proposal, IStation only submitted a SOC 2 Type II report for its hosting provider, Rackspace (which is akin to Curriculum Associates referencing an analogous report from AWS -- in both cases these reports would only extend to the hosting service, not the software application or the vendor's other security controls). Ultimately, it appears IStation’s proposal suffers from the same SOC 2 Type II issues that led to Curriculum Associates and NWEA from being excluded from consideration for award.