State of Florida Department of Education INVITATION TO NEGOTIATE FLORIDA STANDARDS ASSESSMENTS BID NUMBER: ITN 2014-47 DEADLINE FOR TECHNICAL QUESTIONS: November 8, 2013 @ 2:00 PM (There is no deadline for administrative questions) REPLIES ARE DUE BY: DECEMBER 12, 2013 @ 2:00 PM, EASTERN TIME (ET) MAIL OR DELIVER REPLIES TO: Florida Department of Education Bureau of Contracts, Grants and Procurement Management Services 325 West Gaines Street 332 Turlington Building Tallahassee, Florida 32399-0400 Attention: Fran Shewan Phone: (850) 245-9884 Florida Standards Assessments, ITN 2014-47 1 Table of Contents INVITATION TO NEGOTIATE REGISTRATION 5 1.0 GENERAL INSTRUCTIONS TO RESPONDENT SECTION 2 – CONTRACT CONDITIONS 2.0 GENERAL CONTRACT CONDITIONS 6 7 7 SECTION 1 - OVERVIEW SECTION 1 –INSTRUCTIONS SECTION 3 - INTRODUCTION 6 6 7 3.0 3.1 3.2 3.3 3.4 INTENT PURPOSE BACKGROUND DEFINITIONS CRITICAL EVENT DATES 7 7 8 8 10 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9. 4.10 4.11 4.12 PRE-SOLICITATION CONFERENCE: SITE INSPECTION: VISITOR’S PASS TO THE TURLINGTON BUILDING REPLY QUESTIONS & ANSWERS PROCUREMENT PROTESTS / NOTICE OF RIGHTS ORAL INSTRUCTIONS / CHANGES TO THE INVITATION TO NEGOTIATE (ADDENDA) MODIFICATIONS, RESUBMITTAL AND WITHDRAWAL RESTRICTIONS ON COMMUNICATIONS WITH DEPARTMENT STAFF CONFIDENTIAL, PROPRIETARY, OR TRADE SECRET MATERIAL POOR PERFORMANCE NOTICE WITHDRAWAL OF A REPLY DISCLOSURE OF REPLY CONTENTS AWARD 11 11 11 11 11 12 12 12 13 13 13 14 14 5.0 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 5.15 5.16 5.17 5.18 5.19 AUTHORIZED TO DO BUSINESS IN THE STATE OF FLORIDA LICENSED TO CONDUCT SERVICES IN THE STATE OF FLORIDA OTHER CONDITIONS IDENTICAL EVALUATION OF REPLIES DISCLOSURE STATEMENT SUB-CONTRACTING CONTRACTUAL OBLIGATIONS METHOD OF PAYMENT CONVICTED VENDOR LIST DIVERSITY IN CONTRACTING COSTS INCURRED IN RESPONDING SUBMISSION OF REPLIES BY SUBSIDIARIES OR AFFILIATES PROHIBITION OF GRATUITIES INDEPENDENT PRICE DETERMINATION FINANCIAL CONSEQUENCES PERFORMANCE BOND PARTICIPATION IN FUTURE STAGES OF THIS PROJECT ACCESSIBLE ELECTRONIC INFORMATION TECHNOLOGY SCRUTINIZED COMPANIES LISTS (AS APPLICABLE) LIQUIDATED DAMAGES 14 14 14 14 15 15 15 15 15 15 16 16 16 16 16 16 17 17 17 17 6.0 REPLY SUBMISSION 19 Florida Standards Assessments, ITN 2014-47 2 SECTION 4 – SPECIAL INSTRUCTIONS SECTION 5 – SPECIAL CONDITIONS SECTION 6 – SPECIAL INSTRUCTIONS – REPLY FORMAT & CONTENT 11 14 19 6.1 6.2 6.2.1 6.2.2 6.2.3 6.2.4 MAIL OR DELIVER REPLIES TO: (DO NOT FAX OR E-MAIL) REPLY FORMAT INSTRUCTIONS PRELIMINARY ADMINISTRATIVE REVIEW TECHNICAL REPLY (PART I) (20 HARD COPIES) PRICE REPLY (PART II) (20 HARD COPIES) PRESENTING THE REPLY SECTION 7 – SCOPE OF SERVICES 7.0 7.1.0 7.1.1 7.1.2 7.2.0 7.2.1 7.2.2 7.3.0 7.3.1 7.3.2 7.3.3 7.3.4 7.3.5 7.3.6 7.3.7 7.3.8 7.3.9 7.3.10 SCOPE OF SERVICES INTRODUCTION NOTICE TO RESPONDENTS PROJECT SCHEDULE PROGRAM DESCRIPTIONS EXPECTED NUMBER OF DISTRICTS, SCHOOLS, AND STUDENTS POTENTIAL TEST ADMINISTRATION DATES TEST DESIGN AND DEVELOPMENT TEST CONTENT TEST CONFIGURATION DEVELOP ITEMS REVIEW MEETINGS DEVELOP FIELD-TEST ITEM SETS AND FORMS DEVELOP STAND-ALONE ELA/L FIELD-TEST ITEM SETS AND FORMS CONSTRUCTING TEST FORMS PRODUCTION OF PRINTED TEST MATERIALS PRODUCTION OF COMPUTER-DELIVERED PRODUCTS PRODUCTION OF PAPER-BASED AND COMPUTER-BASED TEST, STYLE GUIDE, AND BRAILLE PRODUCTION PROCEDURES 7.3.11 PRODUCE SPECIAL FORMATS AND ACCOMMODATIONS FOR STUDENTS WITH DISABILITIES 7.4.0 TEST ADMINISTRATION 7.4.1 PACK AND DISTRIBUTE MATERIALS 7.4.2 MISSING MATERIALS REPORT AND INVENTORY 7.4.3 DISPOSITION AND STORAGE OF MATERIALS 7.4.4 RETRIEVAL OF ANSWER SHEETS AND TEST BOOKS 7.4.5 INNOVATIVE PRODUCT DESIGN 7.4.6 PRINTED PRODUCTS – ADMINISTRATION 7.4.7 TEST ADMINISTRATION ANCILLARY MATERIALS 7.4.8 ONLINE MANAGEMENT SYSTEM 7.4.9 COMPUTER-BASED TEST ADMINISTRATION OVERVIEW 7.4.10 CUSTOMER SERVICE AND TECHNICAL SUPPORT 7.5.0 SCANNING, SCORING, AND REPORTING 7.5.1 SCORING AND REPORTING SPECIFICATIONS 7.5.2 PREPARING SCHOOL AND STUDENT INFORMATION (MSID AND PREID) 7.5.3 SCANNING/EDITING PBTS AND EXTRACTING CBT RECORDS 7.5.4 SCORING 7.5.5 REPORTING 7.5.6 VERIFYING DATA 7.6.0 SCALING, EQUATING, SCORING, AND SPECIAL PSYCHOMETRIC STUDIES 7.6.1 ASSESSMENTS 7.6.2 SCALING, EQUATING, AND SCORING (PSYCHOMETRICS) 7.6.3 TECHNICAL OVERSIGHT AND SPECIAL STUDIES 7.7.0 PROGRAM MANAGEMENT 7.7.1 PROVISIONS GOVERNING THE WORK OF THE CONTRACTOR Florida Standards Assessments, ITN 2014-47 19 19 19 20 23 23 23 23 23 24 25 25 26 26 27 28 29 30 30 32 32 32 34 35 36 36 39 39 43 43 44 44 44 46 49 52 58 59 59 60 62 63 68 73 77 78 79 81 87 87 3 7.7.2 7.7.3 7.7.4 7.7.5 7.7.6 7.7.7 7.7.8 7.7.9 7.8 CONTRACTOR STAFF AND RESPONSIBILITIES SUBCONTRACTORS PROGRAM MANAGEMENT COMMUNICATION PROGRAM MANAGEMENT MEETINGS PROGRAM MANAGEMENT REPORTS MEETINGS INVOLVING FLORIDA EDUCATORS AND OTHER CITIZENS COMMUNICATION BETWEEN THE CONTRACTOR AND DISTRICTS REIMBURSABLE FUNDING CATEGORIES DELIVERABLES (SEE ATTACHED SECTION 7.8 DELIVERABLES PROVIDED IN AN EXCEL SPREADSHEET.) SECTION 8 – OPENING, EVALUATION AND AWARD 87 89 89 90 90 91 93 94 96 96 8.0 REPLY OPENING 96 8.1 REPLY EVALUATION AND NEGOTIATION PROCESS 96 8.2 EVALUATIONS 97 8.3 POSTING OF INTENDED AWARD 98 8.3.1 PROTEST OF INTENDED AWARD 98 8.3.2 INABILITY TO POST 98 8.4 AWARD OF THE CONTRACT 98 APPENDIX A: TRANSITION TO COMPUTER-BASED TESTING AND ADDITIONAL TEST ADMINISTRATION DATES 99 APPENDIX B: CRITICAL AND EXTREMELY CRITICAL ACTIVITIES 101 APPENDIX C: FLORIDA PUBLIC SCHOOL STUDENT PARTICIPATION ESTIMATES, DISTRICT NUMBERS, AND SPECIAL DISTRICT/SCHOOL NUMBERS 104 APPENDIX D: SAMPLE TEST DESIGN 107 APPENDIX E: PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS 108 APPENDIX F: TEST SECURITY REQUIREMENTS, STATUTE, AND RULE 114 APPENDIX G: TEST ADMINISTRATION MANUAL WORKFLOW 118 APPENDIX H: TEST ACCOMMODATIONS 124 APPENDIX I: DATA PRIVACY & SECURITY POLICY 131 APPENDIX J: REQUIREMENTS FOR SCORING PERFORMANCE ITEMS 137 ATTACHMENT 1: RESPONDENT’S PRICE REPLY 145 ATTACHMENT 2: DRUG-FREE WORKPLACE 146 ATTACHMENT 3: DISCLOSURE STATEMENT 147 ATTACHMENT 4: REFERENCES 148 ATTACHMENT 5: MINORITY SUB CONTRACTORS UTILIZATION SUMMARY 149 ATTACHMENT 6: VENDOR CERTIFICATION REGARDING SCRUTINIZED COMPANIES LISTS 150 ATTACHMENT 7: CONTRACT STANDARD TERMS AND CONDITIONS 151 Florida Standards Assessments, ITN 2014-47 4 State of Florida Department of Education Bureau of Contracts, Grants and Procurement Management Services 325 West Gaines Street 332 Turlington Building Tallahassee, Florida 32399-0400 INVITATION TO NEGOTIATE REGISTRATION ************************************************************************************************************************** COMPLETE AND RETURN THIS FORM ************************************************************************************************************************** Reply Number: ITN 2014-47 Title: Florida Standards Assessments Date & Time Reply Due: December 12, 2013 @ 2:00 PM, EASTERN TIME (ET) Potential Respondents should notify the Florida Department of Education, Bureau of Contracts, Grants and Procurement Management Services by returning this Intent to Submit Reply Form as soon as possible after downloading. Complete the information below and send this sheet only to below e-mail address (preferred). Form can also be faxed to fax number (850) 245-0719, or mailed to 325 West Gaines Street, 332 Turlington Building, Tallahassee, Florida 32399-0400. Company Name: Contact Person: Address: City, State, Zip: Telephone: ( ) Fax Number: ( ) Internet E-Mail Address: Signed: Date: For further information on this process, you may contact Fran Shewan – fran.shewan@fldoe.org at (850) 245-9884. (Revised 10/06/2012) Florida Standards Assessments, ITN 2014-47 5 STATE OF FLORIDA DEPARTMENT OF EDUCATION INVITATION TO NEGOTIATE FLORIDA STANDARDS ASSESSMENTS BID NUMBER: ITN 2014-47 SECTION 1 - OVERVIEW SECTION 1 –INSTRUCTIONS 1.0 GENERAL INSTRUCTIONS TO RESPONDENT This section contains instructions explaining the solicitation process and the actions necessary to respond. General Instructions to Respondent (Form PUR 1001 – incorporated herein by reference) is a downloadable document which must be downloaded for review. This document need not be returned with the Respondent’s Reply. Form PUR 1001 may be accessed at http://dms.myflorida.com/business_operations/state_purchasing under “Documents, Forms, References and Resources”. In the event of any conflict between Form PUR 1001 and other instructions provided in this document, the additional instructions in this document shall take precedence over the Form PUR 1001 unless the conflicting term is required by any section of the Florida Statutes (F.S.), in which case the statutory requirements shall take precedence. It is entirely the respondent’s responsibility to examine this ITN, to confirm that the Department’s requirements are clearly stated, and to submit its reply in a timely, complete, and procedurally correct manner. The products and services described in this ITN will be procured in accordance with Chapter 287, Florida Statutes (F.S.), and Chapter 60A-1, Florida Administrative Code (FAC). THE DEPARTMENT HAS CHOSEN TO USE THE ITN FORMAT FOR THIS PROCUREMENT BECAUSE IT WANTS VENDORS TO PROPOSE THE BEST METHOD FOR ACHIEVING THE GOAL OF THIS ITN AND SOLVING THE PROBLEM STATED HEREIN. THEREFORE, ALTHOUGH THE ITN MAY USE MANDATORY WORDS LIKE “SHALL,” “WILL,” OR “MUST,” AND MAY DEFINE CERTAIN ITEMS AS REQUIREMENTS, THE DEPARTMENT RESERVES THE RIGHT, IN ITS DISCRETION, TO WAIVE ANY DEVIATIONS FROM THESE PROVISIONS AND RESOLVE ANY ISSUES IN THE NEGOTIATION PHASE. THEREFORE, THE DEPARTMENT RESERVES THE RIGHT TO REVIEW THE ENTIRE REPLY TO DETERMINE IF IT ACHIEVES A LEVEL OF COMPETENCY WORTHY OF FURTHER NEGOTIATIONS; REGARDLESS OF WHETHER INDIVIDUAL REQUIREMENTS HAVE BEEN ADDRESSED OR NOT. HOWEVER, VENDORS THAT FAIL TO PROVIDE SIGNIFICANT PORTIONS OF THE SOLUTION OR ADDRESS SIGNIFICANT PORTIONS OF THE PROCUREMENT MAY STILL BE DEEMED NONRESPONSIVE. IN ADDITION, THERE IS NO GUARANTEE THAT SUCH DEVIATIONS WILL BE DEEMED IN THE STATE’S BEST INTEREST OR ANY REPLY CONTAINING THOSE DEVIATIONS PARTICIPATE IN THE NEGOTIATIONS. USE OF THE TERMS “SHALL,” “WILL,” AND “MUST” INDICATE THE DEPARTMENT’S INITIAL VIEW OF THE VALUE OF SUCH ITEMS. VENDORS HAVE THE OPPORTUNITY TO SUGGEST ALTERNATIVES IN THE ITN PROCESS, BUT THERE IS NO GUARANTEE THAT THE DEPARTMENT WILL AGREE THE DEVIATIONS ARE IN ITS BEST INTEREST OR CREATE THE BEST VALUE FOR THE STATE. Florida Standards Assessments, ITN 2014-47 6 SECTION 2 – CONTRACT CONDITIONS 2.0 GENERAL CONTRACT CONDITIONS Standard terms and conditions that will apply to the contract which results from the solicitation event are provided in this section. General Contract Conditions (Form PUR 1000 – incorporated herein by reference) is a downloadable document which must be downloaded for review. This document need not be returned with the Respondent’s Reply. Form PUR 1000 may be accessed at http://dms.myflorida.com/business_operations/state_purchasing under “Documents, Forms, References and Resources”. In the event of any conflict between the PUR 1000 form and any other Special Conditions, the Special Conditions shall take precedence over the PUR 1000 form unless the conflicting term in the PUR form is required by any section of the F.S., in which case the statutory requirements shall take precedence. SECTION 3 - INTRODUCTION 3.0 INTENT The State of Florida Department of Education (hereinafter referred to as the "Department") is soliciting written replies from qualified vendors to establish a term contract of which the term is anticipated to begin upon execution of the contract and be effective for 36) months thereafter. Award(s) will be made to the responsible and responsive vendor that the Department determines will provide the best value to the state. The resulting contract(s) may be renewed for up to three additional 12 month terms, see Form PUR 1000 for renewal requirements. Although this ITN may result in the award of multiple contracts, the term “Contractor” is used throughout this ITN and refers to each contractor providing the required products and services for each grade and subject for which the respective contract is awarded. 3.1 PURPOSE The purpose of this Invitation to Negotiate (ITN) is to solicit replies for activities related to the implementation of computer-based and paper-based statewide summative assessments aligned to the Florida Standards for administrations beginning in spring 2015 and continuing through 2016-17 for the base contract with three oneyear optional renewals for the administrations in 2017-18, 2018-19, and 2019-20. Henceforth, in this document, this test shall be called the FSA (Florida Standards Assessment) In accordance with Governor Rick Scott’s Executive Order, dated September 23, 2013, this ITN is seeking Replies that: A. Provides timely and informative reports of results; B. Includes costs that are in line with current assessment costs; C. Ensures that testing time for students is not significantly different from current assessments; D. Provides for summative assessments to occur as close as possible to the end of the school year; E. Measures student mastery of the standards taught, including comparability to other states; F. Includes test quality metrics that are as rigorous as current assessments; G. Provides results that can be used in conjunction with Florida’s school accountability system; H. Requires technology parameters that are defined and can be supported, including appropriate accommodations for exceptional students. Florida Standards Assessments, ITN 2014-47 7 This ITN addresses four broad assessment components as described below. Respondents may submit bids on any individual assessment component (A, B, C or D), any group of assessment components, or to provide all assessment components (A, B, C and D): A. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 3-8 B. Computer-based and paper-based assessments in Mathematics in Grades 3-8 C. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 9-11 D. Computer-based and paper-based End–of-Course (EOC) assessments in Algebra 1, Geometry, and Algebra 2 A preferred schedule and plan for transition to computer-based testing is included in Appendix A. However, depending on the final award for the individual assessments, the Department may vary when tests will be administered as computer-based or paper-based. The final determination of which tests will be computerbased or paper-based is at the Department’s discretion. In addition, all Respondents agree, by submitting a reply, that any other state or school district in the United States may purchase the services provided in the resulting contract, upon the same terms and conditions, and price, as contained in the contract, or as permitted under the laws of that jurisdiction. 3.2 BACKGROUND The Florida Comprehensive Assessment Test® 2.0 (FCAT 2.0) and Florida End-of-Course (EOC) Assessments implement the requirements of Section 1008.22, Florida Statutes (F.S.), requiring assessments of the academic achievement of Florida’s public school students. The FCAT 2.0/EOC Assessment program is composed of state-developed criterion-referenced assessments. FCAT 2.0 Reading and Mathematics have been administered each spring since 2011 to assess students in selected grade levels. FCAT 2.0 Science was introduced in spring 2012, FCAT 2.0 Writing was introduced in spring 2013, and the Florida EOC Assessments have been phased in beginning in 2011. Prior to FCAT 2.0, FCAT assessments were administered each spring since 1998, and included Reading, Mathematics, Writing, and Science. The final administration of FCAT 2.0 Writing, FCAT 2.0 Reading, and FCAT 2.0 Mathematics will be in spring 2014. The final administration of the FCAT 2.0 Reading Retake for students with this graduation requirement will be in spring 2016, and the Florida Algebra 1 and Geometry EOC Assessments will be eliminated after the summer 2017 administration. (Administration of the FCAT 2.0 Reading Retake and the Florida EOC Assessments will be handled in another procurement.) The Sunshine State Standards, which specify challenging expectations for the educational achievement of Florida students, were revised and approved by the State Board of Education as the Next Generation Sunshine State Standards (NGSSS). The Reading/Language Arts and Mathematics revised standards were approved by the State Board in 2007. Science and Social Studies Standards were approved in February 2008. The NGSSS and associated course descriptions for the Florida End-of-Course Assessments are available at www.flstandards.org. In 2010, Florida adopted the Common Core State Standards for English Language Arts/Literacy and for Mathematics. As of the release of this ITN, these standards were scheduled to be reviewed further by Florida’s stakeholders. The degree to which the standards, and by extension the assessments, will be adjusted is to be determined. Florida plans for the Florida Standards Assessments to be operational beginning in 2014-15. 3.3 DEFINITIONS After the award, said Respondent will be referred to as the "Contractor". For the purpose of this document, the term "Respondent" means a potential Contractor acting on its own behalf and on behalf of those individuals, partnerships, firms, or corporations comprising the Respondent’s team. The term "Reply" means the complete response of the Respondent to the ITN, including properly completed forms and supporting documentation. The term “contract” refers to the agreement between the Department and the Contractor Florida Standards Assessments, ITN 2014-47 8 resulting from this ITN. “Best value” means the highest overall value to the state based on objective factors that include, but are not limited to, price, quality, design, and workmanship. A “responsive bid” is a Reply submitted by a responsive and responsible vendor which conforms in all material respects to the solicitation. However, for purposes of determining what is material, please refer to Section 1 of this ITN. Use of the terms “shall,” “will,” and “must,” or the designation of items as “requirements,” does not mandate that such items are material. The Department reserves the right to waive deviations, in its discretion, and resolve any issues in the negotiation phase. “Deliverable” means a tangible, specific, quantifiable and measurable event or item that must be produced to complete a project or part of a project directly related to the scope of services. Additional definitions/terms as follows: a. b. c. d. e. f. g. h. i. j. k. l. m. n. o. p. q. r. s. t. u. v. w. AL – Achievement Level Braille – Paper-based accommodation provided for eligible students with disabilities CBT – Computer-based test; computer-based testing Cohort – For the purposes of this ITN, refers to a group of students that enters grade 9 and is enrolled at the high school for a total of four years District Assessment Coordinator – Primary contact(s) in each school district with whom FDOE communicates regarding the administration of K-12 statewide assessments and who oversees the administration of statewide assessments at all participating schools in that district DAR – District Aggregated Results files used for reporting purposes DIF – Differential Item Functioning DSR – District Student Results file used for reporting purposes DSS – Developmental Scale Score EOC- End-of-Course – Refers to an assessment that is associated with a specific course FDOE (DOE) – Florida Department of Education FPO – Placeholder for a graphic used in a publication FSA – Florida Standards Assessment, the current name of the assessments to be administered as part of this ITN IEP – Individual Educational Plan for students with disabilities (ESE) IRT – Item Response Theory Large Print – Paper-based or computer-based accommodation provided for eligible students with disabilities; large print usually refers to at least 18 point font MS (Manuscript) – The initial production round when a document is composed and first reviewed MSID (Master School Identification File) – A file maintained by the Department and available online used to identify all active schools that will participate in statewide assessment administrations One-Item-Per-Page – Paper-based accommodation provided for eligible students with disabilities; only one item appears on each printed page P&D – Packaging and distribution; often used in reference to the specifications that describe a contractor’s processes of packaging and distributing all statewide assessment materials prior to each assessment administration PBT – Paper-based test; paper-based testing PreID (Preidentification) – Process for identifying students prior to an administration to print labels with all of the student’s information (PreID labels) or to create computer-based student records that will be used for testing PT (Program team) – Contractor staff members with program-wide expertise who are expected to assist in developing accurate instructions and providing specific information related to the contractor’s processes and procedures regarding Florida’s statewide assessments Florida Standards Assessments, ITN 2014-47 9 x. Rangefinding – Process required for scoring purposes by which student responses to performance tasks are used to illustrate score points on a rubric y. RD – Results Delivery (RD1, RD2, etc.) z. SAR – State Aggregated Results files used for reporting purposes aa. School Assessment Coordinator – A designated school-level assessment contact who oversees administration of statewide assessments at that school, training of test administrators and proctors, and preparing and returning test materials bb. Seals – Method of maintaining security of test content until a test session begins; paper-based tests may have one or more paper or plastic seals that students can easily break without tearing the test book, and computer-based tests may have an electronic means of sealing a test session (e.g., content is locked until a specific code is entered that allows access) cc. Section 504 Plan – Plan detailing accommodations for students with temporary disabilities (may not be ESE) dd. Section 508 Compliant (ADA Compliant) – Section 508 requires that electronic and information technologies which are procured or developed, be maintained by federal agencies and be accessible to people with disabilities, unless it poses an undue burden to do so. Accessibility standards needed to comply with this law were developed by the United States Architectural and Transportation Barriers Compliance Board. Section 508 of the Rehabilitation Act of 1973 is in the federal code of regulations (29 U.S.C. § 794(d). ee. SES (Scaling, Equating, and Scoring) – The set of processes used to ensure that assessment results are valid, reliable, and accurate ff. SS – Scale score gg. SSR – State Student Results files used to check data for reporting purposes hh. STMs – Sample Test Materials ii. Test Administrator – A certified educator who administers statewide assessments to students in a testing room; responsible for reading administration scripts to students and maintaining a secure testing environment jj. TAC (Technical Advisory Committee) – A panel of experts who review and respond to proposed modifications and processes for the assessment program kk. TAM – Test administration manual ll. TC (Technology Coordinator) – A designated coordinator at the school or district level responsible for preparing for computer-based testing and addressing technical issues during testing mm.TDC – Test Development Center, which is overseen by the FDOE and staffed with editors and Florida educators; TDC provides quality assurance for all test content and related products, whether customized or derived from existing products nn. TM – Test Materials Shipment (TM1, TM2, etc) 3.4 CRITICAL EVENT DATES These are tentative dates for critical events, and are subject to change. • October 25, 2013 – ITN Released • November 8, 2013 @ 2:00 PM, ET - Deadline for Technical Questions • November 15, 2013 - Department Response to Technical Questions received by deadline • December 12, 2013 @ 2:00 PM, ET - Technical & Price Replies are due Florida Standards Assessments, ITN 2014-47 10 • December 12, 2013 @ 2:30 PM - Technical Replies will be opened in Room 332 Turlington Building (Step 1) • January 10, 2014 - The Department will evaluate the Technical Replies in Room 1706 Turlington Building @ 9:00 AM (Step 2) Price Replies will be evaluated by the Bureau of Contracts, Grants and Procurement Management Services staff during the evaluation stage. • January 28, 2014 - The Negotiation Committee will start negotiations on or about (Step 4) • February 13, 2014 BAFO’s due on or about • February 25, 2014 – ITA Meeting on or about The Intent to Award will be posted for 72 hours upon completion of the negotiation meeting and a final decision by the Department. SECTION 4 – SPECIAL INSTRUCTIONS 4.0 PRE-SOLICITATION CONFERENCE: A Pre-Solicitation Conference will not be held. 4.1 SITE INSPECTION: 4.2 VISITOR’S PASS TO THE TURLINGTON BUILDING Each visitor to the Turlington Building is required to sign in and obtain a Visitor's Pass at the security desk in the main lobby. Please allow at least 15 minutes prior to Reply due time if hand-delivering the Reply to the Bureau of Contracts, Grants and Procurement Management Services. 4.3 REPLY QUESTIONS & ANSWERS Any technical questions arising from this ITN must be forwarded, in writing, to the purchasing agent identified below. The Department's written response to written inquiries submitted timely by Respondents will be posted on the Florida Vendor Bid System (VBS) at www.myflorida.com (click on Business, then click on Doing Business with the state, under Everything for Vendors and Customers, click on the Vendor Bid System, then Search Advertisement; select the Department of Education in the Agency drop down window and initiate search), under this Reply number. It is the responsibility of all potential Respondents to monitor this site for any changing information prior to submitting a Reply. A Site Inspection will not be held. Only timely received written inquiries will be “officially” addressed by the Department. WRITTEN QUESTIONS should be submitted to: Bureau of Contracts, Grants and Procurement Management Services, Fran Shewan, 325 West Gaines Street, 332 Turlington Building, Tallahassee, Florida 32399-0400, fshewan@fldoe.org (email preferred), or via Fax Number: (850) 245-0719. See SECTION 3.4 Critical Event Dates for question due dates. 4.4 PROCUREMENT PROTESTS / NOTICE OF RIGHTS Pursuant to F.S., Section 120.57(3) (b): Any person who is adversely affected by the agency decision or intended decision shall file with the agency a notice of protest in writing within 72 hours after the posting of the notice of decision or intended decision. With respect to a protest of the terms, conditions, and specifications contained in a solicitation, including any Florida Standards Assessments, ITN 2014-47 11 provisions governing the methods for ranking bids, proposals, or replies, awarding contracts, reserving rights of further negotiation, or modifying or amending any contract, the notice of protest shall be filed in writing within 72 hours after the posting of the solicitation. The formal written protest shall be filed within 10 days after the date the notice of protest is filed. Failure to file a notice of protest or failure to file a formal written protest shall constitute a waiver of proceedings under this chapter. The formal written protest shall state with particularity the facts and law upon which the protest is based. Saturdays, Sundays, and state holidays shall be excluded in the computation of the 72-hour time periods provided by this paragraph. Section 120.57(3) (a) provides: Failure to file a protest within the time prescribed in Section 120.57(3), F.S. or failure to post the bond or other security required by law within the time allowed for filing a bond shall constitute a waiver of proceedings under Chapter 120, F.S." Florida Administrative Code (F.A.C.) Rule 28-110.002(2) defines the term “decision or intended decision,” and includes the solicitation terms (and any addenda), the award of the contract, and a rejection of all bids. At the time of filing the Formal Written Protest the protestor must also file a Protest Bond payable to the Department in an amount equal to one percent (1%) of the estimated contract amount. F.S., Section 287.042(2) (c) and F.A.C. Rule 28-110.005 contain further terms relating to the Protest Bond, including how to determine the estimated contract amount. In lieu of a Protest Bond, the Department will accept cashier’s checks, official bank checks or money orders. The bond shall be conditioned upon the payment of all costs and charges that are adjudged against the protestor in the administrative hearing in which the action is brought and in any subsequent appellate court proceeding. The Notice of Protest, Formal Written Protest, and Protest Bond shall be filed with the issuing office as defined in SECTION 4.3 above. 4.5 ORAL INSTRUCTIONS / CHANGES TO THE INVITATION TO NEGOTIATE (ADDENDA) No negotiations, decisions, or actions will be initiated or executed by a Respondent as a result of any oral discussions with a state employee. Only those communications which are in writing from the Bureau of Contracts, Grants and Procurement Management Services, will be considered as a duly authorized expression on behalf of the Department until negotiation is awarded. Notices of changes (addenda) will be posted on the VBS, under this Reply number. It is the responsibility of all potential Respondents to monitor this site for any changing information prior to submitting a Reply. All addenda should be acknowledged by the Respondent’s signature and subsequent submission of addenda with Reply when so stated in the addenda. 4.6 MODIFICATIONS, RESUBMITTAL AND WITHDRAWAL Respondents may modify submitted replies at any time prior to the Reply due date. Requests for modification of a submitted Reply shall be in writing and should be signed by an authorized representative of the Respondent. Upon receipt and acceptance of such a request, the entire Reply will be returned to the Respondent and not considered unless resubmitted by the due date and time. Respondents may also send a change in a sealed envelope to be opened at the same time as the Reply. The ITN number, opening date and time should appear on the envelope of the modified Reply. Unless specifically requested by the Department, any amendments, revisions, or alterations to Replies will not be accepted after the closing for the receipt of Replies. 4.7 RESTRICTIONS ON COMMUNICATIONS WITH DEPARTMENT STAFF Respondents shall not communicate with any Department staff concerning this ITN except for the Department contact person identified in SECTION 4.3 REPLY QUESTIONS & ANSWERS of this ITN. Only those communications which are in writing from the Bureau of Contracts, Grants, and Procurement Management Florida Standards Assessments, ITN 2014-47 12 Services shall be considered as a duly authorized response on behalf of the Department. For violation of this provision, the Department reserves the right to reject a Respondent’s Reply. Respondents to this solicitation or persons acting on their behalf may not contact, between the release of the solicitation and the end of the 72-hour period following the agency posting the notice of intended award, excluding Saturdays, Sundays, and state holidays, any employee or officer of the executive or legislative branch concerning any aspect of this solicitation, except in writing to the procurement officer or as provided in the solicitation documents. Violation of this provision may be grounds for rejecting a Reply. 4.8 CONFIDENTIAL, PROPRIETARY, OR TRADE SECRET MATERIAL The Department takes its public records responsibilities as provided under Chapter 119, F.S., and Article I, Section 24 of the Florida Constitution, very seriously. If Respondent considers any portion of the documents, data or records submitted in response to this solicitation to be confidential, trade secret or otherwise not subject to disclosure pursuant to Chapter 119, F.S., the Florida Constitution or other authority, Respondent must clearly mark and identify in its Reply those portions which are confidential, trade secret or otherwise exempt. Respondent must also simultaneously provide the Department with a separate redacted copy of its Reply. This redacted copy shall contain the Department’s solicitation name, number, and the name of the Respondent on the cover, and shall be clearly titled “Redacted Copy.” The Redacted Copy shall be provided to the Department at the same time Respondent submits its Reply to the solicitation and must only exclude or obliterate those exact portions which are claimed confidential, proprietary, or trade secret. The Respondent should also provide two (2) electronic copies (compact disc (CD), flash drive, etc.) of their Redacted Copy. Respondent shall be responsible for defending its determination that the redacted portions of its Reply are confidential, trade secret or otherwise not subject to disclosure. Further, Respondent shall protect, defend, and indemnify the Department for any and all claims arising from or relating to Respondents determination that the redacted portions of its Reply are confidential, proprietary, trade secret or otherwise not subject to disclosure. If Respondent fails to submit a Redacted Copy with its Reply, the Department is authorized to produce the entire documents, data or records submitted by Respondent in answer to a public records request for these records. 4.9. POOR PERFORMANCE NOTICE The Respondent should provide for both the Respondent and its employees, subcontractors, and subcontractor employees, copies of any and all documents regarding complaints filed, investigations made, warning letters or inspection reports issued, any notice of breach, notice of default, termination notice, suspension notice, or any disciplinary action initiated or taken under any contract or job performance within the past seven (7) years. For each instance listed, provide a narrative summary of the contract’s purpose and scope of work, the Respondent’s performance, including the concerns of the project owner, and any major adverse findings. In addition, provide the contract or job number, the name of the owner, the term of the contract, the name, address, and telephone number of the owner’s contract manager. Please also include any relevant documentation evidencing the performance issues. The Department reserves the right to seek further information on this matter from the Respondent or to make inquiries with the project owner. The information obtained from this review may be reflected in the Respondent’s score or used to declare the Respondent not a responsible vendor. 4.10 WITHDRAWAL OF A REPLY A Respondent may withdraw a Reply by written notice to the Department on or before the deadline specified for the receipt of Replies in SECTION 3.4 CRITICAL EVENT DATES of this ITN. Such written notice is to be Florida Standards Assessments, ITN 2014-47 13 submitted to the Issuing Office at the address specified in SECTION 4.3 REPLY QUESTIONS AND ANSWERS of this ITN. 4.11 DISCLOSURE OF REPLY CONTENTS All documentation produced as part of this solicitation shall become the exclusive property of the state and may not be removed by the Respondent or its agents. All Replies shall become the property of the state and shall not be returned to Respondent. The Department shall have the right to use any or all ideas or adaptations of the ideas presented in any Reply. Selection or rejection of a Reply shall not affect this right. 4.12 AWARD As in the best interest of the State, the right is reserved to award based on all or none, groups of services, or any combination thereof, to a responsive, responsible Respondent. As in the best interest of the state, the right is reserved to reject any and/or all Replies or to waive any minor irregularity in replies received. In addition, the Department reserves the right, in its discretion, to correct deviations during the negotiation phase. Conditions which may cause rejection of Replies include, without limitation, evidence of collusion among Respondents, obvious lack of experience or expertise to perform the required work, failure to perform, or meet financial obligations on previous contracts. SECTION 5 – SPECIAL CONDITIONS 5.0 AUTHORIZED TO DO BUSINESS IN THE STATE OF FLORIDA Foreign corporations and foreign limited partnerships should be authorized to do business in the state of Florida. Domestic corporations should be active and in good standing in the State of Florida. Such authorization and status should be obtained by the Reply due date and time, but in any case, must be obtained prior to contract execution. For authorization, contact: Florida Department of State Tallahassee, Florida 32399 (850) 245-6053 5.1 LICENSED TO CONDUCT SERVICES IN THE STATE OF FLORIDA If the services being provided require that individuals be licensed by the Florida Department of Business and Professional Regulation or any other state or federal agency, such licenses should be obtained by the Reply due date and time, but in any case, must be obtained prior contract execution. For state licensing, contact: Florida Department of Business and Professional Regulation Tallahassee, Florida 32399-0797 (850) 487-9501 5.2 OTHER CONDITIONS Other conditions which may cause rejection of replies include, without limitation, evidence of collusion among Respondents, obvious lack of experience or expertise to perform the required work, failure to perform or meet financial obligations on previous contracts. 5.3 IDENTICAL EVALUATION OF REPLIES Whenever two (2) or more replies which are equal with respect to price, quality, and service are received, the Department will determine the order of award using the criteria established in 60A-1.011, F.A.C. The "DrugFree Workplace Program Certification" can be found as Attachment 2. Florida Standards Assessments, ITN 2014-47 14 5.4 DISCLOSURE STATEMENT The Disclosure Statement Form (Attachment 4) should be signed and submitted with the Reply. 5.5 SUB-CONTRACTING This contract or any portion thereof shall not be sub-contracted except as permitted herein, with the prior written approval of the Department. No sub-contract shall, under any circumstances, relieve the Contractor of its liability and obligation under this contract; and despite any such sub-contracting the Department shall deal through the Contractor, which shall retain the legal responsibility for performing the Contractor obligations. The Prime Contractor shall report all Minority Subcontractors, identifying the Name, Address, Type of Certification and Dollar Amount on the Utilization Summary form, attached as Attachment 5. A list of subcontractors shall be provided to the Department’s contract manager upon execution of the Contract. The Prime Contractor shall submit the Utilization Summary form with each invoice submitted for payment. The form must be submitted with all invoices, regardless if funds have not been spent with a Minority Subcontractor for the period covered by the invoice. The Office of Supplier Diversity, Florida Department of Management Services, will assist in furnishing names of qualified minorities. The Office of Supplier Diversity can be reached at (850) 487-0915; the Internet Web address is http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd/. 5.6 CONTRACTUAL OBLIGATIONS The Department’s Contract Standard Terms and Conditions are incorporated in this ITN as Attachment 6 and will govern the relationship between the Department and the Contractor. A Reply submitted by the successful Respondent(s) shall be incorporated into the final contract(s). The terms and conditions of this ITN shall control notwithstanding any statement to the contrary by the Respondent, unless such terms and conditions are modified by the Department during the negotiation phase. 5.7 METHOD OF PAYMENT Compensation and payment will be made in accordance with the terms and conditions of the contract. 5.8 CONVICTED VENDOR LIST A company placed on the Convicted Vendor List may not submit a Reply or be awarded a contract to provide any goods or services pursuant to Rule 60A-1.006 F.A.C. The “Convicted Vendor List” is published at: http://www.dms.myflorida.com/business_operations/state_purchasing/vendor_information/convicted_suspen ded_discriminatory_complaints_vendor_lists/convicted_vendor_list. 5.9 DIVERSITY IN CONTRACTING The State of Florida is committed to supporting its diverse business industry and population through ensuring participation by minority-, women-, and service-disabled veteran business enterprises in the economic life of the state. The state of Florida Mentor Protégé Program connects minority-, women-, and service-disabled veteran business enterprises with private corporations for business development mentoring. We strongly encourage firms doing business with the State of Florida to consider this initiative. For more information on the Mentor Protégé Program, please contact the Office of Supplier Diversity at (850) 487-0915. The State is dedicated to fostering the continued development and economic growth of small, minority-, women-, and service-disabled veteran business enterprises. Participation by a diverse group of Vendors doing business with the state is central to this effort. To this end, it is vital that small, minority-, women-, and service-disabled veteran business enterprises participate in the State’s procurement process as both Contractors and sub- contractors in this solicitation. Small, minority-, women-, and service-disabled veteran business enterprises are strongly encouraged to contribute to this solicitation. Florida Standards Assessments, ITN 2014-47 15 Information on Certified Minority Business Enterprises (CMBE) and Certified Service-Disabled Veteran Business Enterprises (CSDVBE) is available from the Office of Supplier Diversity at: http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd/. 5.10 COSTS INCURRED IN RESPONDING This ITN does not commit the Department or any other public agency to pay any costs incurred by the Respondent in the submission of a Reply or to make necessary studies or designs for the preparation thereof, nor to procure or contract for any articles or services. 5.11 SUBMISSION OF REPLIES BY SUBSIDIARIES OR AFFILIATES A Respondent, its subsidiaries, affiliates, or related entities is limited to one (1) Reply. Submission of more than one (1) Reply per activity by a Respondent may cause the rejection of all Replies submitted by the Respondent. In the alternative, the Department my decide, in its sole discretion, which Reply to evaluate and consider. A subsidiary or affiliate of a prime Respondent may also be included as a subcontractor in another Respondent’s Reply. 5.12 PROHIBITION OF GRATUITIES By submission of a Reply, the Respondent certifies that no elected or appointed official or employee of the state of Florida has or will benefit financially or materially from this procurement. Any contract arising from this procurement may be terminated by the Department if it is determined that gratuities of any kind were either offered to or received by any of the aforementioned officials or employees from the Respondent or its agents or employees. 5.13 INDEPENDENT PRICE DETERMINATION A Respondent shall not collude, consult, communicate, or agree with any other Respondent regarding this procurement as to any matter relating to the Respondent’s Reply. 5.14 FINANCIAL CONSEQUENCES The Contractor shall submit to the Department for payment the above deliverables and those proposed and accepted by the Department that are satisfactorily completed in accordance with the terms and conditions set forth in SECTION 7.0 SCOPE OF SERVICES and SECTION XXXXX DELIVERABLES. Payment shall be delayed for any deliverables that are not satisfactorily completed until all deficiencies are corrected and the services have been accepted by the Department. In addition, delays in providing completed deliverables will subject the Contractor to damages. 5.15 PERFORMANCE BOND The Contractor shall supply to the Department a Performance Bond in the amount of ten percent (10%) of the amount of the annual award. The surety shall be in a form acceptable to the Florida Department of Education, such as a bond, cashier’s check, certified check or money order. A Surety must be authorized to do business in the state of Florida. The Performance Bond shall be executed and furnished to the Department within ten (10) calendar days prior to the Contractor beginning work under the contract. The performance bond must state that it includes coverage of liquidated damages assessed against the contractor. A performance bond is not required from Florida State Universities. Cap: For each calendar, contract or fiscal year which corresponds to the period of the performance bond specified, the cumulative total of liquidated damages for which the Contractor(s) may be liable shall not exceed, annually, ten percent (10%) of the total contract amount at the time of assessment of liquidated damages. Florida Standards Assessments, ITN 2014-47 16 5.16 PARTICIPATION IN FUTURE STAGES OF THIS PROJECT As stated in Chapter 287.057 F.S. 17(a)c A person who receives a contract that has not been procured pursuant to subsections (1) through (3) to perform a feasibility study of the potential implementation of a subsequent contract, who participates in the drafting of a solicitation or who develops a program for future implementation, is not eligible to contract with the agency for any other contracts dealing with that specific subject matter, and any firm in which such person has any interest is not eligible to receive such contract. However, this prohibition does not prevent a vendor who responds to a request for information from being eligible to contract with an agency. 5.17 ACCESSIBLE ELECTRONIC INFORMATION TECHNOLOGY Respondents submitting Replies to this solicitation must provide electronic and information technology resources in complete compliance with the accessibility standards required by Section 282.601-282.606, F.S., and Rule 608.002, F.A.C. These standards establish a minimum level of accessibility. 5.18 SCRUTINIZED COMPANIES LISTS (as applicable) Section 287.135, Florida Statutes, requires that at the time a company submits a bid or proposal for a contract for goods or services of $1 million or more, the company must certify that the company is not on either the Scrutinized Companies with Activities in Sudan List or the Scrutinized Companies with Activities in the Iran Petroleum Energy Sector List. Both lists are created pursuant to section 215.473, Florida Statutes. Replies of $1 million or more should include the attached Scrutinized Companies Lists Form (Attachment 6) to certify the respondent is not on either of those lists. The Form should be submitted with the Technical Reply. 5.19 LIQUIDATED DAMAGES Accurate and timely delivery is imperative and, as a result, the contract will include the following provisions for the completion of the Work Tasks/Activities and Interim Deliverables and Final Deliverables identified in this ITN. The Contractor(s)‘s failure to complete Work Tasks/Activities and Interim Deliverables and Final Deliverables both correctly and on time will result in substantial injury to the Department and the State of Florida, but the amount of damages resulting from such injury cannot be calculated with certainty. Each such failure to complete Work Tasks/Activities and Interim Deliverables and Final Deliverables both correctly and on time is hereinafter referred to as a default. Defaults shall be deemed corrected on the date that the Work Tasks/Activities and Interim Deliverables and Final Deliverables have been correctly completed. For each default, the Contractor(s) shall be liable to the Department for liquidated damages and not as a penalty, as set forth below: a. Tasks/Activities and Interim Deliverables: For each default on performance and completion of Work Tasks/Activities and Interim Deliverables (see SECTION 7.8), it is agreed that for each and every day work remains uncompleted beyond the time set for its completion the Contractor(s) shall be liable to the Department in a specified amount of three thousand dollars ($5,000.00) per business day as liquidated damages for such delay, and not as a penalty, from the date of notice by the Department until the default has been corrected. THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK Florida Standards Assessments, ITN 2014-47 17 b. Final Deliverables: For each default on completion of Final Deliverables (see SECTION 7.8), the Contractor(s) shall be liable to the Department in a specified amount per calendar day from the date of notice by the Department until the default has been corrected. The specified amount of damages is defined below: Schedule of Liquidated Damages For Final Deliverables Number of business days Liquidated Damages elapsed after the default First $5,000.00 Second $15,000.00 Third $30,000.00 Fourth $45,000.00 Fifth $60,000.00 Sixth $75,000.00 Seventh/ subsequent days $125,000.00 per day c. Cap: For each calendar, contract, or fiscal year which corresponds to the period of the performance bond specified in ITN Section 5.15, the cumulative total of liquidated damages for which the contractor may be liable shall not exceed, annually, ten percent (10%) of the total contract amount at the time of assessment of liquidated damages. d. Collection: At its option, the Department shall collect liquidated damages by making claims against the performance bond, from time to time, until the bond has been exhausted or by deducting the liquidated damages from contract payments to the contractor. In the alternative, the Contractor may make payment to the Department in the amount of such liquidated damages. e. Deadline Adjustments: (1) In the event the contractor wishes to propose a change of deadline for a Critical Work Task, the contractor may propose the change to the Department, in writing. The Department has no obligation to agree to the change, but shall notify the contractor, in writing, as to whether or not it agrees to the proposed change. If the Department agrees, the modified deadline shall become the effective deadline for purposes of assessing liquidated damages. (2) In the event the Department wishes to change a deadline for an Extremely Critical Work Task to a date which shortens the contractor’s time for meeting the deadline, it shall notify the contractor of the new deadline in writing in the form of a contract amendment. If the contractor determines that by exercise of every reasonable effort, the contractor will be unable to meet the new deadline, the contractor shall offer a proposed shortened deadline which the contractor can meet by the exercise of every reasonable effort. (3) In the alternative, the contractor may propose a contract amendment in which the additional personnel hours needed to achieve the shortened deadline are compensated by a price adjustment. (4) If the parties are unable to reach agreement, the Department shall determine the deadline. However, the issue of whether liquidated damages apply to the contractor’s failure to meet the shortened deadline shall be reached as follows. The parties shall submit to binding arbitration the issue of what date (earlier than the original deadline) the contractor could have completed the Extremely Critical Work Task on time and correctly by exercising every reasonable effort. The date determined through arbitration shall be the deadline for purposes of assessing liquidated damages. Arbitration shall be conducted pursuant to the procedures of the American Arbitration Association, subject to the Florida Arbitration Code, Chapter 682, F.S. Florida Standards Assessments, ITN 2014-47 18 f. Waiver: The Department, in its sole discretion, may waive the imposition of liquidated damages or a portion thereof in a given instance. Such waiver, in any instance, shall not constitute a waiver in any future instance, nor establish any right on behalf of the contractor to a waiver. g. Department Approval: In situations where in the contractor must obtain the Department’s approval of an activity or product before the contractor can complete a Critical Work Task, the contractor shall be responsible for providing a reasonable time for the Department to complete its review and for the contractor to correct any deficiencies. In the event the Department’s review is not conducted within a reasonable time, as described in Section 7.7.4, that shall be grounds for the contractor to propose a deadline adjustment pursuant to Subsection 5.19(e)(1), above. The parties’ contract managers may agree, in writing, in advance, as to the reasonable time for the Department’s review of a specific activity or product. SECTION 6 – SPECIAL INSTRUCTIONS – REPLY FORMAT & CONTENT 6.0 REPLY SUBMISSION Both Technical and Price Replies will be received as specified in SECTION 3.4 CRITICAL EVENT DATES. All Replies and associated forms should be signed and dated in ink by a duly authorized representative of the Respondent. The overall Reply should be written in a concise manner, which is conducive to effective evaluation and selection. All Replies and related documents submitted in response to this ITN shall become the property of the state. 6.1 MAIL OR DELIVER REPLIES TO: (Do Not Fax or E-Mail) Florida Department of Education Bureau of Contracts, Grants and Procurement Management Services Attn: Fran Shewan 325 West Gaines Street 332 Turlington Building Tallahassee, Florida 32399-0400 6.2 REPLY FORMAT INSTRUCTIONS This section contains instructions that describe the expected format for the Reply. All Replies submitted should contain two parts and be marked as follows: PART I TECHNICAL REPLY NUMBER ITN 2014-47 (One Separately Sealed Package for Technical) PART II PRICE REPLY NUMBER ITN 2014-47 (One Separately Sealed Package for Prices) THE SEPARATELY SEALED PACKAGES MAY BE MAILED TOGETHER IN ONE ENVELOPE OR BOX. 6.2.1 Preliminary Administrative Review The absence of any of these documents may result in a determination that the Reply is nonresponsive and the Reply shall not be evaluated. The Reply forms furnished must be used when submitting the Reply. Forms are to be filled out in ink or typewritten. Administrative Review information shall be included in the Technical Reply portion (Part I) of the Reply and consist of the following:   All Addenda DUNS Number Florida Standards Assessments, ITN 2014-47 19    Disclosure Statement Form (Attachment 3) Reference Form (Attachment 4) – Provide at least three (3) references, which demonstrate efforts comparable to the one described in this ITN. The Department reserves the right to contact the references regarding the services provided. Any information provided will be subject to the requirements of the Florida Public Records Law. Transmittal Letter – Provide a Transmittal Letter (on Company Letterhead) that contains the following:  a statement confirming that the person signing the Reply is authorized to represent the Respondent and bind the Respondent relative to all matters contained in the Respondent's Reply  the company’s federal tax identification number  the company’s DUNS number  a statement confirming that the Respondent has read, understands, comply and agrees to all provisions of this ITN, unless changed during the negotiation process. By submitting a Reply, the Respondent agrees to the terms and conditions of the ITN notwithstanding any statement in the Reply to the contrary. However, the Department will attempt to resolve any disagreements in the negotiation sessions.    a statement confirming that the Respondent is authorized to conduct business in Florida in accordance with the provisions of Chapter 607, F.S. In lieu of such statement, the Respondent alternatively should confirm that authorization to do business in Florida will be secured prior to the award of the contract. a statement confirming that the Respondent is registered on the MyFloridaMarketPlace website in accordance with the provisions by the State of Florida. In lieu of such statement, the Respondent should alternatively confirm that registration authorization will be completed prior to the award of the contract. a statement confirming that the Respondent has electronically registered a valid W-9 with the Department of Financial Services (DFS). DFS is ready to assist any vendors with questions, and vendors must submit their W-9 forms electronically at https://flvendor.myfloridacfo.com. Contact the DFS Customer Service Desk at (850) 413-5519 or FLW9@myfloridacfo.com with any questions. In lieu of such statement, the Respondent should alternatively confirm that registration will be completed prior to the award of the contract. 6.2.2 Technical Reply (Part I) (20 hard copies) (Do not include price information in Part I) The Respondent should submit one (1) original in hard copy, nineteen (19) hard copies and twenty (20) in electronic format (compact disc (CD), flash drive, etc.) copies in Microsoft Word 5.0 or higher, or Adobe Acrobat of the Reply which is to be divided into the sections described below. The Department expects all Replies to be in this format. Failure of the Respondent to follow this outline may result in the rejection of the Reply, or result in a lower evaluation score. Any rejection of a Reply, or other adverse affect on a Respondent’s Reply, due to its failure to properly organize its Reply shall be at the sole risk of the Respondent. The Technical Reply should be submitted in a separate sealed package marked "TECHNICAL REPLY FOR ITN 2014-47". 1. EXECUTIVE SUMMARY The Executive Summary should be written in non-technical language to summarize the Respondent’s overall capabilities and approaches for accomplishing the services specified herein. Evidence should include, but not be limited to, respondent’s mission, date of founding, size, and experience. This section should also demonstrate the Respondent’s clear understanding of the Department’s goal to securely and reliably develop, administer, score, and report Florida’s assessments in an efficient and cost-effective manner. The Executive Summary should be limited to no more than six (6) pages. Florida Standards Assessments, ITN 2014-47 20 2. QUALIFICATIONS AND EXPERIENCE The respondent should provide a description of qualifications and prior experience performing tasks similar to those required in this ITN. The discussion should include a description of the respondent’s background and relevant experience that qualifies it to provide the products and services required by this ITN. The respondent should document the contracted services for previous assessment projects similar to the one described in this ITN and any other projects the respondent believes will document its corporate capability. For each, the documentation should include a description of the services and products delivered; the contract period; and the name, address, and telephone number of a contact person for each of the contracting agencies. If subcontractors are to be used for any portion of the work activities, the respondent should name the subcontractor(s) and document the experience and qualifications of the subcontractor(s) in performing tasks identical or similar to those they will be asked to do. A separate chart in the proposal may be used to identify all of the subcontractors proposed to be involved in the project and the services they are expected to provide. The respondent also should provide examples of materials that demonstrate the quality of the work done by the respondent on similar projects. The proposal should include information sufficiently descriptive to permit the proposal evaluation committee to consider the experience and capability of the prime contractor as well as the subcontractors. As part of the proposal evaluation, the proposal evaluation committee will contact one or more of the agencies for whom the respondent previously had been engaged to obtain information and recommendations. Committee members may also depend upon and use their own personal knowledge of the respondent, the subcontractors, and the proposed personnel as they evaluate each proposal. 3. TECHNICAL PLAN The technical plan should provide a description of the respondent’s proposal to meet the requirements of this ITN. This discussion should encompass all of the requirements of Section 7.1.0 through Section 7.6.3.6.3 and associated deliverables of this ITN and should be organized and numbered in accordance with the organization and numbering system used in Sections Section 7.1.0 through Section 7.6.3.6.3. The technical plan should describe the methodologies, quality management standards and best practices that will be employed in managing and completing the requirements. The descriptions should be thorough, clear, yet as concise as possible. The Respondent should also identify where significant difficulties may be anticipated and describe any specific techniques proposed to be used to resolve the difficulties. TECHNICAL APPROACH The Respondent should explain the approach, capabilities, and means to be used in accomplishing the tasks in the Scope of Services, and where significant development difficulties may be anticipated and resolved. Any specific techniques to be used should also be addressed. TECHNICAL APPROACH ORGANIZATION OF THE INVITATION TO NEGOTIATE Section 7.1 of the ITN states the purpose of the ITN; describes the current Florida statewide assessment program and modifications to the program that will be implemented under the ITN; and provides information for preparing replies. Sections 7.2 through 7.7 describe the scope of work for the years covered by this ITN. References are made in Sections 7.2 through 7.7 to the appendices, which provide detailed information about printed products, reports, shipments, and other elements of Florida’s assessment program. Each section includes a description of products and services to be provided. Florida Standards Assessments, ITN 2014-47 21 Section 7.2.0: Program Descriptions. An overview of the standards-based assessments requiring services described in this ITN. Section 7.3.0: Test Development. A description of the test development tasks from item writing through the production of final test forms. Section 7.4.0: Test Administration. A description of the test administration tasks, including the delivery and return of test materials, the production of test administration ancillary materials, providing customer service and technical support, needs for the online management system, and administering computer-based assessments. Section 7.5.0: Scanning, Scoring, and Reporting. A description of the scanning, scoring, and reporting tasks from preparing school and student information through the reporting of results. The psychometric aspects of scoring are covered in Section 7.6.0. Section 7.6.0: Scaling, Equating, Scoring, and Special Psychometric Studies. A description of the scaling, equating, scoring, and special psychometric studies and tasks. Section 7.7.0: Program Management. A description of how the work of the contract should be managed, including meetings, staff, and reimbursable funding categories. SYSTEM DEMONSTRATION (ELECTRONIC DEMONSTRATION) The Respondent should submit two (2) copies with the Technical Reply, in electronic format (compact disc (CD), DVD, flash drive, etc.) a demonstration (demo) of the proposed system. The system demo provided should not exceed 25 slides or 30 minutes of presentation. Do not include pricing information with the Technical Reply or the demo. Any inclusion of prices/discounts in the Technical Reply or demo may be grounds for deeming the Respondent’s Reply nonresponsive. After the Evaluation Phase is completed, the Department will notify the Respondents invited to participate in Negotiations. Only those Respondents entering into Negotiations will be allowed and required to present an on-site oral presentation of the proposed operational system. The Respondent’s on-site presentation will be conducted during the Negotiation meeting. The actual date, time and duration of Negotiation meetings/presentations are to be determined. 4. RESPONDENT'S MANAGEMENT PLAN The management plan should describe the organization and document its capacity to provide the products and services required by this ITN. This discussion should specifically encompass the requirements of Section 7.7.0 through Section 7.7.10 and associated deliverables of this ITN and should be organized and numbered in accordance with the organization and numbering system used in Section 7.7.0 through Section 7.7.10. The descriptions should be complete, clear, and concise. The management plan should identify by name the managers and professional personnel responsible for this project. Charts showing the organizational structure and the incorporation of proposed project staff into the structure should be included, as well as the amount of time in FTE person-days that each person will devote to the major tasks and activities. The Respondent should include a description of the methodology to be used to manage the project work, including control of costs, assuring quality, identifying and minimizing risk, and maintaining schedules; Vitae should be included as an appendix to the proposal and should include relevant educational background and professional experience. Personnel assigned by the contractor should demonstrate qualifications and competencies of the position to which assigned. It is assumed that the contractor may use outside printers for some materials, such as preprinted forms. Printers will be documented as subcontractors, and the management plan should identify the proportion of Florida Standards Assessments, ITN 2014-47 22 materials to be printed by the contractor and by outside vendors. Procedures for quality control and security during printing should be described. 6.2.3 Price Reply (Part II) (20 hard copies) The Respondent should submit one (1) original hard copy, nineteen (19) hard copies and twenty (20) electronic format (compact disc (CD), flash drive, etc.) copies in Microsoft Excel 5.0 or higher. The Respondent’s price information should be submitted on the form provided in this ITN. Prices submitted in the Price Reply form, Attachment 1, will be used to evaluate the price component of the evaluation. Payment for services provided under the contract will be made based on the deliverable schedule submitted, Section 7.8, The Price Reply and deliverable schedule should be submitted together in a separate sealed package marked "PRICE REPLY FOR ITN 2014-47". 6.2.4 Presenting the Reply The Reply should be limited to a page size of eight and one-half by eleven inches (8½" x 11"). Type size should not be less than a 12 point font. The Reply shall contain a table of contents, be typed single-spaced and have separate parts, each clearly labeled including page numbers. The information to be addressed in each part is described in the above sections. The absence of information or the organization of information in a manner inconsistent with the terms of this ITN may result in the rejection of the Reply, or a reduced evaluation score. Bindings and covers will be at the Respondent's discretion; however, elaborate notebooks/hard back binders are discouraged. Unnecessarily elaborate special brochures, art work, expensive paper and expensive visual and other presentation aids are neither necessary nor desired. All Respondent materials should be packaged so that each box of materials shipped to the Department does not exceed 25 pounds. SECTION 7 – SCOPE OF SERVICES 7.0 SCOPE OF SERVICES 7.1.0 Introduction This ITN addresses four broad assessment components as described below. Respondents may submit bids on any individual assessment component (A, B, C or D), any group of assessment components, or to provide all assessment components (A, B, C and D): A. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 3-8 B. Computer-based and paper-based assessments in Mathematics in Grades 3-8 C. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 9-11 D. Computer-based and paper-based End–of-Course (EOC) assessments in Algebra 1, Geometry, and Algebra 2 The Department will continue to phase in computer-based assessments in Grades 3-8, transitioning completely to computer-based by 2017-2018 according to the preferred schedule in Appendix A. However, depending on the final award for the individual assessments, the Department may vary when tests will be administered as computer-based or paper-based. The final determination of which tests will be computerbased or paper-based is at the Department’s discretion. This ITN will specify a complete package of products and services so that the FSA will be developed, administered, scored, and reported separately from other assessments aligned to Florida’s Next Generation Sunshine State Standards under a contract currently held by NCS Pearson. This ITN does not include the Florida Standards Assessments, ITN 2014-47 23 Science, Social Studies, FCAT 2.0 Reading Retake, and Florida EOC Assessments aligned to the Next Generation Sunshine State Standards that will be covered in a separate procurement. Although this ITN may result in the award of multiple contracts, the term “Contractor” is used throughout and refers to each contractor providing the required products and services for each grade and subject for which the contract is awarded. 7.1.1 Notice to Respondents 1. This ITN provides for implementation of Florida’s standards-based assessment program. The contract will include development, psychometric services, administration, scoring, and reporting of Florida Standards Assessments in English Language Arts/Literacy and Mathematics for the test administration cycles for 2014-2015 through 2016-17 with possible extension of services in three one-year renewal periods, 201718 through 2019-2020. Replies will be evaluated on the basis of the respondent’s qualifications and experience and the technical and cost replies for the tasks described herein for the entire span of the ensuing contract. 2. This ITN defines the requirements for implementing Florida’s standards-based assessment program. The ITN and the selected contractor’s reply, together with clarifying documents, define the work to be conducted under contract. These documents will be incorporated into the contract resulting from the state standards-based assessment program project award. Because Florida’s standards-based assessment program is technical and complex, it is possible that a responsive reply may not totally or clearly reflect ITN requirements in all details. If the reply of a respondent selected as a result of the bidding process is inconsistent with the ITN, the requirements of the ITN will prevail; the selected contractor will be expected to perform all ITN requirements without an increase in cost above the proposed cost. Furthermore, by submitting a bid the respondent is agreeing to be bound by the terms of the ITN notwithstanding any language in the reply to the contrary. Any discrepancies between the terms of the ITN and the reply will be resolved in the negotiation process at the Department’s discretion, and when it is in the state’s best interest. 3. The Department reserves the right to substitute or modify tasks on a generally equal basis. Substitutions or modifications will be made prior to the selected contractor's beginning significant efforts to complete the task. All design changes, substitutions, or modifications will be given to the contractor in writing. It is not the intent of this provision to increase the scope of work of the contract but to protect the Department as design changes to improve the assessment become known during the implementation of development and administration activities. The Department also reserves the right to amend the contract resulting from this bid competition to add, modify, or delete designated tasks, products, and/or services related to the core activities of this project. 4. In developing the solution pursuant to this procurement, a respondent may propose to provide a custom built solution, a solution comprised of pre-existing, proprietary materials or products that are the intellectual property of the respondent or third parties, or a combination thereof. A respondent may propose that all intellectual property developed be owned solely by the Department, owned jointly by the Department and the respondent, or, at a minimum, that the Department be granted a perpetual and non-exclusive license for the use, reproduction and modification of the solution for use in Florida. The Department reserves the right to request alternative intellectual property arrangements or combinations of such arrangements during negotiations with respondents, in order to obtain a reply that provides the best value to the state. All permissions or licenses required for the use of materials or products that are the intellectual property of a third party must be obtained by the respondent. All materials, products, and licenses must be transferred to the Department at the end of the contract period such that the Department can maintain and sustain the solution beyond the end of the contract period. 5. The selected contractor will work closely with the Department and with groups of Florida educators to complete all tasks. All procedures, processes, and products used by the contractor to complete contract work must be approved by the Department. Throughout the contract period, the contractor will confer with Florida Standards Assessments, ITN 2014-47 24 the Department on a continuing and consistent basis and will be involved in frequent face-to-face meetings with the Department, as necessary. 6. Several reimbursable funding categories identified in Section 7.7.9 are set aside for specific functions during the course of the project. Expenditures in these categories will be reconciled on a quarterly basis. 7. The dates for reporting assessment results necessitate a restricted time period to complete the work of scoring activities. Respondents are expected to propose efficient and innovative procedures to meet these required reporting dates and are expected to demonstrate in their replies that they have the capability and are willing to commit the human and technological resources needed to complete the scanning, student response file processing, scoring, and reporting tasks in the required time period. 8. The fundamental purpose of Florida’s standards-based assessment program is to provide accurate information on student achievement. The contractor will utilize every means required to ensure that information created by the project is accurate. The selected contractor will be responsible for correcting at the contractor’s expense any errors arising from activities that are the responsibility of the contractor. This may involve activities such as conducting analyses to identify the cause and extent of errors; reprogramming or reproducing products or other materials; replacing data files; reproducing reports; and shipping replacement products or reports to the Department or districts using expedited shipping services. 9. News releases or public announcements pertaining to this ITN, a contract award resulting from the ITN, or work completed under a contract will not be made without prior written approval from the Department. 7.1.2 Project Schedule The deliverables schedule presents a list of major project activities and the dates for completion where these can be determined. Many of the deliverable dates are contingent upon finalized administration dates and will be determined during the negotiation process. Appendix B indicates critical and extremely critical tasks and provides general guidelines for the timeframes for completion, but these are also contingent on finalized administration dates. The dates provided in 7.2.2 are subject to change during the negotiation phase. Respondents should provide a detailed schedule that incorporates the activities included in this ITN for the entire life of the contract. The schedule will indicate essential steps for the completion of project tasks, initiation and completion dates, and task responsibility. Tasks and responsibilities will include an explanation of work effort required and full-time equivalent staff (FTE) committed to this work. The reply should also provide total annual FTE and identify individuals and/or staff roles that will work full-time on this project. Respondents should be aware that the schedules for each project year overlap. To accurately estimate required annual resources, schedules for preceding and succeeding years need to be considered. Following the contract award, the schedule will be further developed into a working document that will be updated weekly, indicate all task completion dates, and be revised as needed with Department approval (See Section 7.7.6). A draft work schedule, based on a sample provided by the Department, will be prepared by the selected contractor for submission to the Department one week prior to the initial planning meeting, which will be held no later than one month after the execution of a contract for the project. The schedule will be submitted in hard copy and electronic format using Department-approved spreadsheet software that permits sorting of the file by start or end date and major task and subtask groupings. 7.2.0 Program Descriptions The selected contractor will provide the Department with a criterion-referenced achievement test, aligned to and measuring mastery of the Florida Standards in English Language Arts/Literacy (ELA/L) and Mathematics proficiency for administration to Florida public school students. For purposes of responses to this ITN, the Florida Standards are to be considered those adopted by the Florida State Board of Education in July, 2010. Any changes to the standards following additional Florida public review scheduled for fall 2013, and resulting changes to the assessment will be addressed through the amendment of the contract that may be awarded as a result of this ITN. Florida Standards Assessments, ITN 2014-47 25 The primary purpose of the assessment is to provide student academic achievement and learning gains data to students, parents, teachers, school administrators, and school district staff. This data is to be used by districts to improve instruction; by students, parents, and teachers to guide learning objectives; by education researchers to assess data; and by the public to assess the cost benefit of the expenditure of taxpayer dollars. The assessment must be designed to: 1. Assess the achievement level and annual learning gains of each student in ELA/L and Mathematics. 2. Provide data for making decisions regarding school accountability, recognition, and improvement of operations and management, including schools operating for the purpose of providing educational services to youth in Department of Juvenile Justice programs. 3. Identify the educational strengths and needs of students and the readiness of students to be promoted to the next grade level, to graduate from high school, and to succeed in college and in careers. 4. Assess how well educational goals and curricular standards are met at the school, district, and state level, and at the national level, to the extent possible. 7.2.1 Expected number of districts, schools, and students The expected number of students, shipping destinations, and schools are provided in Appendix C as a basis for preparing replies. Respondents should calculate the quantities of materials needed to implement the test administrations on the basis of these estimates. If the total number of tests administered across all grades and test administrations exceeds the number of tests described in this appendix by more than ten percent (10%), the Department will execute a formal change to the contractual agreement. Appendix C also lists the 67 school districts and approximately 13 special districts. For the school districts and the special districts combined, the contractor will ship testing materials to approximately 80 separate destinations. In the 67 school districts and the special districts combined, approximately 4,000 separate administration sites (including private schools, community colleges, etc.) will be involved in the test administrations, and the contractor will expect that number to increase by as much as five percent (5%) annually, calculated based on the previous year. Some of this increase is due to legislative changes that increase the number of students participating in Florida’s statewide assessment program. If the increase of administration sites is beyond the anticipated five percent (5%), the incremental cost increase will be handled in an amendment to the contract. 7.2.2 Potential Test Administration Dates Potential projected starting dates for future assessment administrations relevant to this ITN are shown in Table 7.2.2 below for illustrative purposes only. The respondent’s reply should propose starting dates for spring administrations that occur as late in the school year as possible. Replies should take into account the scoring of constructed- and extended-response items, data verification activities, and reports of results by the required dates (Appendix B). THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK Florida Standards Assessments, ITN 2014-47 26 The FSA EOC Assessment administrations will be at the same time as the Florida EOC Assessment administrations. These dates are also provided, for reference, in Appendix A. Table 7.2.2 EXAMPLE: Potential Test Administration Dates for Florida’s Standard-Based Assessments Table 7.2.2 EXAMPLE: Potential Test Administration Dates for Florida’s Standard-Based Assessments Year Grades 3-11 ELA/L Grades 3-8 Mathematics Grade 10 ELA/L Retake 2014-15 March 2-13, 2015 N/A 2015-16 February 29-March 11, 2016 October 12-23, 2015 February 29-March 11, 2016 2016-17 March 6-17, 2017 October 10-21, 2016 March 6-17, 2017 2017-181 March 5-16, 2018 October 9-20, 2017 March 5-16, 2018 2018-191 March 4-15, 2019 October 8-19, 2018 March 4-15, 2019 2019-201 March 2-13, 2020 October 7-18, 2019 March 2-13, 2020 1 Optional Renewal Period Algebra 1 EOC Geometry EOC Algebra 2 EOC April 7-25, 2015 July 13-24, 2015 September 14-25, 2015 November 30-December 18, 2015 April 4-22, 2016 July 11-22, 2016 September 12-23, 2016 November 30-December 21, 2016 April 3-21, 2017 July 10-21, 2017 September 11-22, 2017 December 4-22, 2017 April 2-20, 2018 July 16-27, 2018 September 10-21, 2018 December 3-21, 2018 April 8-26, 2019 July 15-26, 2019 September 9-20, 2019 December 2-20, 2019 April 6-24, 2020 July 13-24, 2020 7.3.0 Test Design and Development All services related to test design, item development and review, item banking, and test production for the FSA are the responsibility of the contractor selected under this ITN. Respondents may propose a customized assessment, or products and services derived from existing products developed and published by the contractor, or developed as part of other initiatives. It is understood that by the time the contract is executed for the services requested in this ITN, test items may have already been developed for assessments that will be operational in 2014-2015. However, the Department expects to be closely involved in the development and review of new test items to be field tested and developed in subsequent years. To the extent possible, the Department also expects to be involved in any test construction activities for 2015 assessments still to be completed after contract execution. The Department’s primary goal with respect to test design is ensuring that the assessment measures student mastery of the Florida Standards, and accomplishes this in the most costeffective way without significantly increasing testing time beyond that of current Florida assessments. A sample test design is provided in Appendix D. The Department has established a Test Development Center (TDC), staffed with editors and Florida educators, to provide quality assurance for all test content and related products, whether customized or derived from existing products. The selected contractor will work closely with the Department and TDC staff to ensure that the test philosophy, design, item specifications, and product quality are aligned with Florida’s expectations. The Department and TDC will also coordinate closely with the selected contractor to conduct Florida Standards Assessments, ITN 2014-47 27 rangefinding meetings with educators in order to generate training materials for the scoring of performance tasks. While the selected contractor will be required to coordinate all meeting activities and materials, the Department and TDC will identify invitees and assist in facilitating all meetings. Required meetings are referenced throughout the ITN. See Section 7.7.7 for a summary of details about educator/citizen meetings. Test development procedures should include review and revision of proposed test and item specifications, review of test blueprints, review and evaluation of items available for use on the test (item development plan), field testing for needed items to complete the desired test design, and final operational test creation. All phases of test development shall use accepted validity, reliability and other testing principles including Universal Test Design. A plan for complete data review of field-tested items should be delineated. The Department must approve the items and the test forms before they become operational. Respondents who propose solutions derived from existing products that are used by other states must take into consideration the requirement for including state educators in the quality control aspect of item development and scoring of performance tasks. The extent of educator involvement will be discussed during negotiations and agreed upon prior to contract execution. To the greatest extent possible, the Department is seeking a solution that provides comparability of results with other states. Such comparability could be obtained by a number of acceptable means. The assessment could be one that is also administered in other states, or the design could include embedding of a small number of norm-referenced items within each criterion-referenced test form. A design that uses embedded norm-referenced items would not be expected to yield student-level comparative data, rather in this case, the Department would expect reporting at an aggregate level. The final design will be discussed during negotiations and agreed upon prior to contract execution. 7.3.1 Test Content 1. The FSA will consist solely of test items as described in the previous section. If an existing “shelf product” test includes additional components beyond those described above, it will be necessary to remove these items. 2. If the test being proposed is to be derived from an existing product, respondents should: a. Describe the general content being measured by the proposed test. b. Provide full-length inspection copies of the test and test administration manual for each grade level. c. Provide a test blueprint that governs construction of the test, including the number of items per objective and reporting category. d. Describe the objectives being measured by the test. e. Describe the taxonomic classification system used to classify each test item and the proportional assignment of items within the taxonomic classification system. f. Describe the range of content difficulty included on each level of the test including the extent to which the items at each level assess content that may be difficult for some students. g. Describe the method by which text complexity and grade-level appropriateness was determined for reading passages h. Describe how grade-level appropriateness of item content for each level of the test was determined i. Describe the extent to which the items included at each level assess higher order thinking skills. If the taxonomic classification system (See e. above) includes a thinking skills dimension, please include this information in that description. j. Describe the manner in which the test is configured into subparts for both test administration and score reporting purposes, including how reporting categories are related to the taxonomic classification system. Florida Standards Assessments, ITN 2014-47 28 k. Provide an indication of the minutes of testing time required for the separate content sections. 3. The respondent should describe the special materials or manipulatives required or permitted for correct administration of the proposed test (e.g., online rulers or Department-approved calculators). Detailed descriptions of or access to each required item should be provided with the technical reply. 4. Reading passages should expose students to grade-level text of appropriate complexity. Texts should reflect a balance of authentic literature (stories, drama, and poetry) and informational text/literary nonfiction, which would include passages that address the literacy standards requirement for history/ social studies, science, and technical subjects that students are expected to read. Selections should include a combination of single texts and paired passages of sufficient complexity for close, analytical reading which would allow students to discern and locate evidence from the text to support assertions in multiple-choice items or writing responses. Respondents may propose passages from three sources, including public domain, permissioned, and commissioned sources. Solely for cost comparison purposes, respondents may assume a test design that includes 1/3 of each passage type, but respondents may propose solutions that best meet the primary requirements of the ITN. The final test design will determine the proportion and genre percentages for passages. 5. English Language Arts/Literacy items aligning to each passage may include multiple-choice items as well as short, constructed responses to text-dependent questions. One extended response item should be drawn from a paired or multiple-text passage set. Extended writing prompts should allow students to produce opinions/arguments, informative/explanatory responses, and narratives. 6. The first tests to be administered by the selected contractor under this ITN will be the spring 2015 ELA/L and Mathematics tests in Grades 3 – high school. a. The selected contractor will be required to provide stand-alone computer-based calculator applications for 4-function, scientific, or graphing calculator use on computer-based tests, depending on final test design. These computer-based calculator applications will be made available to districts and schools for downloading to local computers and for online access and practice throughout the year. b. For any Florida test produced for students, the selected contractor will provide braille, large-print, regular-print, and one-item-per-page versions at each grade level for students with disabilities. The contractor must also provide computer-based accommodations for students with various disabilities, as described in Section 7.3.11.2. A demonstration of these accommodations may be included in the respondent’s presentation. c. The selected contractor will also print and ship Braille and large print sample test materials upon district request for students who will require these paper-based accommodations. These sample test materials are considered “generic,” meaning once they are created by the contractor, there is no expectation that they will be updated during the life of this contract. 7.3.2 Test Configuration This section provides information on the configuration for all test documents and related materials for both print and computer test delivery. 7.3.2.1 Test and Answer Document Design Respondents should describe the proposed document design and layout for each grade and subject test, for computer-based and paper-based tests, as described for each year in Appendix A. The selected contractor designs and produces camera-ready art for the test forms that are used to print test books and answer books or prepare computer-based test items in delivery-ready format. Contractors are expected to provide the necessary human and other resources necessary to meet extremely critical deadlines for delivery of documents and the computer-based test system and files. Florida Standards Assessments, ITN 2014-47 29 Respondents have been requested to submit bids for a paper-based and computer-based format for each assessment. However, even if a computer-based assessment is the accepted solution, any tests offered on the computer must also be prepared for use in a paper-based format (Braille, large print, regular print, one-item-per-page) as noted in these tables. In this instance, the paperbased format will only be offered by special request for up to five percent (5%) of the students and can be used at the Department’s discretion. To the extent that a respondent’s design solution allows for some customization, the Department will consider replies that include: 1. Security-controlled booklets containing only FSA Reading passages and no test items for the computer-based tests to be available by special request for up to five percent (5%) of the students. See Section 7.3.11.1. 2. Blank work folders for all students taking the FSA Mathematics computer-based tests. The work folder consists of four pages for students to do scratch work, the fourth page containing a full-page grid. 3. Blank worksheets for all students taking the FSA ELA/L computer-based tests. The worksheet consists of two pages for students to make notes. 4. A copy of the appropriate reference sheet, if proposed by the respondent and accepted by the Department, may also be provided for students taking the assessment for which it is proposed. 5. Work folders, worksheets, and reference sheets/periodic tables are not numbered or inventoried by the contractor; they are destroyed by the school after results arrive. For respondents who do not offer such customization, replies should address how test design and presentation eliminate the need for these considerations. The Department may choose to modify the design of any state-specific test documents/files prior to any test administration, and reserves the right to change this configuration beyond the specifications but within the scope of the contract, if necessary, through change orders or contract amendments. Appendix E provides additional details concerning these materials. 7.3.3 Develop Items For items developed after contract execution, it is the responsibility of the selected contractor to ensure that the quantity of items developed and submitted for review annually is sufficient to create operational test forms for each administration. Respondents should propose item and passage counts by item type, standard or cluster, as appropriate, needed to be developed annually in order to meet these needs. The contractor must identify and recruit a sufficient number of highly-qualified writers for passages and items to ensure variety and quality of development. Resumes for these writers should be provided and approved by the Department prior to contract execution. Note that no active Florida educators will be approved as item writers. The contractor must conduct in-depth training of passage/item writers at least annually. Department content experts will attend the passage/item writer training to provide input and oversight for the process. The selected contractor will be responsible for internal reviews to ensure item clarity, content accuracy, adherence to Test Item Specifications, psychometric properties, difficulty, readability requirements, and potential bias or insensitivity. It is unacceptable to submit to the Department proposed test items at any time that contain blatant errors, such as violation of Test Item Specifications, spelling errors, or grammar mistakes. The Department will not review any submissions with these types of obvious errors even if the impact creates schedule challenges for the contractor, for which the contractor will be responsible. 7.3.4 Review Meetings For test items developed for use on assessments following the spring 2015 administration, the selected contractor will be responsible for organizing and convening meetings for the purpose of reviewing assessment Florida Standards Assessments, ITN 2014-47 30 items for content and bias and community sensitivity issues by Department-appointed review committees composed of Florida educators and other citizens. A detailed list of meetings to be conducted under this contract and the funding mechanism for these meetings are provided in Sections 7.7.7 and 7.7.9 of the ITN. For each meeting, the selected contractor will be required to maintain original forms of all committee member test security and non-disclosure agreements. These will be made available to the Department on an asneeded basis. The meetings will be held in a Florida location that is easily accessible (hub cities) to the majority of participants and capable of handling the logistical requirements of a large-scale, multi-committee meeting. The selected contractor will prepare assessment items and all other materials required to conduct the meetings. Separate content review meetings will be conducted for each subject/grade combination. If more than one contract is awarded as a result of this ITN, the Department will coordinate with the contractors to determine the most efficient method of planning and conducting the review meetings. The respondent may propose recommended solutions in the event of the award of multiple contracts. 7.3.4.1 Bias and Sensitivity Committee Reviews The selected contractor is responsible for coordinating all logistics and for preparing materials for the bias and community sensitivity review committees. The committees are normally composed of representatives of Florida citizens and educators from various backgrounds who review each newly developed test item. The primary purpose of each committee is to consider whether the subject matter and language used is free of potential bias and acceptable to Florida students, parents, and other community members. Respondents should propose methods for conducting these reviews. 7.3.4.2 Item Content Review Committees For the subject/grade combination content review meetings, each item should be projected on a screen and changes should be made to items onscreen throughout the review process. The selected contractor will be responsible for supplying the computers, projection devices, and other hardware and software needed to project and modify items onscreen. The contractor will also provide calculators, reference materials such as Test Item Specifications, textbooks relevant to the grade and subjects being reviewed, dictionaries, and thesauri for each room as is appropriate to the nature of the meeting. Print copies of the items being reviewed will be provided to each participant in securitycontrolled notebooks. At the conclusion of each meeting, Department and contractor staff will review and evaluate changes and ideas recommended by each committee. The Department will have final approval of all changes to be made. After these meetings, the selected contractor will conduct an editorial and content review of the items to ensure that changes have been applied correctly and that all editorial and content issues are corrected. This process is referred to as the “Cleans” review process. The Department will review and have final approval of all items. The selected contractor’s development project manager, the development team, and necessary assistants will participate in the assessment item content review meetings to manage the meetings and to be responsible for organizing, distributing, secure material tracking, and data entry functions required for meetings. Arrangements will be required for collecting and keeping secure materials locked in secure storage when not in use and for copying materials during the meeting. 7.3.4.3 Data Review Committees Once items have been field tested, the contractor is responsible for convening a meeting to review the statistical data on each item from field testing to identify any issues from a psychometric perspective. This Data Review Committee may include state content experts, grade-level teachers, Florida Standards Assessments, ITN 2014-47 31 accessibility and accommodations experts, and psychometricians, as appropriate. The primary purpose is to ensure that all dichotomously-scored items and tasks behave as expected, include all correct and appropriate responses, and are fair for all students. This committee may review differential item functioning (DIF) statistics to ensure that an item did not unintentionally favor or disfavor a particular subgroup of students. Additional considerations include determining whether an item can be used as is or needs to be revised and, if so, whether it will be returned to the field test pool for further testing. The contractor will be responsible for applying any recommended edits to items and will submit the revised items for final Department approval. 7.3.4.4 Content Advisory Committees Of particular importance for the design and development of Florida assessments are the Content Advisory committees. The members of the ELA/L and Mathematics Content Advisory committees advise the Department about the scope of the assessments for these areas. Their recommendations may include: standards to be assessed; the number of benchmarks, passages, and items to be addressed per grade level; the item types recommended for each assessed standard; and the content of the Test Item Specifications documents. 7.3.5 Develop Field-Test Item Sets and Forms For designs using embedded field testing, it is the responsibility of the selected contractor to group the developed items into item sets that will be included in the operational forms of the test for each content area. In making the decision as to which items will be field-tested, the selected contractor’s field-test proposals must consider the areas of greatest need in the item bank, the range of content and rigor included in the items, and the number of items available to be field tested. 7.3.6 Develop Stand-Alone ELA/L Field-Test Item Sets and Forms Depending on the measurement model for the proposed ELA/L assessment, the Department will consider a stand-alone field test for ELA/L extended constructed-response tasks. Respondents who propose a standalone field test should prepare field test sampling specifications detailing the sampling methodology. The Department prefers that such a field test, if proposed, take place in the December time frame of each year of administration. A field test administration is not listed in other sections of this ITN (schedules, test administration manuals, etc.); however, if it is determined during negotiations that a stand-alone field test will take place, all test administration and scoring schedules and deliverables will be agreed upon prior to contract execution. 7.3.7 Constructing Test Forms This section describes the requirements for constructing test forms, including the creation and maintenance of test construction specifications. The test construction process must be supported by sophisticated computer software, preferably a component of the item banking system that will generate test statistics and related curves. The construction of operational forms must be completed in time to meet the requirements of this ITN for delivery of test materials to districts. 7.3.7.1 Test Construction Specifications The specific processes, guidelines, requirements, and schedule to be followed in constructing all forms of the assessments are to be provided to the Department as detailed, expert-developed, and referenced specifications. The test construction specifications must also include information about the criteria for item selection, content representativeness, design of the assessment, evaluation criteria for reviewing statistical characteristics of items, and desirable assessment characteristics. A section of the specifications must address considerations and requirements for those assessments that are pre-equated, if applicable, as well as a section for unique requirements of constructing computer-based assessments. Careful and extensive development of the creation of linking or anchor item sets must be included with details about statistical and content requirements for selecting sets of items and criteria for annual refreshing/replacement of portions of the anchor item set. Additionally, the specifications must include necessary information regarding item context/position Florida Standards Assessments, ITN 2014-47 32 based upon previous use. The selected contractor must also retain assessment construction records during the lifetime of the contract. These records must include assessment summaries (item IDs and item/assessment statistics) of the various versions of proposed assessments and reasons for decisions made regarding each version, and Department approval documentation for the final version of each assessment. 7.3.7.2 Test Construction System The contractor will provide a computer-based test construction system that interfaces with the item banking system to select and sequence items, generate test curves, and provide statistical summary tables and test curves upon demand. The system proposed by the respondent should be fully described in the reply including technical specifications, screen shots and access to a demonstration version of the system. 7.3.7.3 Construct the English Language Arts/Literacy, and Algebra 1, Algebra 2, and Geometry Retake Forms Grade 10 English Language Arts/Literacy and Algebra 1, Algebra 2, and Geometry retake tests will be administered under this contract beginning in fall 2015. Respondents should propose pre-equated retake forms that can be scored for reporting on accelerated timelines. Solutions for retakes that do not include performance tasks will be considered, even if performance tasks are included in the operational forms. Overall test characteristics must mirror those of the operational assessments; however, the Department will discuss the design of the retakes during negotiation and a decision will be finalized prior to contract execution. 7.3.7.4 Prepare Camera-Ready or Screen-Ready Test Forms For each grade or subject test, the selected contractor is responsible for preparing camera-ready documents or screen-ready files after the test items have been selected, composed in forms per grade/subject, and reviewed. Preparing test documents for publication includes developing test book configurations, test maps, specifying page layouts, proofing composed PDFs, and completing digital proofs in conjunction with the Department and external reviewers, as appropriate to the selected contractor’s proposed solution. 7.3.7.5 Interpretive Products The selected contractor will develop interpretive products (IP) to interpret or describe test items for specific subject areas or to explain test results. The contractor will be expected to propose sample items, review publication PDF and digital rounds, provide alternate text for ADA-compliant files, create or refine annotations for student responses, and ensure coherence and consistency with the operational test. Interpretive products follow the production and review guidelines outlined in Section 7.3.10. Required products include the following: Understanding Florida’s Assessment Reports This publication describes the various reports of test results distributed to students, schools, and districts. An explanation and an image of each type of report are provided. The booklet also includes information about the test content areas and a glossary of technical terms. The contractor will deliver the electronic ADA-compliant PDF for posting to the DOE website so files can be posted to coincide with the first results delivery (RD1). Sample Test Materials (STMs) STMs will be delivered for all assessments, including assessments that are primarily computerbased, and include 10-20 developed sample items in a sample test booklet for students with a separate answer key booklet for teachers. Both documents are produced in ADA-compliant PDF files for unsecure web posting and in large print and Braille accommodated versions. The printed accommodations are delivered to schools each fall so students can practice answering various item types. The ADA-compliant version is delivered at the same time so all students have the same opportunity for practice. Respondents should propose designs for sample test books and answer Florida Standards Assessments, ITN 2014-47 33 keys for the ADA-compliant PDF version and for printed large print and Braille accommodated versions. English Language Arts/Literacy and Mathematics Scoring Guides The contractor will be expected to post electronic publications that provide information to help educators understand the scoring of constructed-response items included on student reports. These publications provide information about the test design, standards and benchmarks, item types, and how items are scored. They also provide scoring rubrics and the annotated papers used to train scorers for the constructed-response items. Each sample item from the scoring papers is displayed as it appeared on the test and responses for each score point, images of student responses, and annotations explaining why the response received a particular score are provided. Computer-Based Test (CBT) Practice Tests Respondents should specify a plan for delivering computer-based practice tests for all computerbased assessments, including accommodated computer-based assessments to be reviewed and revised as needed in conjunction with Department and external reviewers. The purpose for these practice tests is to familiarize students with the computer-based testing system and to provide students with an opportunity to practice answering various item types. All practice tests will include items similar to test format and content. The practice session may combine subject items or use separate practice sessions for each subject area as the Department will determine. Practice sessions will include approximately six (6) to twenty (20) items per subject. Review and development of the student tutorials/practice sessions must follow the Department’s procedures for forms review, outlined in Section 7.3. In addition, the Department reserves the right to request updates to the practice sessions to reflect any changes in CBT platform functionality or necessary changes to the practice questions throughout the life of the contract. The contractor must anticipate that these updates may occur and must be prepared to provide these updates at no additional cost. (These would not qualify as “author’s alterations” defined in Section 7.7.9.3.) Additional information about the computer-based format and practice tests is provided in Table 7.4.9.6. Test Item Specifications Respondents should propose a detailed plan for producing and reviewing test item specifications for ELA/L and Mathematics grades and subjects in consultation with the Department and Florida educators. The specifications govern the writing and review of passages, prompts, stimuli, and test items and generally identify the item/task requirements and constraints for measuring each standard. The publications may include introductory information and appendices as necessary to fully explain how the tests and items are developed. The test item specifications may include clarification of the standard being assessed, specific content and item limitations, and item formats by which each standard will be assessed. Sample items may be included for each item type. Item writers are the primary audience for these publications, but Florida educators also use these materials, so ADAcompliant PDF files will be posted to the Department website. These publications will not be distributed in print. Respondents may propose approximate document lengths and document design. In conjunction with the review of the test item specifications, an external review panel of Florida educators will review the test design and philosophy contained in each manuscript and will make recommendations for improvements to the item specifications for each assessment. 7.3.8 Production of Printed Test Materials 1. The selected contractor is responsible for producing all versions of the materials and products developed for this project, including test forms. 2. The selected contractor is responsible for proofreading documents during all stages of production. The contractor should propose well-defined procedures to ensure drafts are checked by computer for spelling and Florida Standards Assessments, ITN 2014-47 34 proofed for complete accuracy by qualified proofreaders from the respondent's staff before they are sent to the Department for review. Qualified persons within the respondent's organization and independent editors, who will be responsible for proofreading, should be named in the reply and their resumes attached. The respondent should include a provision for the cost of changes made during the final proof stage of production as “author’s alterations” to be authorized by the Department. 3. The selected contractor will design and implement a procedure for assessing the quality of printing. The procedure must include on-site quality control plans to ensure 100% correct printing, sealing, and collating test documents. In their replies, respondents should explain what arrangements they have reached with printers to establish technology or procedures to prevent miscollating test and answer books. Printing subcontractors that produce miscollated test and answer books may not be approved for subsequent test administrations. Procedures for ensuring printing quality must include plans for proofing all materials before and after printing. The selected contractor will be responsible for replacing misprinted or otherwise defective materials at the contractor’s expense within a time period acceptable to the Department. 4. In addition to the quality of printing, the Department requires stringent test security procedures to be followed during all stages of document production and printing. In their replies, respondents should explain the security plan to be implemented while test documents are in production and at the printer. See Appendices F and I for these restrictions. 5. If errors occur during production, the selected contractor will be responsible for providing to the Department within 30 days of the event a root-cause analysis, providing details on the cause and steps for improvement. 7.3.9 Production of Computer-Delivered Products 1. The selected contractor is responsible for producing all versions of computer-delivered products developed for this project, including test forms. These products include all information, text, and graphics that will be presented to computer-based test users. 2. The Department recognizes that the development, review, and approval processes for computer presentation of information and materials parallels similar work for printed tests and products. 3. The respondent should specify its suggested plan for computer-based-test production with the objective of ensuring that industry-leading quality is attained. The plan for the test content review should include steps required to develop computer-based tests and practice tests to be reviewed and revised as needed prior to approval in conjunction with Department and external reviewers, as appropriate to the respondent’s proposed solution. 4. The functionality of CBT forms (e.g., tools, reference sheets, navigation) must be checked carefully to ensure that errors have not been introduced, and that the functionality remains consistent across forms and administrations. The Department will provide a checklist that can be used as a starting point for the functionality review, but the selected contractor must be prepared to devote significant time to thoroughly check each form. 5. A final review round will include an “end-to-end” check of the product, including checks to ensure user access to the product and accurate navigation and functionality. The respondent should include a provision for the cost of changes made following approval of a computer-based-test form as “author’s alterations” to be authorized by the Department (see Section 7.7.9.3). 6. Qualified persons within the respondent's organization and independent editors, who will be responsible for proofreading, should be named in the reply and their resumes attached. 7. The Department requires stringent test security procedures to be followed during all stages of document production and printing. In their replies, respondents should explain the security plan to be implemented while test documents are in production. See Appendix F for these restrictions. Florida Standards Assessments, ITN 2014-47 35 8. If errors occur during production, the selected contractor will be responsible for providing a root-cause analysis to the Department within 30 days following the event, providing details on the cause and steps for improvement. 7.3.10 Production of Paper-Based and Computer-Based Test, Style Guide, and Braille Production Procedures 1. The selected contractor will be responsible for all aspects of production for the CBT system, secure webbased reporting site, and printed materials, including formatting, graphics, and key entry. For each publication, the contractor will submit product specifications that identify printing plans, type size and style, ink and paper color, paper quality, and layout. Printing examples that show type size and style will be included. The contractor will provide copies of the final electronic files for products. 2. The selected contractor will also print any additional materials needed to implement the project, such as transmittal memoranda, labels for packing and packing lists. 3. The selected contractor will be responsible for all aspects of production for publishing printed products, including formatting, graphics, and key entry. For each publication, the contractor will submit printing plans that identify type size and style, ink and paper color, paper quality, and layout. Printing examples that show type size and style will be included. The contractor will present these plans annually. The Department expects attractive, quality printed materials at reasonable cost. Guidelines for the quantities of each product to be printed are indicated in Appendix E. 4. Editorial specifications must include editorial and stylistic considerations for the development of test questions and the layout of test pages. The selected contractor is responsible for updating and revising the specifications annually as style and layout requirements may change. The contractor will work with the Department to identify the components of the specifications that need revision and will revise and print copies for all users. 5. The selected contractor will submit plans that identify screen layouts and navigation elements, font style and size requirements, background and foreground color palettes, and requirements for graphics. The contractor will identify and propose a branding plan to identify and provide consistency for Florida assessment components delivered via computer. The Department expects an attractive, high quality, engaging, and functional presentation. The contractor will present this information, as well as the proposed scheduling and process for review rounds, in a specifications document that addresses computer-based presentations. 7.3.11 Produce Special Formats and Accommodations for Students with Disabilities 7.3.11.1 Paper-Based Accommodations For all assessments, the selected contractor will provide Braille and large print versions for visually impaired students. For all assessments, the selected contractor will also provide one-item-perpage/one-sided accommodated versions as unique accommodations. For test documents, the selected contractor will create the electronic files in the correct format needed to produce these special formats. Although multiple forms of the tests may be constructed for each administration, only one form for each test will be converted to Braille, large print, and one-item-per-page versions. Large print documents will be printed in a minimum of 18-point type on approved paper and bound using agreed upon methods. Reformatting of documents may be necessary to meet these specifications. The selected contractor is responsible for having Braille materials proofed by an independent party that includes a certified Braille reader. The Department may also employ the services of a Braille proofreader. See Appendices E and G for more details and specifications on special formats of test documents. 1. Respondents should include proposed subcontract publishers of Braille and large print materials in their response. This publisher will produce the accommodated versions of the test and answer books and other documents at the contractor’s expense. Test administrator notes and Florida Standards Assessments, ITN 2014-47 36 scripts to accompany each administration’s Braille test versions will also be developed by the contractor. 2. While most blind students in Florida require contracted Braille, there will be some students who require uncontracted Braille. The selected contractor will be responsible for providing either format based on district requests. Approximately 50 copies of the Braille versions and up to 400 copies of the large print version may be required for each grade level and subject area per administration. Test administrator notes and scripts to accompany each administration’s Braille test versions will be provided by the contractor as a supplement to the manual, and will undergo the same iterative review process as other printed materials. 3. If requested by districts, the selected contractor will also provide large print and/or Braille versions of sections of the test administration manuals. These publications will be produced so that they will be delivered to districts along with the shipment of the regular print versions of these products as identified in Appendix E. A maximum of five (5) each per format may be produced for each administration. 4. The selected contractor will maintain and update an internal document that identifies which person, by position and name that will perform each task in the Braille materials process. Each task must be described in detail and the document must be accessible online with other downloadable materials. 5. On request, the selected contractor will develop and deliver one-item-per-page documents to districts. Approximately 25 copies of the one-item-per-page version may be required for each grade level per administration. The need for these specialized documents will be determined by the Department through its unique accommodations requests process and the Department will provide an approved list identifying students needing this accommodation to the contractor. 6. The one-item-per-page documents will be packed and shipped by the selected contractor for identified students. These documents will be printed on 8 ½” x 11” paper, three-hole punched, and placed in a binder so that individual pages may be removed as necessary. Because these are also used for students who require true black and white print, the selected contractor will provide for the Department’s approval a sample of the materials to be used that demonstrates the paper quality and font/style. The document package will include the appropriate scannable answer document for student responses and the contractor will include directions for administration and management of this test version. The contractor will be responsible for transcription of one-item-per-page student responses. 7. The respondent’s reply should describe the processing of large print, Braille, and one-item-perpage versions of answer documents, including handling and transcription during the scoring process. All Braille, large print, and one-item-per-page materials require separate packaging but must be shipped so that they arrive in districts at the same time as the materials for regular students. 8. The contractor will also deliver paper-based regular print or large print Reading Passage Booklets for computer-based Reading tests. The booklets, which contain reading passages but do not contain test items, will be for use by eligible students who need to utilize handwritten strategies in order to respond to test items on the computer. See Section 7.3.2.1. 7.3.11.2 Computer-Based Accommodations Florida’s computer-based tests must also be available in paper-based format as an accommodation at all grades and subjects. The use of this format will be dictated by a student’s IEP or Section 504 plan. The Department anticipates that the number of students taking a paper-based test (PBT) based upon allowable accommodations will not exceed five percent (5%) of the total population for each grade and subject tested on computer, and the selected contractor will produce this quantity five Florida Standards Assessments, ITN 2014-47 37 percent (5%) of paper-based tests. For these students, paper-based accommodations described in Section 7.3.11.1 must be available. The reply should address the capability of the computer-based system to provide the following special accommodations for students and to handle any additional requirements related to the use of these accommodations. 1. Screen Reader The respondent’s reply should include a description of a screen reader for computer-based tests. The respondent should propose a method for reading text within the computer-based system that does not require additional, proprietary software installed on the user’s computer. The screen reader should be able to read all text and/or text descriptions of graphics or pictures, including graphics or pictures used as answer options. The system should also have the capability to use modified text if necessary (e.g., reading an algebraic equation with grouping symbols), and allow user control of the reading speed and volume. The screen reader should be designed for use by both sighted and visually impaired students. The Department recognizes that the contractor may have different types of screen readers for these two purposes, but this difference should be transparent to the user. The Department must approve the usability and voice of the screen reader. In addition, the contractor must ensure that the pronunciation of the screen reader is accurate, consistent across forms and subject areas, and would be recognizable to students. If the screen reader does not have accurate, recognizable pronunciation, it must be modifiable to meet the Department’s standards. In addition, the contractor must check the pronunciation on every form to ensure it is correct, is tracking properly, and is understandable prior to it being provided to the Department for review. The Department prefers that the contractor adhere to the Department’s screen reader specifications to ensure consistent performance of the screen reader. However, the Department will review screen readers that meet the intent of this section and the needs of the Department. 2. Increased Font Size (Large Print and Zoom) The respondent’s reply should describe available options on the CBT system to enlarge font size, graphics, and CBT interface (i.e., buttons, tools, cursor, scrollbar) for students with disabilities and describe how these options are set for a student. See Section 7.4.9.6 for large print and zoom size requirements. 3. Font and Background Color (Color Contrast) The respondent’s reply should describe available options on the CBT system for changes to the font color and background color in order to optimize the ability of students with visual impairments to read the screen and describe how these options are set for a student. See Table 7.4.9.6 for the font/background color combinations. 4. Assistive /Adaptive Devices The respondent’s reply should describe the ability of the CBT system to interface with assistive/adaptive devices. The reply should include a description of specific devices that may be used and known devices that are not supported, as well as any special requirements necessary for use, and should describe how options for use are set for a student. 5. English-to-Heritage Language Translation Dictionary The respondent’s reply should describe the provision on the CBT system of an on-screen English-to-heritage language translation dictionary for English Language Learners, if this can be made available. The accommodated student should be able to select a word and be presented with a translation sensitive to the context of the text. The language translation dictionary should be available in Spanish and Haitian Creole. The respondent’s reply should describe the highlevel tasks involved, note the general timelines required, including the major/essential deliverables and services required. Florida Standards Assessments, ITN 2014-47 38 6. English-to-Sign Language Translation Dictionary The respondent’s reply may describe the provision on the CBT system of an on-screen Englishto-sign language translation dictionary, if this can be made available. The accommodated student should be able to select a word and be presented with a translation sensitive to the context of the text. The respondent’s reply should describe the high-level tasks involved, note the general timelines required, including the major/essential deliverables and services required. 7.4.0 Test Administration The contractor is responsible for all aspects of test administration described in this section. 7.4.1 Pack and Distribute Materials 1. The contractor is responsible for all arrangements and costs associated with packing, distributing, and retrieving physical materials. There must be 100% accounting for all secure materials (e.g., test books, test and answer books) distributed to and returned by districts using barcode labeling systems, as well as for any physical media (e.g., CDs) used to transmit electronic files. The contractor must guarantee that distribution procedures are accurate and make corrections to the system in the event of errors. Respondents’ replies should include descriptions of the procedures they will use to complete these tasks. Respondents should identify carriers they propose to use and the procedures for delivery and retrieval of test materials. The Department must approve all carriers. 2. The contractor will prepare packaging and distribution (P&D) specifications for each test administration. The contractor will use the information in the specifications to program packaging systems and train personnel involved in the processes. The packaging specifications will include the contractor’s procedures for packing and distributing materials to districts and retrieving materials from districts. The specifications will include a description of how the materials are packaged, how quantities of materials will be calculated, examples of packing and inventory lists for boxes sent to districts and schools, methods used for distributing and retrieving materials, a description of the procedures used to inventory materials as they are returned, as well as any special instructions or administration-specific information. The contractor will also include samples of all memoranda required in the delivery/pickup process. For reference, the Department will provide the contractor with the most recent packaging and distribution specification document at the initial contract management meeting. Proof of Department approval for each specification document will be evidenced through email confirmation of approval. 3. The number of districts and special schools that serve as districts may change slightly during the life of the contract. Appendix C provides information about the number of districts, schools, and students that will receive test materials. Respondents should anticipate shipping materials to district assessment coordinators at approximately eighty (80) separate sites. Materials will be shipped to a central district location and distributed to the schools by the district. 4. Several school districts have contracts for services with school sites that are not geographically located within the school district itself. School district personnel supervise the sites and the district assessment coordinator provides training for site-based test administrators related to the administration, including security, packing, and shipping. To enable prompt receipt and return of test materials, the contractor may be required to ship materials directly to and retrieve materials directly from these remote sites. The Department will work with the district assessment coordinators and the contractor to identify and approve the remote sites to which the contractor will be permitted to ship and retrieve materials without going through the district office. For the purpose of preparing replies, respondents should expect that up to twenty-five (25) such locations might be identified. 5. At the beginning of the contract, the Department will provide the contractor with a data file containing a list of the districts and schools (names and identification numbers). The contractor will be responsible for maintaining this list, updating it, and providing a copy of the data file to the Department at the end of the contract or at other times within the contract period on request. Florida Standards Assessments, ITN 2014-47 39 6. The contractor will collect preidentification information from districts for each administration via the online management system and assume that preidentification will be used for 100% of the school population. These files will be used to load student data into computer-based testing systems, as well as produce barcode PreID labels and rosters for paper-based tests. The contractor will establish a system to ensure that the labels/documents delivered to districts contain all requested data, reflect the options selected by districts, are accurate, and are printed at a level of quality that permits accurate scanning and precludes the possibility of smudging, smearing, and/or flaking. Contractors will provide quality control throughout the printing process to ensure the quality of label/document printing. The contractor will provide a separate roster of students using the same sort order used to prepare the student labels/documents. This roster will be used by districts and schools to verify the accuracy of certain information that must be correct before the label/document can be used. 7. A list of the primary materials to be shipped for each administration, the quantities to be packaged for schools, districts, and the Department, and other packaging specifications is provided in Appendix E. 8. The contractor’s specifications for packing, distributing, and receiving materials will include the following provisions: a. The contractor will shrink-wrap materials in quantities specified by the Department. Some materials will not be shrink-wrapped (e.g., Test Administration Manuals). All secure materials will be shrinkwrapped. b. Schools within a district will be ordered by school number on all lists and for shipping purposes, unless the district has requested that materials be sorted in route code order, in which case the materials will be sorted by route code, by school number. c. The contractor will package and ship materials by district size as determined by the Department, beginning with the largest district. The largest district will receive materials at the beginning of the delivery window; the smallest district will receive materials last. District rank by size information will be updated annually by the Department. d. The contractor will label the boxes of test materials with the message “TO BE OPENED BY ASSESSMENT COORDINATOR ONLY.” District boxes will be differentiated from school boxes by color (e.g., white district boxes and brown school boxes). Boxes containing the district/school packing lists, materials return kits, and ancillary materials must appear different (e.g., different color, colored stripe) from all other district and school boxes for easy identification. School boxes will be labeled with the name/number of the district and name/number of the school in at least a 16 point font. Label design must be approved by the Department. Only boxes directed to the district-level staff, such as boxes containing district overage, will be labeled with the name of the district assessment coordinator. The contractor will label district boxes (that contain overage and other materials for the district) on all four sides and the top, and number boxes as “Box 1 of X,” where X is the total number of boxes sent to that district. The contractor will label school boxes on all four sides and the top, and number boxes as “Box 1 of X,” where X is the total number of boxes sent to that school. If materials for more than one administration or type of material are shipped during the same window, the contractor must have a way to distinguish these shipments from each other (e.g., colorcoded label on boxes). Boxes containing secure test materials must never indicate secure content on the outside of the boxes. e. For districts receiving more than one pallet in a shipment, a map listing the identity and location of boxes on each pallet will be provided electronically and will be provided at the time of delivery as a hard copy to the district assessment coordinator to assist with distribution of boxes in the district. f. The contractor must have a method to collect route code information from districts and package materials accordingly. If district assessment coordinators provide internal district delivery route code Florida Standards Assessments, ITN 2014-47 40 information, the contractor will use this information to build pallets for districts to match district delivery route codes. g. The contractor will pay charges on all materials shipped to and from each district. The contractor will make arrangements for and pay for shipment if, due to a delivery error, a district is asked to ship materials to another district. The contractor must use an overnight delivery service for such shipments. The contractor is not responsible for any costs schools may incur in shipping test materials from their schools to the district office, unless the need to ship is the result of a packaging error by the contractor. The contractor must also have a method for districts to order additional test and ancillary materials, as needed, in the online management system. The contractor must be aware that requests for additional materials are often critical and such shipments must be shipped overnight for early-morning delivery, on request. h. The contractor must receive Department approval before shipping materials to districts. Approval will be provided after the Department has received and proofed examples of printed products and reviewed sample shipments, as described in the packaging and distribution specifications. Approval is sent via an Approval to Ship form. i. Secure materials will be packaged sequentially in ascending barcode order (for example, if a school receives 200 secure documents and 100 documents fit in each box, Box 1 will contain the documents with barcode numbers 1-100, and the document with barcode #1 will be at the top of the documents contained in Box 1 and the document with barcode #100 will be at the bottom of the documents contained in Box 1. Box 2 will contain the documents with barcode numbers 101-200, and the document with barcode #101 will be at the top of the documents contained in Box 2 and the document with barcode #200 will be at the bottom of the documents contained in Box 2.). Packing lists must list all box contents in order as they are packaged with the first item on the list being the first item (on top) in the first box. j. The contractor will package materials for each shipment by school in returnable boxes (doublewalled) suitable for multiple uses. The contractor will provide additional boxes statewide for the return of materials that are available as an additional order for districts. k. No box will be packed to weigh more than 25 pounds, and boxes will be of the dimensions, or similar to, those indicated in Table 7.4.1. If test materials require different box dimensions than those listed below, the specifications for boxes will be approved by the Department during the negotiation phase. Table 7.4.1 Box Dimensions* Table 7.4.1 Box Dimensions* BOX SIZE MATERIALS 9.5” x 12” x 13”, Double-walled Regular Print 9.5” x 12” x 6.5”, Double-walled Regular Print 19” x 15” x 4.5” Large Print/Braille 19” x 15” x 6” Large Print/Braille 19” x 15” x 9” Large Print/Braille * Box height is dependent upon the quantity of materials in each shipment. l. Upon completion of the delivery of each shipment, the contractor will provide proof of delivery to the Department indicating, but not limited to, a description of the materials, date/time of delivery, and the name of the staff member accepting the delivery. Proof of 100% delivery will be used for invoicing purposes. Florida Standards Assessments, ITN 2014-47 41 m. The contractor will provide fifteen percent (15%) materials overage to each district for paper-based test materials and computer-based ancillary materials (work folders, etc.). The Department and districts will decide how materials overage will be distributed to districts and schools for each shipment. Districts may select from four (4) distinct algorithms for overage distribution:  All fifteen percent (15%) overage to the district  All fifteen percent (15%) overage to the school  Ten percent (10%) overage to the district and five percent (5%) overage to the school  Five percent (5%) overage to the district and ten percent (10%) overage to the school n. The contractor must anticipate that some districts will require an additional pickup after the specified pickup date. This may occur in up to 15 districts for as many as 500 boxes for each spring administration, and in up to 10 districts for as many as 200 boxes for each retake and End-of-Course administration. o. Large-print, Braille, and one-item-per-page materials are to be packaged and labeled separately and delivered to districts during the same delivery window as the shipment of other test materials in Test Materials Shipment 2 (TM2). p. Preidentification labels and rosters will be packaged with test materials (included in TM2). If applicable, Wave 2 labels will be shipped separately. q. It may be necessary for the contractor to ship test materials to districts/schools and the Department in multiple shipments per testing window, not including requests for additional materials. The number of shipments may vary depending on the administration. The respondent should refer to the approximate shipment dates provided in Appendix B. The contractor must be prepared to deliver some materials on alternate schedules per district request (e.g., to avoid conflicts with spring break or closures) and must have a method to effectively track these requests and ensure that shipping companies meet the revised dates. Shipping companies must never contact districts directly to arrange alternate delivery/pickup dates unless agreed upon by the Department. r. The contractor will prepay charges on return shipments from districts. Return labels, prepaid postage labels, instructions, and other materials needed for return will be provided in a Materials Return Kit shipped in TM2. s. Districts will return answer documents using overnight delivery, if deemed necessary to meet reporting deadlines. Materials will be returned according to a pickup schedule developed by the Department and the contractor that allows schools enough time to package materials and also meets scoring and calibration deadlines. t. The contractor will be responsible for mailing or shipping any miscellaneous materials to the Department and districts by overnight delivery service or other means, as requested. The contractor is required to secure the services of shippers who will provide inside delivery and unload large shipments onto loading docks. All carrier vehicles must display the carrier name, and districts must be made aware of what carrier name to expect prior to the pickup or delivery date. Districts may or may not have a loading dock or the appropriate equipment to support the delivery and pickup of materials (e.g., shrink-wrap; hand-trucks). In this event, the contractor’s shipper must be prepared to provide such services/materials. District staff is not expected to assist in loading or unloading materials, but will be available to provide instructions to the carrier. Carriers must communicate with district staff face-to-face when delivering or picking up materials, never leave materials unattended, and be prepared to protect materials in the event of inclement weather. u. The contractor must develop procedures to monitor the receipt of all materials and develop error logs. The date materials are received and any errors made by districts in packaging and completing forms Florida Standards Assessments, ITN 2014-47 42 must be documented. When problems arise, the contractor must contact districts and the Department to resolve them. Error logs will be identified by school and district, and will document all failures to follow the established procedures and, if appropriate, how the errors were resolved. The error logs will be delivered to the Department within three weeks after the scheduled receipt of materials from all districts. v. The contractor will use a barcode numbering system to verify all secure materials returned after testing. The contractor will provide a written report to the Department documenting the check-in of all secure materials. f the contractor’s system for barcode verification is demonstrated to be unreliable (e.g., materials not scanned properly), each document will be required to be scanned twice, creating independent data files that can be matched for scanning errors. Beyond these measures, the contractor will be required to manually check answer documents to ensure 100% accuracy of secure document check-in. w. The contractor will include in its check-in procedures a method of checking for and retrieving used answer documents erroneously packaged with used and unused materials, invalidated materials, or test books. These “orphan” answer documents will be processed and scored under extended timelines and individual reports will be generated and shipped to the districts. x. The contractor will produce statistical reports summarizing responses to the online test administrator, school coordinator, technology coordinator, and district coordinator comment forms. A summary, analysis, and list of recommended actions derived from the comment form responses, as well as a data file containing all responses, will be delivered to the Department no later than two (2) months after completion each administration. Contractor staff will be responsible for summarizing the openended responses received from district and school staff. This summary must be a categorized report, not simply a list of open-ended responses. 7.4.2 Missing Materials Report and Inventory The contractor will prepare a missing materials report for all secure materials, based on the scanning that is completed during materials check-in. The report must be produced after all materials have been scanned, including as many orphan and late-returned documents as possible, and must not reflect materials that are not missing (e.g., contractor has verified against the soiled/destroyed documents list). The contractor must implement quality control measures to ensure the accuracy of the reports. The contractor will produce a preliminary missing materials report for Department review. The Department will review and approve the preliminary missing materials reports for distribution to districts. The Department requires ten working days to review preliminary reports prior to approval. Reports will be prepared for each affected school listing the title of the missing products and the corresponding barcode numbers. School reports will be delivered to district assessment coordinators with a district-level summary and memorandum, and a state-level summary report will be delivered to the Department. Any missing materials returned by districts will be recorded in the missing materials inventory maintained by the contractor. The contractor will deliver a final summary report of missing materials to the Department by the date indicated in the Deliverables Schedule. 7.4.3 Disposition and Storage of Materials After testing, the contractor will inventory and store unused test and answer books, manuals, and other materials for a period of at least six (6) months. During this time, the contractor will be required to ship quantities of these materials, as necessary, to the Department and districts. Replies should include a plan for secure destruction of excess test materials that requires Department approval. However, the contractor will store 100 copies of each test and answer book for each administration throughout the life of the project. In addition, the contractor will store some digital record of the CBT assessments as they appeared to the students for that administration. Any materials that may be used in subsequent assessments will be stored by the contractor throughout the life of the project. Additionally, six (6) months after the contract ends, the subcontractors and print vendors will be required to destroy all electronic files and used print copies according to the test security requirements approved by the Department. Florida Standards Assessments, ITN 2014-47 43 On verification of the individual barcode numbers of all secure materials returned by the districts and acceptance by the Department of accurate results files, the contractor will store used documents containing student responses, at contractor expense, for a period of two (2) years. Likewise, all student CBT response files must be electronically archived for a two-year period. At the end of the two-year period, the contractor will ship or destroy the materials according to instructions from the Department. The contractor must submit a letter to the Department requesting permission to destroy specific materials. This process is already welldefined by the Department and will be shared at the initial contract management meeting. Test security requirements will be maintained throughout the destruction process. It may be necessary to retain a small number of used documents for a longer time period due to Department security investigations. At the end of the contract, the contractor will be expected to ship these materials to the new contractor. Destruction of secure documents and CBT student response files must be requested in writing and authorized by the Department. The contractor and all subcontractors must submit certificates of destruction that describe in writing the specific items/files destroyed. 7.4.4 Retrieval of Answer Sheets and Test Books In addition to the document retrieval specified in Section 7.4.1, the contractor will periodically be required to retrieve up to 1200 used test and answer books per grade, per administration from storage. Likewise, individual student response files from the computer-based testing system for up to 1200 students per grade, per subject, per administration may need to be retrieved. The contractor must maintain information related to warehouse location for each school’s used materials to allow for retrieval as may be required for Department investigations or audits. In some cases, retrieval of answer documents will require the contractor to conduct a manual verification of machine scoring. The contractor will be responsible for costs associated with retrieval, review, and possible delivery of these materials to the Department. In addition, if errors are found, the contractor may be required to re-score and re-report these documents. 7.4.5 Innovative Product Design New, innovative designs are required for all Florida-specific products developed by the contractor. Designs for publications will include the organization, format, and page or screen layouts for administration manuals, reports of assessment results, information publications, and other materials. Designs will also be produced for other products, as appropriate. The contractor will produce all graphics, charts, and illustrations for the products for which it is responsible and share designs and graphics with other contractors, as required by the Department. The design process will include furnishing conceptual art, design development, and final cover designs. The contractor will produce all graphics, charts, and illustrations and will secure the full range of copyrights and/or permissions that allow for production in print, secure web, and unsecure web format. The contractor must arrange for personnel with professional expertise and experience in the areas of visual/graphic arts and document/software/web design to work on the design and development of documents and computer/online interfaces. If the contractor does not have qualified, professional personnel devoted primarily to visual/graphic arts and document/web design on staff, a subcontractor or subcontractors must be identified to provide these design services. Respondents should indicate in their replies the qualified personnel/subcontractors who will provide these professional services and include their vitae or other documentation of expertise and experience in the reply. 7.4.6 Printed Products – Administration This section provides guidelines, including descriptions of and duration of review rounds, for all printed products required for test administration services in this ITN. Contractors must provide high-quality, carefully edited materials for each round of review by the Department. 7.4.6.1 Stages of Development of Printed Products – Administration The contractor is responsible for producing the first written draft and all subsequent versions of all materials and products developed for this project, and for submitting documents for Department review. The review period for the Department is seven (7) work days, but longer review times will be required if multiple documents are under review simultaneously. For each review round, the contractor Florida Standards Assessments, ITN 2014-47 44 will post an electronic version (PDF) by 9:00 a.m. ET on the first scheduled day of Department review. If materials are delivered or posted after 9:00 a.m., the review schedule will be revised accordingly. Documents for CBT-related materials will be posted to a secure website for review in their presentation format. The delivery of administration materials is a critical task and workflow must be planned carefully. The overlap of reviews could create a burden on both the Department and contractor staff. Excessive burdens can be avoided with sufficient staffing, advanced planning, and an early start. The contractor must propose a detailed materials development schedule for managing the workflow and volume of documents to be reviewed in each seven (7) day period in a format approved by the Department. The contractor must develop a schedule for the production of test administration manuals that follows the review schedule of test books/answer documents or computer-based test forms by one round. The schedule must consider what is reasonable given the Department’s staffing constraints. All proposed schedules must be approved by the Department. The number of documents to be reviewed in any seven (7) day period must be negotiated with and approved by the Department. The Department is committed to reviewing products submitted by the contractor as efficiently as possible. The contractor is responsible for proofreading all documents during the production stages. The contractor will propose well-defined procedures to ensure that all drafts are checked by computer for spelling and proofed for complete accuracy by qualified proofreaders before they are sent to the Department for review. The contractor will submit a written report by the proofreaders indicating proposed results and actions. Before moving to digital proof stage, the Department will review and approve the proofreaders’ report. Respondents should indicate in their replies the qualified personnel/subcontractors/independent editors who will be responsible for proofreading and include their vitae or other documentation of expertise and experience in the reply. Test Administration Manuals For respondents proposing solutions derived from other products, the Department understands that certain test administrations must be standardized, however, there are certain Florida requirements that must be met. Replies from such respondents should indicate which aspects of test administration are considered customizable, and a final determination will be agreed upon during the negotiation phase. For instance, the Department may require the production of test administration manuals specifically for Florida that include the standard information, or the production of Florida-specific supplementary publications that complement the contractor’s more generic publications. The following information details the Department’s expectations for Floridaspecific stand-alone products. For each year of the contract (Fall – Summer administrations), the following test administration manuals will be produced: One (1) ELA/L and Mathematics manual for paper-based tests; One (1) ELA/L and Mathematics manual for computer-based tests; One (1) ELA/L Retake manual (beginning with the first retake administration in 2015); (1) End-of-Course manual. Administration requirements and scripts for all assessed grade levels/subjects will be included in the manuals. Instructions for Special Format Tests will include scripts necessary for accommodated testing (e.g., Braille, one-itemper-page, and others) and will be provided as separate addenda to the manuals. Computer-based manuals will also include scripts for the practice tests necessary to familiarize students with the computer-based testing system. All manuals are published in print and on the contractor’s website. Minimally, manuals will contain instructions and scripts for administering each portion of the assessments for the appropriate grade levels for that administration. They also include information about security of materials, packing and returning materials, receipt and distribution of materials, and the Department’s test administration and security policies and procedures, which are subject to change with each administration. Revisions will be made as necessary to reflect program changes. Florida Standards Assessments, ITN 2014-47 45 The contractor must plan to schedule production of test administration manual covers as separate mini-deliverables that will eventually be married to the final products. Each cover requires multiple review rounds by graphics designers, editors, and content specialists, and careful planning is required to meet production deadlines. Test Administration Manuals are to be packaged (no shrink wrap) and delivered to districts for each administration as a part of TM1 according to the specifications in Appendix E, Part 3. All manuals must also be available in a Section-508 compliant (see Appendix E, Section 2) electronic format on both the Department’s and contractor’s websites within seven (7) days of Departmental approval of the web-tagged files. The suggested workflow for the Department and the contractor is provided in Appendix G to help the respondent anticipate the level of work required on the review rounds. 7.4.7 Test Administration Ancillary Materials For the purpose of this ITN, test administration “ancillary materials” refers to printed or online products other than manuals produced by the contractor. Ancillary materials are developed with review rounds and email approval. Some ancillary materials will be printed and shipped with test materials (e.g., document count forms, return labels, paper bands). Some will be included as perforated pages in the manuals (e.g., security agreements, chain of custody form, blank administration record/security checklist, security log). Some will be developed and provided in an online format only (e.g., comment forms). This section contains details regarding the ancillary materials required for administration. These materials are designed to ensure overall security, efficient and secure handling of materials, and standardized administration activities. In addition to the materials listed here, the contractor will provide, based on Department request and approval, a number of memoranda to assist in communications with districts regarding test administration processes and requirements. The contractor will provide the required publication production and software development resources to produce the ancillary materials using current staff that have the necessary expertise and serve in such capacities on a full-time basis, through temporary staffing, or by subcontracting. Respondents should identify in their replies the staff who will be assigned to develop the ancillary materials and/or subcontractors proposed to complete these project tasks, as well as their qualifications (education and experience) to do so. Most of the products identified in this section will be provided in formats that permit them to be accessed electronically via the internet on both the contractor’s and Department’s websites. Posted products must be available for viewing and downloading and must be provided in an ADA-compliant format, that is, a format compliant with Section 508 [29 U.S.C. § 794(d) & 36 CFR Part 1194] of the Rehabilitation Act, as amended, and s. 282.601, et seq., F.S. The Department will run standard Section 508-compliance checks on all documents to be posted on the Department website to ensure their accessibility. The contractor will be required to modify any noncompliant elements and provide compliant documents. The contractor is responsible for securing all copyright requirements, if applicable, for materials submitted for nonsecure web posting. (See Appendix E, Part 2 for additional information about Department guidelines for Section 508 pdf publications.) The contractor will work with the Department to identify the specific requirements for files to be made accessible on the online management system and will provide the required files in a timely manner. The contractor is responsible for developing the following materials for distribution to districts for each administration. These materials will be produced following the quantities and specifications identified in Appendix E. The contractor is also required to provide various other materials required to implement administration activities, including forms, labels, and documents. 7.4.7.1 Training Resources for School and District Technology Coordinators (CBT) The contractor must provide training materials to support CBT test administrations. These materials may include screen-shots and will be delivered via narrated PowerPoint web presentations, WebEx, training module, or other similar mode through the CBT test platform. Florida Standards Assessments, ITN 2014-47 46 These training materials will consist of an outline of key issues to be covered during school and district technology coordinator training with explanations accompanied by full-color graphics, including screen-shots, representing critical elements or live demonstrations of processes in the online management system. All training materials are to be made available to districts in a Section-508 compliant electronic format on both the Department’s and contractor’s website within seven (7) days of Departmental final approval of the web-tagged files. 7.4.7.2 Instructions and Materials for Braille Tests The contractor must develop Braille scripts and Braille notes for the administration of Braille tests for each subject and grade. The braille notes are secure materials and must be handled accordingly. The Braille notes and Braille scripts are distributed with the secure test materials, and they must be approved by the Department prior to production. In addition, some items may require manipulatives to allow access for visually impaired students. These manipulatives are provided, prepared, and shipped to the districts by the contractor. 7.4.7.3 Student Preidentification Labels The contractor will produce student preidentification labels for each student to be tested on a paper-based assessment for whom preidentification data were provided by the districts. (See Section 7.5.2 for more information about the preidentification process.) Labels will be provided for each student answer document as needed. Local school staff will be responsible for placing labels on student answer documents. • One label for each answer document not already preprinted • Sorted as specified by each district (may be different for each grade/subject and different for schools within a district) • Sorted with page breaks as specified by the district/Department • 3” x 3 ½” adhesive labels or approved equivalent • A supplementary supply of labels will be produced for spring administrations (Wave 2) based on PreID information gathered in January 7.4.7.4 Student PreID Rosters The contractor will provide a preidentification list containing all information included on the student PreID labels for each school in a format approved by the Department. The list is produced from the same file used to print student PreID labels and contains the same information. (See Section 7.5.2 for more information about the preidentification process.) One row of information per student will be used to verify the accuracy of student demographic information. Rosters will be provided in hard copies to schools and electronically via the contractor’s secure website. 8 ½” x 11” paper is preferred, but the Department may approve 8 ½” x 14” if this is required for readability. 7.4.7.5 Document Count Form (DCF) The contractor will provide Document Count Forms (DCF). The DCF is a pregridded, scannable, 8 ½” X 11” document used to inventory answer documents returned by each school. The form specifies the subject and grade level test placed underneath it and ensures that the correct number of documents is scanned. Forms must be preidentified with district/school names and numbers, and blank forms will also be sent as this information may also be hand-gridded. The district/school number should match the numbers on the answer documents underneath them, and the number of documents gridded on the DCF should match the number of documents beneath it. If discrepancies are discovered during scanning, the contractor will work to resolve them, as described in the Scoring and Reporting Specifications. Florida Standards Assessments, ITN 2014-47 47 If the respondent employs a different acceptable method/form for the process described above, the Department and the contractor will agree on the final process during the negotiation phase. 7.4.7.6 Administration Record/Security Checklist Security barcode numbers of all secure materials listed in ascending order by product, will be provided to a school/district with space to indicate assignment and return of materials. The Department expects that the checklist will also include box numbers and section breaks between packages and will be provided electronically in an editable format to districts. If the respondent employs a different acceptable method/form for the process described above, the Department and the contractor will agree on the final process during the negotiation phase. 7.4.7.7 Comment Forms The contractor will work with the Department to create the following ADA-compliant online forms that will be posted to the contractor’s website for each administration: • Test Administrator Comment Form (One for paper-based and one for computer-based) • School Coordinator Comment Form (One for paper-based and one for computer-based) • District Coordinator Comment Form (One for paper-based and one for computer-based) • Technology Coordinator Form (CBT only) • CBT Student Comment Form The final output of the comment form responses is described in #26 under Section 7.4.1. 7.4.7.8 Return Shipping Labels The contractor will provide preprinted, adhesive, color-coded (as required by the Department) labels for district use in returning materials. 7.4.7.9 Materials Return Kit – District The contractor will provide envelopes containing return shipping labels, bills of lading, return instructions memorandum, and Braille notes and scripts, if applicable. 7.4.7.10 Materials Return Kit – School The contractor will provide envelopes containing document count forms, paper bands, and colorcoded return shipping labels. 7.4.7.11 Miscellaneous Return Materials The contractor will provide envelopes and boxes necessary for packaging large-print, Braille, and other special program and special format materials for district use in returning materials. 7.4.7.12 Paper Bands Printed paper bands, preferably 4” x 24” adhesive, will be provided by the contractor for bundling scorable materials and provided to schools and districts. The adhesive must not cover the entire width of the band to avoid adhering to the documents contained in the band. The bands must be printed on one side to identify the type of answer document in bundle. Bands must be 8pt. Carolina paper or approved equivalent. In addition, paper band quality must be suitable for filling in information lines on band with multiple pen/pencil types (e.g., markers, ball point pens, pencils) without smearing/smudging. If the respondent employs a different acceptable method for the process described above, the Department and the selected contractor will agree on the final process during the negotiation phase. Florida Standards Assessments, ITN 2014-47 48 7.4.7.13 Rulers / Manipulatives / Calculators If any test items require the use of a ruler or manipulative, the contractor will produce/provide these materials for testing in a reliable, cost-efficient manner. Respondents should describe proposed functionality and features of calculators required for Mathematics tests and the Department will provide an approved list of calculators for student use. (Contractors will not be required to provide hand-held calculators to districts.) Any testing accessories required for computer-based tests must be built into the platform and must also be available for paper-based tests. If materials not listed in this ITN are needed for testing, they should be clearly identified in the respondent’s reply and the Department will specify its requirements during the negotiation phase. 7.4.7.14 Miscellaneous Memos, Forms, Labels, and Other Products The contractor will provide the following products: • Miscellaneous memoranda to the district coordinators may be necessary for each administration. Letters and memoranda will be of a length to be determined by the Department, up to five (5) pages each, and length of documents may change at each production round. • Cover memoranda for all shipments of materials indicating what is being shipped and how it is packed. • Packing lists must be provided with all shipments (district coordinator receives a hard copy of the district and all school packing lists and receives all lists electronically as soon as they are available). • Pallet lists (maps) must be provided to any district that has materials on pallets. • Other miscellaneous memoranda, labels, forms, etc., will be required to implement the program. 7.4.8 Online Management System The contractor’s online management system described in this section will be the primary resource for all student data management (paper-based and computer-based), as well as test administration and reporting tasks. The expectations of this ITN are written with the full understanding that capabilities for an online management system are expanding quickly. The Department is interested in reviewing innovative procedures and techniques for managing student assessment online, which take advantage of the maturation of technology in educational assessment through the life of the contract arising from this ITN. Therefore, the respondent may submit a reply that varies from the expectations herein as long as the solution meets the needs of the Department and the intent of the ITN. Respondents should agree to work with the Department to amend the specifications, expectations, and requirements to ensure that the Florida testing program keeps pace with the expanding capabilities of test delivery and management via computer. Any such variations will be resolved and agreed to during the negotiation phase. The online management system should be a secure site that requires usernames and passwords and accounts should be customizable based on the access a user will have (state-level, district-level, school-level, etc.). Depending on the access level, a user should be able to view other organizations in the system (i.e., a state-level user may select and view any district or school in the state) and have the ability to make changes or alter data in those organizations. The Department will provide the contractor with a list of the current names, addresses, email addresses, and phone and fax numbers of the district assessment coordinators. The contractor will populate the passwordprotected online system with this information to create initial district-level user accounts. New accounts should be able to be created individually or via file upload to create multiple new accounts or make updates to existing accounts. Florida Standards Assessments, ITN 2014-47 49 The system should allow districts to upload enrollment counts, preidentification files, and to place additional orders for materials. Using the Department’s Master School Identification (MSID) file, the contractor will populate the system with all active districts and schools. The system must have the ability to add additional schools that are not on the MSID file (e.g., private schools), and these schools may require special school types as determined by the department. Components of the contractor’s online systems (e.g., websites, databases) must be submitted to the Department for review of content, layout, aesthetic quality, and functionality. The contractor will make any requested changes to such systems. The online systems may not be used for contractor advertisements or other promotional purposes. The “branding” of these systems should be primarily for Florida’s assessment system, but may include contractor logos and other information as approved by the Department. The system should also accommodate CBT delivery. All systems that identify, prepare for, and monitor student test participation must have the capacity to distinguish between paper-based and computer-based tests, including all applications of preidentification and enrollment files. These systems must also be able to accurately track student participation by delivery mode (paper-based, computer-based, or both) and reflect the test delivery mode(s) in each student file. The system should be designed to provide user access to various non-secure and secure links including: • • • • • • • • • • • • • • • • • • • general system introduction and descriptive information; updates to current operational status and special situations; self-running software to test the suitability of computers for test delivery; self-running software to download a “secure browser” to ensure test and system security; all user guides, manuals, training, and ancillary materials in downloadable format; procedures to add, modify, or remove students or student information in preparation for testing or reporting; record test group codes and accommodations for computer-based tests; ability to invalidate computer-based tests; procedures for school and district staff to update contact information; access to an online help-desk chat system; system status website to display operability status of testing components; access to instances of all computer-delivered tests serving Florida K-12 education programs; access to order additional test materials and track shipments; access to packing lists, pallet maps, security checklists (pre-populated with security numbers of secure materials), and material summaries; results files and reports; web-based student practice tests; self-paced training resources for test setup and administration tasks; Training center to enable practice for test setup tasks and to run an infrastructure test to test capacity at testing sites; and an infrastructure test. The system must collect and maintain records for students as they move through Florida’s public school systems. An individual student's test results must be maintained in electronic archive for the life of the contract and must be returned to the Department at contract termination. Florida Standards Assessments, ITN 2014-47 50 7.4.8.1 Test Staff Contact Information It is necessary for the contractor to maintain accurate lists of district and school information and staff associated with Florida public education testing programs as specified by the Department. This information must be easy to update by school staff as information changes. This information must include each user’s security authorization level and provide an accessible log, in viewable and printable format, of each user’s participation in the system, including type of participation, date and time of participation, etc. (see also Section 7.4.1). 7.4.8.2 Test Staff Passwords The system must securely deliver and maintain system access passwords to users as necessary. The passwords must access a hierarchical permissions structure available for school, district, and state users. The contractor will work with the Department to determine expiration timelines for passwords and user accounts. 7.4.8.3 Test Session Requirements The system must have available during all steps in initiating and delivering tests: • a school look-up sub-system; • a searchable, sortable student look-up sub-system; • rosters of students in each test session. During the test session, automated backup, recovery, and restart procedures for the system must be in place. The system will recover data from any unforeseen test interruption and return the testtaker to the point of interruption. The system must also show student status and progress through the items in the test. The Department should be able to view uploaded student responses realtime while students are completing the assessments in the event of an unforeseen test interruption. Contractor and department users must have the ability to restart a submitted test. The tests should only be available during specified administration windows. The features and interface of the system may be customizable, and the contractor will consider requests for customization from the Department. 7.4.8.4 Security Requirements Security of tests and individual test results must be maintained. Access to student test results must be restricted to authorized Department staff and school district personnel based on a secure, encrypted, password protected system. The system must assure proper identification of each student and the accurate matching of the student to his/her test results. Preferably, the system must ensure that a student can take each computer-based test just once per login unless authorized by the department. If an error is found (e.g., student tests with the wrong login information) the system must have the ability to link the test to the correct student profile. The ability to make changes to data or processes in any part of the system must be based on a password-protected, hierarchical permissions structure. Respondents should respond to each of the following items: • Describe the security controls over all system aspects. • Describe the “levels” of security provided in the system. • Describe how the ability to limit access to specific system functions or modules is provided. • Describe the log-in security method and assurances to maintain individual confidentiality of test results. • Describe the authentication process to ensure that an individual is the person logging into the system (e.g., the student taking the test is who s/he says s/he is). Florida Standards Assessments, ITN 2014-47 51 The contractor and Department staff will create a report that includes a checklist of all operable aspects of the system to be used each invoice period. The contractor will prepare the report listing any malfunctions or unavailability of the system for Department signoff each period. 7.4.9 Computer-Based Test Administration Overview The expectations in this ITN are written with the full understanding that computer-based testing (CBT) capabilities are expanding quickly. The Department is interested in reviewing innovative procedures and techniques for item development, test delivery, scoring (see Section 7.5.4), and reporting (see Section 7.5.5) which take advantage of the maturation of technology in educational assessment through the life of the contract arising from this ITN. Therefore, the respondent may submit a reply that varies from the expectations herein as long as the solution meets the needs of the Department and the intent of the ITN. Respondents should agree to work with the Department to amend the specifications, expectations, and requirements to ensure that the Florida testing program keeps pace with the expanding capabilities of test delivery and management via computer. Any such variations will be resolved and agreed to during the negotiation phase. The computer-based test system should be capable of being customized to meet the needs of Florida students, school and district personnel, and the Department. The reply should indicate procedures used to ensure that the computer-based tests accurately assess students possessing a diverse range of physical, sensory, and cognitive abilities; the system should ensure adherence to universal design concepts. The system should be easy to use for all participants, intuitive, and deliver a consistent “look and feel” for all computer-delivered test programs provided by the Department. This includes item layouts, navigation tools, and administrative controls such as managing participant demographic and performance information, as well as CBT setup activities. The contractor will modify any proposed system to achieve these objectives. The contractor and Department staff will create a report that includes a checklist of all operable aspects of the system to be used each invoice period. The contractor will prepare the report listing any malfunctions or unavailability of the system for Department signoff each period. If the Department determines that the computer-based system is not performing adequately, exhibits malfunctions that may significantly affect the student testing experience, or is unable to meet the requirements listed in this ITN, the contractor will propose a solution to the Department, including providing paper-based tests at no additional cost, that allows for students to participate in the assessments and for the Department to gather the necessary data. 7.4.9.1 Hardware and Software Requirements The system should: • Support both wired and wireless connections to the system and all eligible devices that meet the specifications in the table in Section 7.4.9.2. The respondent should also commit to continued support for future updates similar to those made available on the latest version of Partnership for Assessment of Readiness for College and Careers' (PARCC) Technology Guidelines that can be found at (http://www.parcconline.org/technology). Other assistive technologies may be needed for students requiring accommodations. • Support eligible devices with capabilities to “lock down” the device to temporarily disable features, functionalities, and applications that could present a security risk during test administration. Features that will need to be controlled during test administration include, but are not limited to, unrestricted Internet access, cameras (still and video), screen capture (live and recorded), email, instant messaging, Bluetooth connections, application switching, and printing. The contractor will develop, deliver, and continuously improve capabilities to “lock down” the device for the applicable operating systems and browsers. Florida Standards Assessments, ITN 2014-47 52 • Support the current version within six months of release and two prior versions of the web browsers listed below (as of June 2013). Web Browser Support List: Apple Safari Google Chrome Microsoft Internet Explorer Mozilla Firefox 7.4.9.2 Eligible Device Specifications At the time of the development of this ITN, the device specifications that are recommended for Florida’s public schools are provided below. The respondent’s platform for computer-based testing should be able to perform, at a minimum, on devices that meet the specifications in Tables 7.4.9.2.A and 7.4.9.2.B. Table 7.4.9.2.A. Device Specifications for Desktop, Laptop, Netbook, and Thin Client/VDI Computers Table 7.4.9.2.A Device Specifications for Desktop, Laptop, Netbook, and Thin Client1/VDI Computers Operating System Windows Specifications Windows 7 or newer Mac OS Mac OS X 10.7 or newer Linux Linux: Ubuntu 11.10, Fedora 16 or newer Chrome OS Chrome OS 19 or newer Memory 1 GB RAM or greater Connectivity Screen Size Screen Resolution Input Device Requirements Headphone/Earphone and Microphone Requirements (Some student accommodations may require headphones/ earphones (e.g., text to speech). Additional Guidance Computers must be able to connect to the Internet via wired or wireless networks. 9.5 inch screen size or larger 1024 x 768 resolution2 or higher Keyboard Mouse or Touchpad or Touchscreen The input device must allow students to select/deselect, drag, and highlight text, objects, and areas. The input device must allow students to enter letters, numbers, and symbols and shift, tab, return, delete, and backspace. To meet security guidelines, each Bluetooth/wireless keyboard must be configured to pair with only a single computer during assessment administration. Other assistive technologies may be needed for students requiring accommodations. Headphones/Earphones Microphone 1 Each computer operating in a thin client environment must meet or exceed minimum hardware specifications, as well as bandwidth and security requirements. 2Computers must accommodate the 1024 x 768 screen resolution minimum without panning. Some netbook computers may have screen resolutions slightly less than the 1024 x 768 minimum, yet may meet all other minimum requirements. Depending on netbook model specifics, school technology administrators may be able to reset screen resolution to meet guidelines. Florida Standards Assessments, ITN 2014-47 53 Table 7.4.9.2.B. Device Specifications for Tablets Table 7.4.9.2.B Device Specifications for Tablets Operating System Android Specifications Android 4.0 or newer (with 1 GB RAM or greater) Apple iOS iPad 2 or newer running iOS 6 or newer (with 512 MB RAM or greater) Windows Memory Connectivity Windows 8 or newer (with 1 GB RAM or greater) By operating system Tablets must be able to connect to the Internet via wired or wireless networks. Screen Size 9.5 inch screen size or larger Screen Resolution 1024 x 768 resolution2 or higher Input Device Requirements Keyboard Touchscreen or Mouse Due to the onscreen space occupied by a tablet’s virtual keyboard, assessments will require external keyboards for test takers using tablets so as not to limit or obscure the view of test item content and related functionalities when text input is required. External keyboards must allow students to enter letters, numbers, and symbols and shift, tab, return, delete, and backspace. Tablet touchscreen interfaces can be used for student interactions with the assessments other than text input, including to select/deselect, drag, and highlight text, objects, and areas. To meet security guidelines, each Bluetooth/wireless keyboard must be configured to pair with only a single computer during assessment administration. Headphone/Earphone and Microphone Requirements (Some student accommodations may require headphones/ earphones (e.g., text to speech). Other assistive technologies may be needed for students requiring accommodations. Headphones/Earphones Microphone Florida Standards Assessments, ITN 2014-47 54 7.4.9.3 System Performance Requirements The system must demonstrate and report that it can meet the following performance requirements in a controlled Internet access environment, requiring a bandwidth connection of no more than five (5) kilobits per second (kbps) per simultaneous test taker using a caching solution or fifty (50) kbps per simultaneous test taker without caching: • deliver less than one second mean screen refresh rate for 90% of all users; and, • ensure that no users have a refresh rate of greater than five (5) seconds. The respondent should describe the ability to support a substantial number of simultaneous online users of the system (up to fifty percent (50%) of the current Membership in Florida Public Schools as determined through the most recent Survey 2 data under the Education Information & Accountability Services (EIAS) found at http://www.fldoe.org/eias/eiaspubs/pubstudent.asp). The respondent should consider any services provided to other states when presenting this information. The contractor must continually monitor and report to the Department the number of simultaneous online users of the system during student testing periods. 7.4.9.4 Connectivity Requirements The respondent should consider existing and planned communications infrastructures when proposing its system. Preferably, school, district, and Department technology architecture and computing hardware should not have to be replaced. 7.4.9.5 Test Site Set-Up/Certification Requirements The contractor will develop specifications for, and deliver a system to operate/deliver online tests and that serves as: • a secure browser, which both [a] locks access to all other computer and web programs, applications, and files, and [b] masks and directly accesses the testing host web site; and, • a system check, monitoring required computer software and connectivity readiness for testing and which can be run from individual machines. The contractor will be responsible for assisting schools in certifying that computers are properly prepared and can make the necessary Internet connections to successfully deliver tests. The contractor will develop and make available an infrastructure test, consisting of mock items, that replicates the size of the largest CBT and is run on the actual testing platform (i.e., users log in to the testing platform as if they were logging into a live test). The contractor will assist the Department in developing a guide to walk users through running the trial on machines that will be used for testing to help them identify technical issues and make adjustments prior to live testing. The respondent should describe the proposed test site set-up, certification, and system check measures in the reply. THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK Florida Standards Assessments, ITN 2014-47 55 7.4.9.6 Computer-Based Test Presentation Requirements Table 7.4.9.6 CBT Tools and Features The following table lists required and preferred tools and features in the CBT platform. Accommodated features are required. More information about the test accommodations that must be provided for students are provided in Appendix H. Table 7.4.9.6 CBT Tools and Features Tool Description Required Features Calculator Computer-based calculators will be available to students for the mathematics tests. Numbers and functions on the on-screen calculator will be activated by mouse-click on the displayed button, or by keystroke of the appropriate keyboard key. User-acceptance testing will be required to assure that calculations are performed in the same sequence and with the same rounding rules applied as the hand-held calculators. Respondents should describe proposed functionality and features of calculators to be included in computer-based test forms. Reference Sheet The system must be capable of displaying a pop-up reference sheet for mathematics tests. This will be capable of being enlarged and the Department prefers that, when open, the reference sheet not obscure portions of the test item. Ruler If it is determined that some mathematics test items require the use of a ruler, a pop-up ruler will be available to students. The ruler will be easily moved and rotated on-screen by the student. Preferred Features Straight Edge A pop-up straight edge, without measurement increments. The straight edge will be easily moved and rotated on-screen by the student. Option Eliminator A feature for eliminating options. However, the feature cannot eliminate the option by placing any mark on or in close proximity to the bubble that would be used for marking a response. It must allow the student to easily read the option after it has been eliminated. Students must be able to undo the elimination. Bookmark Mark test items for review and navigate easily to return to previous items, including those bookmarked for later review. Bookmarked items should be visible on an item summary page at the end of the assessment. Highlighter Highlight text on all tests and to erase the highlighting. If a student highlights in a reading passage, the highlighting should remain through all items related to that passage until the student erases it. Notes Allow students to create electronic notes for reading passages if desired. These notes must not hinder reading of the passage. Notes should travel for each passage and item associated with that passage, then refresh with each new passage. Footnote Pop-up Reading items require a pop-up footnote. That is, if the student clicks on a footnote number occurring in the passage, the text for the footnote appears. Footnotes must also appear at the end of the passage. Florida Standards Assessments, ITN 2014-47 56 Table 7.4.9.6 CBT Tools and Features Tool Description Student Tutorials/Practice Sessions Each student that takes the computer-based test must have the opportunity to learn how to use the tools, select/enter responses, and navigate the system in a short practice session with instructions that may be either teacher-directed or independently conducted, as described in Section 7.3.7.5. The contractor will produce and make available a web-based practice test for each assessment. The contractor may also provide access to a selectable practice test at the beginning of any test session within the platform. Accommodated Features (Required) Variable Font The system must be able to display items and text using variable font sizes. For example, footnotes for words in passages must appear smaller at the end of the passage than the text used in the passage. AdministratorSelectable Zoom The system must be able to Zoom in increments from approximately 12 point to approximately 72 point available for the student to zoom or for the administrator to preset. Graphics must be able to be enlarged also. Any buttons, tools, the pointer/cursor, and the scrollbar must also enlarge based on the zoom level selected. AdministratorSelectable Large-Print Font The system must allow for selection of a large print font of at least 14 point font and up to 72 point font. The student must then be able to zoom as indicated above. AdministratorSelectable Variable Font and Background Colors The system must allow for selection from a pallet of colors for font and background colors. The Department prefers fonts of black, dark blue, light blue, pink, and yellow with backgrounds of white, black, brown, or dark blue. Other colors may be recommended. Any buttons, tools, the pointer/cursor, and the scrollbar must also change based on the colors/fonts selected. AdministratorSelectable Screen Reader Provide an integrated (preferably password-enabled) selectable screen reader capable of reading via headset to the student, all text, mathematical symbols, and text within graphics. The contractor must ensure that the screen reader has one consistent voice and is not operating-system specific. The contractor must run quality control checks on all products, test forms, graphics, and items to ensure the screen reader pronounces words correctly. AdministratorSelectable Assistive Devices Integration Support assistive/adaptive devices commonly available to support computer input and interaction to persons with disabilities. 7.4.9.7 Computer-Based Test Screen Layout Requirements The contractor is required to recommend a set of screen layouts for the presentation of computerdelivered test items. (See also Sections 7.3.11.2 and 7.4.8) These layouts must include: • prominently displayed student identifying information [name, student ID, etc., as specified by the Department]; • icons to access the item tools/references noted above; and • icons presented with consistent location and look and feel across all screen layout options. Florida Standards Assessments, ITN 2014-47 57 Additionally, the system must code and capture in the item bank the basic screen layout used to deliver the item on computer to students. 7.4.9.8 Training Delivery Requirements The contractor will develop specifications for and produce a training plan to provide initial training and supply materials to support on-going training for audiences including, but not limited to: • the Department K-12 Student Assessment program team; • other Department staff; • district assessment coordinators; • school assessment coordinators; • school and district hardware/network technical staff; and, • school and district test administrators. The Department prefers the contractor to deliver in-person training to approximately 500 key Department and district personnel in as many as ten (10) sessions of not more than one-half (½) day each, held throughout Florida prior to the first implementation of the new computer-delivery platform, which will occur in the spring of 2015. Additional training should be provided via WebEx or similar web-delivered, interactive sessions scheduled at least six (6) weeks in advance of the opening of each online assessment window. These training requirements are in addition to the training materials described in Section 7.4.7.1. 7.4.9.9 Technology-Supported Item Types The Department is interested in leveraging empirically-validated item formats which take advantage of the capabilities of computer technology. Examples of these item types may include: • Drag-and-Drop – in which students select from multiple possible responses to select one, via mouse-click or key-stroke combination, and then “drag” it to insert (drop) it in an indicated blank to correctly complete a mathematics formula, text-editing task, etc. This process may be repeated for the same test item so that multiple answers are placed into multiple blanks. • Rotate and other image transformation items. • Other formats as the contractor may recommend or the Department may request. • The respondent may include the capacity to support these types of items in the reply. 7.4.10 Customer Service and Technical Support The contractor will provide customer service via toll-free phone and email between 7:00 a.m. and 8:30 p.m., Eastern Time. The service will be staffed appropriately, as needed, with personnel designated to respond to assessment program calls and perform other Florida assessment program tasks specific to the assessments detailed in this ITN. Customer service staff must document all communications in a log and furnish it daily to the Department during each administration or upon request as needed. The contractor will develop a call topic matrix, with the Department’s guidance, to ensure that calls are responded to in a consistent and appropriate manner. Beginning not less than four weeks before and extending one (1) full week after each test administration, the contractor must use additional customer service personnel to handle computer-based test administration calls, and personnel who are able to provide advanced technical support must be immediately available for any issues that cannot be resolved by call center staff. The Department is interested in systems which provide help desk functionality via telephone and Internet “chat” systems. The system should employ a Department auditable job-ticket method of tracking issue types, status, and individual client trouble calls/emails including current aggregate status information for resolution. Florida Standards Assessments, ITN 2014-47 58 7.5.0 Scanning, Scoring, and Reporting This section of the ITN provides a description of the scanning, scoring, and reporting requirements for this project. Section 7.5.2 provides details for preparing school and student information, and Sections 7.5.3, 7.5.4, and 7.5.5 provide the requirements for the scanning, scoring, and reporting, respectively. The steps necessary to verify the accuracy of these procedures are described in Section 7.5.6. The contractor must assign experienced staff to complete the scanning, scoring, and reporting tasks and to coordinate with the test development, the test administration, and the psychometric staff during various phases of the scanning, scoring, and reporting. As described in Section 7.7.2, it is anticipated that one of the four project managers will focus on scoring and reporting and one will focus on computer-based testing. The one-page vitae for each of the professional personnel must indicate their scanning, scoring, and reporting experience within the organization to ensure the critical and extra-critical dates may be met. The contractor’s scoring and reporting staff must coordinate with the development team to ensure the file layouts accurately reflect the final tests; with the test administration team to obtain information about testing procedures, schools, and students; and with the psychometric staff to ensure timely file verification and transfer during the scoring and reporting process. The contractor must verify the accuracy of all files and reports prior to submission to the Department for review. The contractor will develop staffing plans to complete its review of files and reports and then provide the Department time to complete its review. Because of the secure nature of the personally identifiable student information that is used for this assessment program, stringent security measures must be in place for the Department and the selected contractor. See Appendix I for more information on these measures. 7.5.1 Scoring and Reporting Specifications In April of each year, the contractor will submit the first draft of the Scoring and Reporting Specifications for all tests within a school year. These specifications provide details to ensure accurate and timely scanning, scoring, and reporting for the following school year. The specifications are initially drafted by the contractor using the specifications from the prior school year, updating procedures or products as instructed by the Department. This initial draft includes anticipated schedules for planning the next school year, details on file naming conventions, as well as the scanning, editing, scoring, and reporting procedures. These specifications do not contain confidential information or exact replicas of products; they provide the general requirements and processing steps using information from the prior year to assist in planning. If the initial draft needs extensive editing, a second draft must be submitted by the contractor. These specifications are used to communicate between the contractor and the Department throughout the school year with details added as necessary during the year to ensure that procedures and products are sufficiently described and any changes in procedures are documented. The contractor will anticipate having additional documentation for its internal communications. At the initial planning meeting, the Department will provide the contractor with the 2013-2014 Scoring and Reporting Specifications with a list of updates for the contractor to make for the initial draft of the 2014-15 specifications. The reply should clearly indicate how the scoring and reporting staff will coordinate with the other staff members to enable the contractor to develop the Scoring and Reporting Specifications each April for the following school year while at the same time conducting the current year’s spring reporting activities. During the school year, the contractor will maintain an addendum to the specifications noting updates, issues, or changes to the specifications. This specifications addendum will be used to document changes made in the specifications during the school year and to update the specifications for the next school year. If necessary, the contractor may be requested to provide the Department with updated versions of the specifications throughout the year. The contractor will submit the final specifications addendum at the same time as the last closeout file of student results for that school year. Florida Standards Assessments, ITN 2014-47 59 After the initial draft of the specifications has been reviewed by the Department and any edits or comments have been incorporated by the contractor, development of file layouts and other procedures may begin for the first administration in that school year. See Appendix B for the critical and extra critical dates associated with the reports described in this section. 7.5.2 Preparing School and Student Information (MSID and PreID) 1. To aid in the accuracy of reporting results, before the initial wave of reporting the contractor must perform checks to be sure that school names and numbers are accurate and that each student’s scores are reported to the correct school. Due to the numerous types of special schools (e.g., home schools and virtual schools) and numerous students testing in schools that are different than the schools where their scores are reported, school numbers must often be updated after scanning. The contractor must provide staff necessary to make updates to student information and school information between scanning and reporting when conflicting information has been provided. The contractor will assist the Department and districts to ensure that each student’s scores are reported to the correct school during the initial wave of reporting. 2. The Department maintains a Master School Identification (MSID) website with active school information. The Department will determine dates that the contractor will download information from the MSID website to be used for each administration. Due to multiple waves of reporting for each administration, as well as reporting annual EOC results, it is important to use consistent school information and to report scores using the information from the MSID website. The Department will provide the contractor with any changes or updates to any information on this site, such as changing a school name for the annual reporting. The contractor will not make changes to school information or add/delete schools unless directed by the Department. 3. To reduce the amount of information a student must provide during testing, for each administration the contractor will obtain student information from each district prior to testing. This preidentification (PreID) information includes student demographic information, requests for special print materials, and school numbers of the school where the student will test and the school where their results will be reported. In the past, the Department has obtained PreID information in a text file from each district during one time period each fall and summer, and two time periods each spring. The PreID information was then loaded into the CBT system or used to create adhesive PreID labels for answer documents. The reply should include a description of how the contractor would obtain and update the PreID information to provide the most accurate results. Respondents may propose other options to adhesive labels, such as pregridding (preslugging) answer documents. The Department wants to obtain the most cost efficient methods that will ensure accurate results. 4. The Department will determine the dates the districts will be permitted to submit PreID information, the number of PreID windows, how and which information may be updated, the rules for accepting or rejecting records or files, and the final method of preidentifying the answer documents. 5. The contractor will draft the PreID file layouts and the PreID instructions for each administration and incorporate changes as requested by the Department. The Department will send the final PreID instructions and file layout to the districts. The contractor will also post the layout on the online management system as instructed by the Department. 6. The contractor must complete numerous steps to ensure the accuracy of the PreID information submitted by each district, and the contractor must work with district staff when necessary to ensure that the PreID files are properly uploaded in a timely manner with just one unique record for each student in the district. a. First, the contractor must be sure that only schools that are active in the MSID file or approved by the Department (e.g., a participating private school or community college) are used in each district’s PreID file. Then, the contractor must ensure that each field in the PreID file submitted by the district Florida Standards Assessments, ITN 2014-47 60 is completed with valid characters as described in the layout and that the values are reasonable. For example, if all records submitted for Algebra 1 are coded as first-time takers in a district or all are coded as female, the contractor must alert the district and permit updates as necessary. Also, the contractor must compare the numbers of students in each district to the number of students tested in the previous comparable administration to identify any issues with the number of records in the district’s file and take corrective action as necessary. The reply should include a description of the system it will use to ensure the accuracy and reasonableness of these files. b. Two days prior to the close of each PreID window, the contractor will contact the districts that have not successfully submitted a PreID file and aid the district to ensure each district is able to submit its data. c. During the PreID window, to assist district staff in reviewing the files as read by the contractor, the contractor creates PreID Frequency Reports with n-counts for various demographic groups for each district to review in the online management system. The contractor will also generate a PreID Frequency Report with state totals for the Department to review. d. The district may submit additional files to make corrections during the PreID window. The contractor will draft the format of these reports and make updates as provided by the Department. The reply should include examples of the types of reports that might be used to aid districts in reviewing the PreID data and methods to make updates and corrections while ensuring the final files include one record per student with accurate information. e. After the PreID window closes, the contractor provides the Department with a state PreID file containing all the accepted records that were submitted by the districts. This file will be in the same format as the files provided by the districts, one record per student, except with a unique PreID Number, generated by the contractor, for each student. The PreID Number is used for each test the student is assigned within that administration and the PreID Number is unique across administrations for each school year. For example, if a student in grade 8 is taking ELA/L and Algebra 1 in the spring, that student will have one PreID number for both spring assessments, and that number will not be used again during that school year. f. If a student‘s school is incorrect in the district’s PreID file in (e.g., student moved), school or district staff will grid the student information on the cover of the answer document or add student information using the online management system prior to testing. g. If a student will be tested in a school other than the school used for reporting, PreID labels are sent to the school where the student will test or the CBT record is set up where the student will test. This is called the Alternate District/School Number. h. The contractor will print PreID Student Rosters and include them with the shipments of PreID labels (or preidentified answer documents) to districts. These rosters list the information on the PreID labels and an explanation of the codes used on the PreID labels. The contractor will provide hard-copy PreID Student Rosters with the PreID labels and electronic copies on the contractor's secure website as specified in the Scoring and Reporting Specifications. Each spring, via a second PreID window to provide updates closer to testing, the contractor will provide a supplemental shipment of PreID labels and rosters to districts that update their PreID files. i. As permitted by the Department, districts may subsequently edit some of the information using the online management system or add new students for the CBT system. Students who were not included in the PreID files that are taking paper tests will not be loaded into the PreID system, and PreID labels are not sent for new students taking paper tests. Florida Standards Assessments, ITN 2014-47 61 7.5.3 Scanning/Editing PBTs and Extracting CBT Records Scanning and editing requirements in this section refer to the PBTs. The contractor’s CBT system must be capable of creating a file with each student’s information and the exact responses entered by each student, including the fill-in response and performance tasks, character by character before editing, similar to the scan files described in this section for the PBTs. The reply should include a description of how the scanning and editing procedures in this section will be completed, as well as a description of the types of CBT extract files that will be available for the Department prior to the creation of the scored files described in the next section of this ITN. 1. For each administration, the test administration manuals describe in detail the test materials provided to districts and schools and the procedures for returning the materials and completing information in the CBT system. The contractor will establish procedures to verify that the district and school staff completed these procedures correctly. The contractor will work with the Department and districts if any anomalies are found and make corrections as necessary. This includes mistakes made by districts such as returning to-be-scored answer documents to the wrong site or miscoding a school number on a document. 2. The reply should include a description of where the answer documents will be scanned, the types of scanners to be used and their capacity during the time this project will be scored, general editing and proofing procedures of documents scanned, the types of documents and procedures the contractor proposes to ensure that the correct number of answer documents have been received and scanned to be scored. In the past districts packaged to-be-scored answer documents with a Document Count Form (DCF header) on top of documents of the same type. Alternate methods to ensure that the correct number of answer documents is scanned from each school may be proposed. See Section 7.4.1 for more information on the return of materials. 3. The contractor will provide the Department with scan files for the PBTs and the extract files from the CBT system prior to creating the scored files. The contractor will draft layouts for these scan and extract files and incorporate changes as requested by the Department. The Department will specify the fields in the file, valid values for the fields, and the contents of each file (e.g., by grade or by subject). 4. The contractor will provide two versions of each scan file. The original scan file contains the exact information as scanned and the second, the edited scan file, contains the final information from the scanning system incorporating any edits made as described in the Scoring and Reporting Specifications. For example, if the student wrote her name on the top of the answer document, but did not grid the bubbles on the document, the original scan file would have a blank in the name field and the edited scan file would have the name as entered by the contractor based on the hand-written name. The edit rules are designed to ensure a student will receive the correct report during the initial wave of reporting. The reply should include a description of how the edits would be made to original scan file data to ensure accurate and timely reporting of results. If necessary, the contractor will update the edited scan files to ensure the correct information is transferred to the first version of the State Student Results file (SSR). 5. If any pages with responses cannot be scanned, all responses key entered into the system must be checked by a qualified person other than the person who initially key entered the responses. 6. The contractor will maintain edit logs that indicate why records on scan files were edited and by whom. The logs will indicate the names of anyone who key entered or verified responses. 7. The contractor will provide the original scan files and the edited scan files before scored files are created. See Section 7.5.6 for more information on the steps for verifying data and file delivery. If any edits were not made correctly, the contractor will update the edit files. See Section 7.6.2.2 for information on the records that must be scanned or extracted prior to the beginning of the psychometric tasks. The contractor’s psychometric and scoring/reporting staff must coordinate schedules and activities to ensure that all files are transferred as expected (times, names, layouts, etc.). Florida Standards Assessments, ITN 2014-47 62 7.5.4 Scoring The basic types of scored files created by the contractor and history files used and maintained by the contractor are described in this section. The contractor drafts detailed layouts for each file and then incorporates changes requested by the Department. 7.5.4.1 Types of Scored Files From the PBT edited scan files and the CBT extract files, scored files are created. Scored files that contain data for students throughout the state are called State Student Results (SSR) files. These files contain information from the scan and extract files, information from the students’ preidentification labels when used, students’ scores, and additional information regarding item (test question) data and scoring. The districts are also provided a truncated version of the SSR with just their students’ data, but no item data. This file is called the District Student Results (DSR) file. 1. Before the full-state SSRs are prepared, the psychometric staff (Department and contractor), begin their calibration and equating activities with a subset of the state records. These files are called calibration SSRs. Initially they only contain the raw score data, then after the calibration and equating, they are completed with the other derived scores as specified by the layout. See Section 7.6.0 for more information on the calibration, equating, and scaling procedures. Coordination of the details for the transfer of these files from scoring/reporting staff and psychometric staff is crucial to the timely release of reports and files. 2. Aggregated files contain school, district, and state data based on the scores reported on the SSRs. These files contain information that is reported to the schools and districts as well as additional information such as standard deviations of the scores. The files with all the data are called State Aggregated Results (SAR) files, and the districts receive a District Aggregated Results (DAR) with the data for the state and their schools and district. 3. After the initial release of files and reports, there are up to three waves of late reporting for students whose scores were not included in the original release of data, whose records had to be updated, or whose scores are released after suppression. The contractor will generate late reporting SSRs, DSRs, and provide student reports in a manner similar to the initial wave of reporting. SARs and DARs are not created for late reporting. 4. After the last wave of late reporting, the contractor creates and provides close-out SSRs and DSRs that contain the final information and scores for that administration. 5. After each summer EOC administration, the contractor will create annual SSRs, DSRs, SARs, and DARs by concatenating the SSRs and aggregating data combined for the fall, winter, spring, and summer administrations for the school year for subject tests that require multiple test administrations within a school year. The Department will specify how to handle any anomalies in reporting across administrations (e.g., a school name changed). 6. For each spring close-out SSR and each annual SSR, the Department inserts demographic information from its Survey 3 membership file of student data and aggregates the data for approximately 50 demographic groups. The Department will provide these Demographic SSRs and SARs to the contractor. From these files, the contractor will create the Demographic DSRs and DARs and post these files to the online management system for district-level access. 7. The contractor posts most files to the online management system, but may also be required to provide some files to the districts on CD/DVDs. See Table 7.5.5 for additional information. 7.5.4.2 Types of History Files Three types of history files may be used for this project. The Department will provide the contractor with the layouts and necessary history files to begin the project, and then the contractor will maintain the files and update the files and layouts as instructed by the Department. Florida Standards Assessments, ITN 2014-47 63 1. One type of history file may be used for the ELA/L Retake. Depending on policy decisions regarding scoring, the ELA/L Retake history file may be called the Alternate Passing Score (APS) file and would provide information on students who were grandfathered into a lower passing scale score requirement. Students who take the first administration of the Grade 10 ELA/L before standards are set may be grandfathered into a lower passing cut score than set for other students. If this occurs, for each administration, the contractor will use this file to determine if a student is eligible for the APS and will update the records for students who test. The Scoring and Reporting Specifications will detail the matching criteria. 2. The second type of history file will be used for the Algebra 1, Geometry, and Algebra 2 EOC Assessments for FDOE verification of the testing status (Retaker vs. First-time Tester) of each participating student. This kind of history file provides information for students beginning with the first time the subject test is administered statewide. Each administration, the contractor will use these files to determine if a student is a first-time tester or a retaker. The Scoring and Reporting Specifications will detail the matching criteria and process for using these files. Some aggregated data are based on first-time testers only, so the accuracy of the first-time status for each student is very important. 3. A third type of history file will be used to track scores across grade levels for the Grades 3-11 ELA/L and Grades 3-8 Mathematics students. Starting with the first ELA/L and Mathematics administration, the contractor will begin maintaining a file with student data for Grades 3 – 10 ELA/L and Grades 3 – 8 Mathematics, called the ELA/L and Mathematics History File. Scores for up to 7 administrations will be added to this file using the closeout files for each administration. The contractor will update the file after each administration and will provide it to FDOE at the end of the contract. The Scoring and Reporting Specifications will detail the matching criteria and process for using these files. The contractor will update these history files each administration and provide a copy to the Department. 7.5.4.3 Types of Scores and Codes on the SSR/DSRs 1. The scoring process begins with the Test Maps that are created by the contractor when a test form is created. The Test Maps are used to identify item placement in the test book and other information about the item. The Test Maps include information regarding which items are core items and field test items, which responses are correct, and the category used for subscores within each content area. 2. If the Test Map or a core item is determined to have a flaw, the Department may choose not to include an item in the students’ scores. The contractor must have procedures to provide the flexibility to edit the Test Map information during the scoring process, if necessary. 3. Excluding field test item data, subject raw scores and raw subscores are then generated by the contractor. 4. See Section 7.6.1 for information regarding the calculation of scale scores (SS) by the contractor. 5. Once the SS are determined, the score is classified in one of the Achievement Levels (AL) by the contractor. 6. Florida statute requires that passing scores be set for all grades and subjects. The contractor will report the scores as specified by the Department. 7. The ELA/L Retake may have two (2) cut scores for passing, depending on policy decisions that will be made prior to spring 2014. If so, the contractor will first determine if the student is Florida Standards Assessments, ITN 2014-47 64 on the APS file described in this section and then determine the passing status for each student. 8. For each special school, there is a defined School Type number that is used to determine which reports are prepared for a school and whether the results are included in district and state aggregated data. For example, home schools are school type 10, and they are not included in aggregated results within a district. 9. Scores are not reported for some answer documents (e.g., students who did not answer enough questions to have a score reported). The student Score Status Flag determines whether the student scores are reported and aggregated. The Scoring and Reporting Specifications provide details on these codes. 10. Many codes are related to each other and all values must be internally consistent. For example, the School Type Codes and the ESE Primary Exceptionality codes are used to determine the Curriculum Group and the Standard Curriculum Codes. The Ethnicity Code and the Race Codes are used to generate the Report Race Code. Students who use large print and Braille documents must have specific Primary Exceptionality Codes and accommodation codes that must be checked and updated by the contractor if necessary. 11. Numerous other codes (approximately 15-20 for each subject) are provided on the SSRs. These types of codes may change occasionally based on changes in the assessment program and needs of the schools and districts. The contractor must be flexible in modifying the codes and including additional codes as needed to aid in the accurate and timely reporting of useful results. 12. After the scores and codes are verified by the contractor, the first version of the full-state State Student Results (SSR) file is provided to the Department. The contractor will verify that the data were properly transferred from the corresponding edited scan file, that all scores, flags, and codes were set correctly, and that all fields contain valid values as defined by the layout. See Section 7.5.6 for additional information on the steps for verifying mock data, early district return data, and state data. 7.5.4.4 Handling Special Print Documents 1. This project includes Large Print, Braille, and One-Item-Per-Page (LP, BRL, and OIPP) special print test materials. The contractor is responsible for numerous tasks to prepare the special print documents for scoring and reporting. The scores for students using these materials are included in the initial release of reports. 2. The Scoring and Reporting Specifications will include the steps the contractor follows from the time these materials are unpacked until the scores are reported. If any to-be-scored answer documents are not packaged correctly, the contractor is responsible for properly preparing the materials or contacting the district if necessary. 3. The contractor is responsible for the transcription of the responses from the LP book to the regular print book according to guidelines provided by the Department. The contractor may suggest ways to complete this process using the CBT system to expedite the scoring process. 4. The contractor is responsible for transcribing Braille answer documents to regular print or into the CBT system. If a student uses an OIPP document, the contractor is responsible for the transcription of the responses from the OIPP book to the regular print book according to guidelines provided by the Department. Transcription services may be subcontracted, but all transcriptions must be completed in time to report the students’ results in the initial reporting. This process must be carefully monitored and transcription must be double checked by a person other than the person who initially transcribed the responses. Florida Standards Assessments, ITN 2014-47 65 5. The contractor will create a Special Document Tracking Log to inventory and track the processing of all special print documents. The contractor and the Department will determine the format of the log to track the answer document information and times for the log to be provided to the Department. 7.5.4.5 Handscoring Handscoring refers to the processes necessary for determining the rating of a student’s response on the writing prompts or on the performance-based tasks in ELA/L and Mathematics that respondents may propose. Both paper-based and computer-based tests will contain performance tasks, and so the Contractor must have the capability to score both hand-written and key-entered student responses. The Department requires handscoring processes that are reliable and valid, as well as efficient in terms of time and expenditures. This section describes procedural, design, and implementation requirements for scoring student writing and performance tasks. Respondents may propose other processes and procedures that accomplish the tasks in an equally reliable and valid manner, and are at least as efficient as those outlined below. The final processes will be agreed upon during the negotiation phase. See Appendix J for more details regarding handscoring. 1. Students’ constructed responses should be scored by trained readers using imaging technology. Respondents should provide a detailed explanation of the system that will be used. 2. Respondents should also provide a detailed description of how the security of the test items and student responses will be maintained throughout the handscoring process. 3. Trained professional readers will score written answers using criteria established by the Department in coordination with the contractor. Readers must have a college degree and must qualify to score Florida student responses by accurately scoring a series of training and qualifying papers. The readers must be monitored carefully and continuously throughout scoring for their accuracy (validity) and reliability (inter-rater agreement). In addition, team leaders must “read behind” each reader to determine the accuracy of their scoring. Readers must also be periodically given papers that have been previously scored to determine the accuracy of their scoring (validity responses). Quality control metrics will be established by the Department after contract execution. Student responses for constructed-response items in ELA/L tests in grades 3 and 10 and for Algebra 1 will be scored by at least two scorers (100% second read). For all other ELA/L and Mathematics performance tasks, responses will be scored by one reader, with twenty percent (20%) of these papers having a second reader. For these papers that are read a second time, if the two (2) readers’ scores do not match, a formal resolution process will be established in order to obtain exact or adjacent scores for quality control purposes only. 4. Handscoring may be done at the contractor’s established scoring site(s). Respondents may also propose solutions that include distributed scoring. The scoring sites must be secure facilities that employ procedures and technology to control access and maintain the security of materials. Access to the sites must be limited to full and part-time employees and authorized visitors. Secure materials must be accounted for, and computers containing secure materials must be secured from unauthorized access or copying of files. 5. The contractor should incorporate the following requirements into written handscoring specifications prior to scoring. The handscoring specifications will be a detailed guide to conducting handscoring that will be used by the contractor’s handscoring managers and the Department. Draft handscoring specifications must be delivered by the contractor for review and approval at least three (3) months before the first test administration. The handscoring specifications should include the following: Florida Standards Assessments, ITN 2014-47 66 a. Handscoring schedules. b. Site requirements (includes security and access during scoring). c. Scoring design (includes expected number of reads; groupings of items assigned to readers; assignment of first and second reads). d. Personnel (includes minimum requirements, such as education and experience, for scoring directors, team leaders, readers; number of personnel; security agreements). e. Rangefinding meetings (includes staffing, meeting procedures). f. Rangefinder review meetings (includes staffing; meeting procedures). g. Training and qualifying materials (includes listing/description of required materials; annotations for training and qualifying rounds; review and approval process). h. Validity sets (includes number and type of validity responses; transcription). i. Scorer degree verification procedures. j. Training procedures. k. Qualifying standards required of scorers before live scoring of student work. l. Handscoring reports (includes listing/description of handscoring reports; distribution). m. Scoring process (includes administration of validity sets; rules for determining and assigning resolution reads). n. Monitoring for scorer drift and other sources of scorer inconsistency and retraining (includes read-behind guidelines; use of daily and cumulative reader reports). o. Processing requirements (includes programming and design requirements for implementing online handscoring of images). p. Providing electronic copies of annotations used for training materials, calibration, and recalibration of readers, 6. Department staff must be provided on-line access to all handscoring systems and reports and will communicate frequently with the contractor throughout the scoring process. During live scoring daily handscoring status calls must be held between the Department and the contractor. 7. At the end of handscoring, the contractor must produce a report that captures at least the following results for all prompts and performance-based items and tasks that required handscoring: a. Item or task ID, the form number, and sequence number b. Total number of reads for each item c. Inter-Rater Reliability statistics including number and percentage of perfect agreement, number and percentage of adjacent agreement, and number and percentage of nonadjacent agreement d. Validity Statistics including the number of validity reads, number and percentage of perfect agreement, number and percentage of adjacent agreement, and number and percentage of non-adjacent agreement e. Distribution for scores and nonscorable codes from handscoring Florida Standards Assessments, ITN 2014-47 67 f. The mean, median, modal score, and score variance and standard deviations for scorable responses g. Final quality-assurance monitoring reports from the handscoring system that include rescoring and retraining rates h. Scoring notes handscoring. The handscoring statistical data will be provided in an electronic data file in a format agreed upon by the contractor and the Department. Provide copies of all materials used in training and recalibration. 8. It is expected that Department staff and Florida educators will be involved in the process to select student responses that will be used in training and qualifying scorers. 7.5.5 Reporting This section includes requirements for reporting results to students, schools, and districts. This includes the posting of the scored files (DSRs, SARs, and DARs) described in Section 7.5.4, the release of hard copy student reports and PDFs of other reports described in this section, and the creation of files produced for the Department’s website. Although the reports used in the past may be used as a model for these reports, the Department would consider changes to the report design to be more attractive or to present the data more clearly. The reply should include a description of how the respondent will design the reports, sample reports, and any suggestions for improvements to the current models. The report design is reviewed by the Department after each administration. If requested, the selected contractor will make modifications to the design after an administration and prior to the next administration. In addition, if high-level changes (change in Commissioner of Education, significant policy change) occur after designs have been approved; the contractor will make the appropriate modifications to the report design. If the respondent does not have qualified, professional staff devoted primarily to visual graphic arts and report design, a subcontractor(s) should be identified to provide these design services. The reply should include the names and documentation for the qualified staff or the subcontractors who will provide these design services. The final report design will take numerous iterations to ensure that all types of scores have been clearly presented. The Department will determine the final colors, design, and content of all reports. A general description of each type of report is provided below. Most of the reports are designed by test, that is, one (1) report form for English Language Arts/Literacy, one (1) for Mathematics, one (1) for each EOC subject test, and one (1) for Retake. Student Reports • For each student that tested by test. • Includes the student’s scores (e.g., scale scores, achievement levels, raw subscores, passing status), information about the scores (e.g., passing scores, points possible), and comparison data (e.g., state means). • Two (2) hard copies (one (1) for parent and one for school) shipped to districts, packaged by school. • Printed front and back with up to eight (8) colors plus black. • Some text translated to Spanish and Haitian Creole. The Spanish and Haitian Creole translations on the student reports must be created by someone familiar with the terminology used in Florida’s schools and proofed by an independent party, that is, someone not involved in the original translation. • Delivered with cover memoranda and packing lists following specifications similar to those for distributing test materials. • Hard copies produced for initial reporting and late reporting waves. Florida Standards Assessments, ITN 2014-47 68 For ELA/L and Mathematics test, the student report will include historical scores that are up to seven (7) test administrations with a bar graph. • At all grades/subjects containing performance tasks, each student will receive a report showing his/her response to one performance task in each grade/subject. The report will include a copy of the answer given, the number of points earned on the item, descriptive information about the question, and scores (points earned) earned on all of the performance items. School Report of Students • For each school by subject test. • Includes the scores for all students in the school that tested. • One (1) PDF by school posted on the online management system for school access for initial reporting only. • One (1) PDF of all schools in a district posted in the online management system for district access for initial reporting only. • One (1) color plus black. • PDFs produced for initial reporting and late reporting wave. • Reports created for the late reporting wave include only the students with reports in that wave. The reports will denote that they are part of late reporting and the reports will be delivered to districts on CD/DVD. District Report of Schools • For each district by test. • Includes aggregated data for the schools in the district (mean scores, percent passing, etc.). • One (1) PDF by test by district posted in the online management system for school and district access. • One (1) color plus black. • One (1) Excel spreadsheet for all districts and schools by test with the same data as the PDFs for the Department to use for its web postings. This may be a slightly modified version of the PDF without color. • PDFs produced for initial reporting only. State Report of Districts • For each test. • Includes aggregated data for the districts in the state. • One (1) PDF by test, posted in the online management system for school and district access. • One (1) color plus black. • One (1) Excel spreadsheet by test with the same data as the PDF for the Department to use for its web postings. This may be a slightly modified version of the PDF without color. • PDFs produced for initial reporting only. District Summary • For each subject. • Includes the aggregated data for each grade level tested in the district and the state. • One (1) PDF by subject by district posted in the online management system for school and district access. • One (1) color plus black. • PDFs produced for initial reporting only. State Summary • For each subject. • Includes the aggregated data for each grade level tested in the state. • Florida Standards Assessments, ITN 2014-47 69 • One (1) PDF by subject posted in the online management system for school and district access. • One (1) color plus black. • PDFs produced for initial reporting only. CD/DVD/Jump Drive with Student Report Shipments • The initial shipment of Student Reports includes a CD, DVD, or jump drive with the DSRs and DARs and their layouts. • The late reporting shipments include a CD, DVD, or jump drive with the DSRs and layout and the PDFs of the School Reports of Students for the late reporting students. The Scoring and Reporting Specifications provide specific details on handling special student score flags and school types and the specific information and scores provided on each report. The packaging and distribution requirements for test materials (see Section 7.4.1) also pertain to the packaging and distribution of the reports. Table 7.5.5 below shows these reports grouped as they are released to districts and schools. Table 7.5.5 Results Released to Districts and Schools Table 7.5.5 Results Released to Districts and Schools Results Release Number Released To Method of Release Contents of Release To Districts Online Management System District Student Results File (DSR .txt) School Report of Students (PDF of all students in district) 2 To Districts Online Management System State Summary (PDF) District Summary (PDF) State Report of Districts (PDF) District Report of Schools (PDF) State Aggregated Results File (SAR .txt) District Aggregated Results File (DAR .txt) 3 To Schools Online Management System State Summary (PDF) District Summary (PDF) State Report of Districts (PDF) District Report of Schools (PDF) School Reports of Students (PDF of own school only) 4 To Districts and Schools Shipments to Districts 5 To Districts and Schools Waves of Shipments to Districts Late Reporting Student Reports (hard copies) Late Reporting School Reports of Students (PDFs on CD/DVD) District Student Results Files (DSR .txt) 6 To Districts and Schools Shipment to Districts Writing Images (PDFs on CD/DVD) (spring only) Online Management System Demographic SARs, DARs, and DSRs 1 7 To Districts Florida Standards Assessments, ITN 2014-47 Initial Release Student Reports (hard copies) CD/DVD with DSR, SAR, DAR and Layouts 70 7.5.5.1 Initial Reporting Guidelines The Department coordinates the schedule for the release of data to the public. The contractor must be flexible with the reporting release schedule. The general guidelines are provided in this section. Detailed guidelines are specified in the Scoring and Reporting Specifications. The basic steps for the initial reporting of full-state data for Department review are listed below. The Department may request slight modifications to these steps. a. The contractor posts the full-state scan and edit files version A for the Department on the SFTP site. b. If necessary, the Department provides an issue log for the state scan and edit files. c. The contractor posts the full-state SSR version A for the Department and any independent checkers specified by the Department (see Section 7.5.6). d. Contractor provides data and recommendations to the Department based on the review for anomalous scores; the Department will review and identify any data to hold for investigation. e. If necessary, the Department will provide an issue log for the SSR files. f. The contractor posts the full-state SSR version B and the SAR version B (no version A) g. If necessary, the Department will provide an issue log. h. If necessary, the contractor corrects and reposts SSRs and SARs until approved. i. The contractor’s psychometric and scoring/reporting staff verify the accuracy of the final files. j. As requested for initial checking, the contractor creates DSRs and DARs and posts on the SFTP site for the Department review and updates as necessary until approved. k. During this process, the Department will identify a version of the files to be used to check the shipment of student reports and CD/DVDs. The contractor ships a sample of this shipment to the Department to check. l. If requested, the contractor posts additional samples of district and school PDFs of the reports on the SFTP site. m. After the PDFs are approved by the Department, the contractor will create all DSRs and DARs and reports on the SFTP site and, for state access only, in the online management system. n. After Department approval and guidance, the files and reports will be available for districts and schools in the online management system. o. The student reports and the CD/DVDs will be shipped for delivery by the critical date in the contract. 7.5.5.2 Late Reporting Guidelines For each administration, various reasons make late reporting necessary. For example, to-bescored answer documents were sent to the wrong location, updates were made based on the review of documents for graduating students (see Section 7.5.6.5), or the original student report was sent to the wrong school or student. Immediately after the initial wave of reporting, the contractor begins checking scores of graduating students as specified in Section 7.5.6.5. Any updates will be provided in the first wave of late reporting. The contractor will prepare for up to three waves of late reporting. The contractor and the Department check the late reporting files in a manner similar to the initial reporting wave. Florida Standards Assessments, ITN 2014-47 71 Depending on the administration and number of inquiries from districts, the Department will determine the late reporting schedule. The contractor must be prepared to adhere to a rigid late reporting schedule to resolve any issues with reporting for one administration before the next administration begins. The late reporting shipments include the following reports and files. • Student Reports – two (2) hard copies as in initial reporting with a note or footnote denoting it is a late report. • CD/DVD with DSR and layout (no DAR) • CD/DVD of School Report of Students (not in the online management system) with a note or footnote denoting it is a late report. After the initial release of data, the Department opens a website called the Score Inquiry System for district staff to inquire about missing reports or to ask for a student’s score to be reviewed. This system is designed so the contractor and the Department can communicate requests to search for a student’s answer document or report, or to review a student’s score. Maintenance of this system is the responsibility of the Department. The contractor must search for missing answer documents in the not-to-be scored materials if requested by the district to find any documents that were mislabeled or returned incorrectly. 7.5.5.3 Close-Out and Annual Reporting Guidelines After the last wave of late reporting, the contactor creates close-out SSRs for each administration. These SSRs contain the final records for all student with erroneous records removed. For example, if the district/school number was incorrect in the initial reporting and corrected in late reporting, the close-out SSR contains the corrected record. The incorrect record is removed. The close-out files are provided to the Department for review and approval, but they are not posted for districts. After the summer EOC close-out files have been approved, the contractor creates annual SSRs following guidelines provided by the Department for tests that require multiple administrations within a school year. These annual SSRs contain the records for students from each administration that school year. A few fields may be slightly altered from the close-out files for each administration, for example, if a school name changes during the school year, the name will be updated so the SAR shows annual aggregated data for that school using the correct name. 7.5.5.4 Demographic Reporting Guidelines After the spring close-out SSRs are approved by the Department, the Department inserts demographic information from its Survey 3 membership file of student data and aggregates these data for approximately 50 demographic groups. The Department then returns the Demographic SSR and SAR to the contractor to create Demographic DSRs and DARs. After the Demographic SARs are created and approved, the contractor also creates Excel spreadsheets with the data that can be used for the Department’s interactive demographic websites, which can be viewed here for reference: https://app1.fldoe.org/fcatdemographics/ and http://app1.fldoe.org/FEocDemographics/MainMenu.aspx. The Department will specify the format for the spreadsheets. After the annual SSRs and SARs are created for the subjects that require the multiple test administrations within a school year, this same process is completed for those files. Florida Standards Assessments, ITN 2014-47 72 7.5.6 Verifying Data Due to the tight timelines and quantity of data, all initial scoring and reporting processes are checked with mock data prior to the administration. Then, once actual data are available, a small sample of answer documents are used to begin checking, called early district return data. After all answer documents have been scanned, the files and reports are checked for completeness, accuracy, and possible irregularities (cheating) as described in this section. The verification process may require numerous iterations of the files to ensure that each student’s scores are accurately reported. The reply should include a description of the procedures to be used to verify the accuracy of data as outlined in this section. This includes the staffing necessary to complete the mock data process before the administration begins and the verification of each version of the files given the tight time constraints. The Department may have independent contractors assist in a review of the procedures used in this project and verification of the files and reports. Currently the Department has contracts with two independent parties for this purpose. One (1) contractor is primarily concerned with reviewing the procedures and the specifications for the scanning, scoring, and reporting with an emphasis on the psychometric aspects and primarily coordinating with the psychometric staff. The other contractor is primarily concerned with ensuring the procedures are followed, verifying codes and scores on the scan files, the SSRs/DSRs, the SAR/DARs, the Student Reports, and the PDFs of the other reports coordinating with the scoring and reporting staff. The contractor will be responsible for coordinating with the Department’s independent contractors for the review. This may include touring the scanning site and discussions with the contractor’s scanning, scoring, psychometric, and reporting staff. During the psychometric activities, the independent contractors may participate in the meetings and discussions, and during the scoring activities, the contractor may be asked to provide a portion of the reports and files to the independent contractors when reports and files are provided to the Department. 7.5.6.1 Checking Mock Data 1. All mock data files, reports, and procedures are generated and delivered using methods as close as possible to the same way final files will be generated and delivered. This process begins with mock PreID files, mock PreID labels, mock PreID Frequency Reports, mock history files, and mock answer documents, and then all headers, files, and reports for the initial wave of reporting are generated using the mock data. The mock student reports are packed and shipped as specified by the Department with the media containing the DSR and DAR, and the mock files and PDFs of reports are also posted to the SFTP site for the Department review. 2. Actual school information from the MSID file will be used in the mock data process to check the processing of the special school types and special districts. 3. Mock data cases may be generated using different methods. To check scanning of PBTs, some mock cases must be manually gridded, but others may be auto-gridded (preslugged) documents. Some CBT data must be generated by the contractor staff simulating the processes that a CBT student would use for each form of each test. To have enough cases to check all scoring and aggregating rules without using too many paper answer documents and time entering CBT responses, some additional cases may be computer generated, meaning for PBTs physical documents are never gridded or scanned for those cases and, for CBTs no one sat at the computer and key entered the responses in the same manner as a student enters the responses. 4. For PBTs, only the cases to check erasures must be manually gridded and other references in this section to cases with gridding may be auto-gridded. Florida Standards Assessments, ITN 2014-47 73 5. For each CBT, the PBT accommodated form must also be checked. Large Print and Braille paper documents are not used for mock processing; however the regular print answer documents and the headers are created with the appropriate Large Print and Braille codes. 6. The Scoring and Reporting Specifications indicate the various cases that must be included in the mock data. To easily check that all values scanned on PBTs and key entered on CBTs are properly recorded on the files, the mock data will include cases to check the values using specific patterns (e.g., alternating A,B,C,D,A,B,C,D responses). Cases must be included to be sure that each value on the SSR, SARs, and reports are accurate. This includes processing erasures, multigrids, demographic values, score flag codes, school type values, scores, etc. 7. The contractor must design the mock data files to check all procedures through the release of results. To reduce the amount of time to review these data and iterations required, the contractor will complete their review of all mock cases, possible values of codes and scores, and all reports before providing the data to the Department. The reply should include a description of how the respondent will complete these mock data, mock files, and mock reports prior to the first day of testing for each administration. 7.5.6.2 Checking Early District Return Data 1. To begin the verification of actual (live) data, the Department will identify some districts to be processed first, called Early Districts. The contractor will arrange for the pickup of answer documents from these districts so they will be scanned first for Department review. The CBT records for students in these districts will be extracted and provided to the Department for review. 2. The selected contractor must anticipate that the spring scan check will be completed at the contractor’s scanning site. The scan check for other administrations may be conducted at the contractor’s scanning site or at the Department. If the scan checking is performed at the Department, the contractor will post PDFs on the SFTP site of the headers and answer documents or ship the required documents to the Department to complete the checking. 3. For the on-site review of answer documents, the contractor will provide the Department the original scan and the edited scan files at least three (3) business days prior to the on-site review. 4. The contractor will pay travel expenses for up to five (5) Department staff from Tallahassee to the contractor’s scanning site to complete this checking. These expenses must be included in the reimbursable funds described in Section 7.7.9. 5. During the on-site review, the contractor will provide computers to Department staff with the following: • Access to the CBTs, access to email, access to a high-speed printer, and, if requested, SAS. • Easy access to the answer documents. Batches and stacks will be logically sequenced with the headers used. • LP, Braille, and OIPP documents and updated Special Document Tracking logs. • Key Entry verification logs (e.g., double entry of responses if document is unscannable) 6. The Department’s independent contractors may participate in this review or they may visit the site prior to this review. 7. Depending on the Department staff availability, the on-site review may be completed during one week or in two consecutive weeks. Florida Standards Assessments, ITN 2014-47 74 8. The reply may include additional and alternate methods to aid in meeting the data verification and reporting requirements in the time allowed. 7.5.6.3. Checking Data for Anomalous Scores Before data are released to the public, the contractor must review the student and school scores for anomalous data, that is, data that may not be a valid measure of student achievement. The contractor must review the data for testing irregularities, such as a student copying another student’s answers or an administrator changing answers. If the respondent does not have staff to perform these analyses, a subcontractor should be proposed. The Department will approve the subcontractor and final procedures to be followed for this checking. Anomalous scores would include the following. • Students in the same testing group have similar response patterns • A student has an excessive number of erasures from wrong answers to right answers • Students in a testing group have an excessive number of erasures from wrong to right answers • There was an unusual increase in school performance from a comparable administration The guidelines and procedures used to perform the anomalous score checking will be refined by the Department with the contractor’s assistance. If an irregularity is found in the data, the Department may request for the contractor to hold (i.e., not release) some data. This is handled using the Score Status Flags between the first and second versions of the full-state SSRs. Schedules must be set to allow time for this review and updating before the initial reporting for each administration. 7.5.6.4. Checking Full-State Data – Initial Reporting 1. The contractor will verify that all files and all PDFs of reports have been produced according to the Scoring and Reporting Specifications before providing them to the Department. In addition to the checks described in this section, the contractor’s psychometric staff must also approve the final SSRs and SARs before final reports are generated and data are released to the public. 2. The verification begins by checking that the DCFs have been completed properly so student reports will be sent to the correct school. If the school identified on the header is different from the school on the student’s answer document, and both are active schools, the contractor must identify the correct school for the student by contacting the district, if necessary. 3. The contractor will compare the edited scan files to the SSR and then the SSR to the SAR, and compare reports to the SSR and SAR to be sure all information (scores, codes, school numbers, etc.) is accurate and flowing through the systems correctly before submitting to the Department for review. The contractor will verify that all scores, flags, and codes were set correctly, and all fields contain valid values as defined by the layout. 4. The contractor will compare the final number of students tested in each school to the number of students on the PreID files to identify any anomalies, such as a missing box. To ensure all answer documents have been received for scoring prior to the delivery of the first scan file to the Department, the contractor will review the numbers of records and contact the Department and districts, using guidelines defined in the Scoring and Reporting Specifications, of any concerns. The contractor will assist districts in returning any materials that were not handled properly. Florida Standards Assessments, ITN 2014-47 75 5. As specified by the Department, the contractor will also provide sample shipments including the student reports and the CD/DVDs with files simulating the shipments to districts. The Department will determine which districts shipments will be included in this verification to ensure that a variety of reports are checked. They may be different from the districts selected for early scan if those records are not diverse. As described earlier in this section, the contractor may also be requested to send some reports to the Department’s independent contractor. 6. Before the files and reports are available in the online management system, the contractor and the Department will verify that the correct reports and files are available for district and school users prior to the release of the data. The contractor will provide documentation to show that the correct files and reports are being released as specified in the reporting schedule. See Section 7.5.5 for more information on the reports. 7.5.6.5 Checking Graduating Student Scores (Manual Rescore) Due to the high stakes of the tests for graduating students, the contractor must perform additional checks for graduating students who test on paper for Grade 10 ELA/L and for Algebra 1. The checks must be completed so any updates may be included in the first wave of late reporting. 1. For the tests required for graduation, the contractor must identify student in each grade who failed by 5 or fewer Scale Score points or 15 Developmental Scale Score points for Manual Rescoring. If there is an APS cut score established for the assessment, the APS cut score must be used for selecting the students. 2. For each of these students, the contractor will review each answer for the subject failed and verify that the items have been scored as the student intended. For example, the contractor will look for unintentional stray marks in the answer bubbles that do not appear to be an intentional grid, will confirm the hand written name on the document is the same as on the PreID label if one (1) is used, and will also confirm that no incomplete erasures in the griddedresponse columns were recorded as a mark (e.g., double gridding because of incomplete erasure). The item responses will be compared to the values on the SSR for any irregularities in editing or processing. 3. If any editing, scanning, or processing irregularities are detected, or if an incomplete erasure would change an item response, the student’s paper must be rescored with the updated response(s), and a new SS must be calculated to determine the passing status of the student. If the student passes, the record must be updated for the first wave of late reporting. If not, no change is made in the record. 4. A new student report will be generated and shipped to the district and the student will be included on the School Report of Students sent with the other late reporting wave 1 student data, if any. 5. The contractor will provide the Department with the number of answer documents reviewed by subject and the number that will have a Passing Status changed to Y when the new files are submitted. 6. This process will be completed within sixty (60) days of the original release of scores for the administration. 7.5.6.6 Checking Late Reports The contractor and the Department check the late files, reports, and CD/DVDs in a manner similar to the initial files and reports, although the files and reports produced are slightly different. The Department checks the files and reports after the contractor. See Section 7.5.5.2 for information on the procedures for the late reporting waves. Florida Standards Assessments, ITN 2014-47 76 7.5.6.7 Checking Close-Out and Annual EOC Files The contractor checks the close-out files by comparing each to the files previously generated for the administration to ensure that records reported to the incorrect schools have been removed and any scores or information for students have been updated or added as necessary. The contractor will ensure that there are not erroneous duplicate records caused by the multiple waves of reporting on the close-out files. The contractor must review the Late Reporting issue logs to ensure erroneous records have been removed from these files. The Department checks the files and reports after the contractor. The contractor checks the annual EOC files similar to the initial files, ensuring that the firsttime/retaker status codes are properly set. The Department checks the files and reports after the contractor. 7.5.6.8 Checking Demographic Files Each spring the contractor will check the Demographic SSRs created by the Department to verify that no records or scores were changed; only Survey 3 data will have been inserted. The contractor will verify that only valid values are in the Survey 3 fields before creating the Demographic SARs. The contractor will check the Demographic DSRs, SARs, and DARs prior to release in a manner similar to the initial release of the files. The Demographic Excel spreadsheets must also be checked by the contractor to be sure they contain the same information as on the SAR. The Department checks the files and reports after the contractor. See Section 7.5.5.4 for information on the procedures for the demographic files. 7.6.0 Scaling, Equating, Scoring, and Special Psychometric Studies During the timeframe of this ITN, the contractor will be responsible for drawing samples from the populations of students, for calibrating field-test items, for constructing the new test forms for the future administrations, and for Scaling, Equating, and Scoring (SES) of the assessment system. The system is complex and requires the contractor to provide knowledgeable and experienced staff, including psychometricians and programmers. Other than the annual regular psychometric operations, such as sampling, test construction, and SES for the assessment system, the contractor will conduct a set of special psychometric studies for these assessments, described in Section 7.6.3.6. The contractor will also cooperate to conduct another set of psychometric studies as Florida transitions from FCAT 2.0 and EOC Assessments to the Florida Standards Assessments (FSA) systems. As the Department transitions to the FSA, the contractor will assist the Department in the establishment of FCAT 2.0 and EOC equivalent score scales, appropriately calibrate items in the new assessments, properly combine performance indicators, construct meaningful score scales, possibly establish vertical scales for ELA/L and Mathematics assessments, assist in defining achievement level descriptors, build cut scores with criterion-referenced meaning on the new scales, link the performance of Florida’s students to the scales of other assessments to make comparisons, and conduct other special analyses that may be required by the Department or required for a peer review under the federal Elementary Secondary Education Act (ESEA) program. This section of the ITN provides descriptions of related services and products and the expected quality assurance steps that must be employed to ensure that all tested students receive assessment results that are valid, reliable, and accurate. Florida uses a Technical Advisory Committee (TAC) composed of selected Florida District Coordinators of Assessment and selected experts from around the nation representing the areas of measurement and statistics, students with disabilities, English language learners, and related alternate assessments. The TAC members listen to issues and offer guidance on implementation of Florida’s assessment program. Currently Florida’s TAC meets twice a year – in March and in November. The contractor is responsible for convening and conducting these meetings. Florida Standards Assessments, ITN 2014-47 77 The contractor may utilize proprietary software for any of the annual regular psychometric activities or the special psychometric studies for the assessment system during the transition to the FSA, but is required to provide this software for use by all relevant staff, including the Department and any subcontractors. Whatever software is selected must be able to handle the large number of test cases, including large calibration samples, and also accommodate full-state calibration with numbers of records in excess of 200,000 at a time. The contractor will be responsible for assembling a psychometric team to support work performed under this contract consisting of the contractor’s psychometricians and programmers, and at least one (1) outside entity subcontracted to participate in all SES activities to independently perform these particular operations and verify all results. The Department’s psychometric team consists of two psychometricians plus three graduate interns pursuing their doctorate in educational measurement to be part of the team replicating all SES activities and overseeing the entire process. A separate procurement is being issued for the FCAT 2.0 and Florida EOC Assessments for subjects not covered as part of this ITN (Science, Social Studies, etc.). The contractor selected as part of this ITN process should anticipate coordinating with the contractor from the FCAT 2.0/Florida EOC Assessments procurement for some of the tasks described in this scope of work. 7.6.1 Assessments 7.6.1.1 Florida Standards Summative Assessments (Scaling, Equating, and Scoring) 1. The contractor will develop and maintain test construction specifications describing the details of test construction, including content and psychometric guidelines for building equivalent forms. Item selection will be based on matching target test characteristics, test information functions, and standard error curves in addition to meeting other content and psychometric requirements, plus further constraints. 2. For the scaling of items, the Department prefers to use a three-parameter logistic Item Response Theory (IRT) model for multiple choice items, two-parameter logistic IRT model for gridded-response items, and two-parameter generalized partial credit model for constructedresponse items. For the equating of the tests to their base year scales, the Department prefers to employ test characteristic curve equating using the Stocking & Lord procedure and triangulating the equating with IRT based other equating procedures, such as Haebera, Mean/Mean, and Mean/Sigma. For scaling student performance, the Department prefers to use Maximum Likelihood Estimation to score examinee response patterns. For dimensionality, the Department supports using a variety of item types measuring various student skills on the Florida Standards, but maintains validity for a single dimension. However, these are just the Department’s preferences for scaling, equating, scoring, and dimensionality. The contractor must conduct thorough research studies and collect empirical evidence to determine whether these options are psychometrically defensible, whether items measuring various constructs can be combined into a single assessment instrument and scored, and whether the preferred options can be implemented operationally to obtain assessment results that are valid, reliable, and accurate. If research and empirical evidence(s) suggest the use of a different set of techniques in scaling, equating, and scoring of the assessments, the respondent should provide a detailed description of the measurement models proposed for use in calibration, equating method for creating interchangeable scale scores, and scoring algorithm for placing examinees onto a score scale. The contractor will be responsible for designing sampling plans, drawing the samples from populations of examinees, and supplying the data files to the scaling and equating team described in Section 7.6.2.1 so that the required analyses can be performed in a timely manner. The contractor will be responsible for calibrating the tests, placing item parameter estimates onto the base year’s scale, and providing the final scaled item parameters for the purpose of scoring. After the Department approves the final scaled item parameters, the contractor will score student responses, the Florida Standards Assessments, ITN 2014-47 78 entity subcontracted to participate in all SES activities will independently verify the scale scores, and the Department’s psychometric team will replicate and verify the scale scores. 3. For the Assessments in Algebra 1, Algebra 2, and Geometry, the contractor must construct multiple forms of these assessments by following content and psychometric requirements as indicated in the test construction specifications. As for the scaling, equating, and scoring process for these assessments, the Department desires to follow the same process as indicated in the previous paragraph, but since there are multiple forms of the assessments at this time, the Department prefers to concurrently calibrate these forms so that all item parameter estimates can first be placed onto a common scale, and then, equating these item parameter estimates to the base year scale via anchor items. In other words, equivalency of the multiple forms of new tests is verified through a concurrent calibration, plus post-equating via common items between the item bank and the new test forms. However, as indicated earlier, the contractor must show evidence that the Department’s preferences are psychometrically defensible and operationally feasible. The respondent may include in the reply a different proposal for scaling, equating, and scoring of these assessments to obtain assessment results that are valid, reliable, and accurate. Also, when multiple forms of these assessments are being created, the contractor must construct the new test forms that are sufficiently different in an effort to discourage cheating and to ensure a valid measure of student learning. In addition, the department will consider the use of pre-equated forms for administrations other than the spring administration. 4. Because every effort is made to develop items and construct tests according to universal design principles, we anticipate all Braille versions of the assessments will contain the same items as the regular version of the tests. If they do not contain the same items, the contractor will develop a separate scoring process to be approved by the Department and used by the contractor and subcontractor. 7.6.1.2 Retake Assessments (Scaling, Equating, and Scoring) The contractor’s content and psychometric teams will work together with the Department’s content and psychometric teams to select retake test forms, which may consist of a previously-used intact form, or another proposed design that yields a test of similar difficulty to the operational form. The inclusion of performance tasks in the retake form may be proposed, however, consideration will also be given to forms that do not include performance tasks. The contractor will administer the selected test form and will score student responses. The entity subcontracted to participate in all SES activities must independently verify the scale scores generated by the contractor. The same procedure will be repeated to select different forms for subsequent years. 7.6.2 Scaling, Equating, and Scoring (Psychometrics) 1. The contractor will construct Scaling, Equating, and Scoring (SES) specifications for the assessments. The specifications will describe the specific steps for implementing the scaling, equating, and scoring requirements and will explain how, by when, and by whom all of the necessary steps will be completed and verified. The specifications will also address the steps, procedures, and software required for replication of the scaling, equating, and scoring procedures by the Department. The specifications will be reviewed by the Department and TAC members, if necessary, and updated by the contractor for each subsequent administration. The Department will provide the contractor with copies of existing specification documents. 2. Before operational SES activities start, the contractor and other members of scaling and equating team must conduct dry-run SES activities on mock data files to make sure all operations laid out in the SES specifications can be successfully executed during the operational SES process. For this purpose, the contractor will prepare “mock” data for the assessments using the State Student Results file (SSR), as Florida Standards Assessments, ITN 2014-47 79 indicated in Section 7.5.6.1. The purpose of this mock data will be to provide a vehicle for the SES dry run; i.e., a practice check of the operation of computer programs used in scaling, equating, creating adjusted item parameters, and scoring. The psychometric staff for the identified calibration team will be available for daily discussions and consultation throughout the parallel dry-run calibration activities. 3. It is the responsibility of the contractor to keep all the documents/electronic files from the scaling, equating, and scoring activities, such as the classical and IRT item statistics, alternative equating solutions summarizing the reasons why anchor items are removed or kept in the anchor set, the impact of decisions on the transformation coefficients and scale scores, and the other pertinent decisions made by the Department. The format of these documents must be proposed by the contractor and mutually agreed upon by the contractor and the Department. 4. The SES specifications must include how the contractor’s subcontractor will independently verify the scaling, equating, and scoring operations. For the entire SES process, the contractor will provide all of the resources, including software, staff support, and data files to permit parallel calibration, equating, and scoring of data during the same time period that contractor are producing the scaled item parameter estimates. 7.6.2.1 Scaling and Equating Team The contractor will assist the Department in coordinating the work of the scaling and equating team during dry-run activities and regular administrations each spring. The contractor is required to subcontract with at least one outside entity to work on the scaling and equating team during all SES activities. The contractor’s, the subcontractor’s, and the Department’s psychometric staff will each independently perform SES tasks to verify results. The specifications will clearly describe the scaling and equating team consisting of the Department’s, the contractor’s, and third party’s psychometricians and programmers, as well as the roles assigned to each entity. The Department’s psychometric team will oversee the entire SES process. Respondents should indicate in their replies the qualified personnel/subcontractors who will provide these professional services and include their vitae or other documentation of expertise and experience. 7.6.2.2 Selection of Calibration Samples Anchor, core, and field-test items are calibrated on representative samples. Thus, the contractor is responsible for designing sampling plans and drawing the representative samples from the respective populations, which will include at least 65% of the respective population, including approximately 90% of Miami-Dade. The contractor will also plan in a way so that a representative set of approximately 5,000 students take anchor and field-tests forms. The contractor’s sampling specifications must include analyzing the most recent SSR files and establishing a road map for gathering the required data proven to represent the demographic characteristics of the state and student performance. The specifications will also address the steps, procedures, and software required for replication of the sampling procedures by the Department. The specifications will be reviewed by the Department and TAC members, if necessary, and updated by the contractor for each subsequent administration. Before SES activities start, the contractor must also check and verify the representativeness of the most recent data, and then provide the files to scaling and equating team members so that necessary analyses can be performed in a timely fashion. The contractor must repeat this process for each assessment administered each spring. THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK Florida Standards Assessments, ITN 2014-47 80 7.6.3 Technical Oversight and Special Studies 7.6.3.1 Contractor’s Role The contractor must provide psychometric direction and oversight for all aspects of the assessment programs described in this ITN. The psychometricians assigned to this project must have extensive experience in state assessment programs. The psychometricians will plan, conduct, analyze, and report on test construction, scaling, equating, scoring, and the special studies described in this section. 7.6.3.2 Technical Report Each year, the contractor will produce a technical report based on an overall analysis of that year’s administrations. The technical report will also include a comparison of the characteristics of the current test administration to previous administrations. The technical report will be reviewed by the Department and by the Technical Advisory Committee (TAC) prior to completion of the final copy of the report. The technical report will include a section on field-test administrations. The report will include tabular and graphic displays of data to illustrate the characteristics and quality of test scores. At a minimum, the technical report will include the topics listed below in the draft table of contents. A. DRAFT TECHNICAL REPORT TABLE OF CONTENTS B. BACKGROUND OF ASSESSMENTS C. ITEM DEVELOPMENT & TEST CONSTRUCTION • Item Development • Item Review • Field Testing • Item Statistics o Classical Statistics o Item Response Theory Statistics: Difficulty, Discrimination, and Guessing o Differential Item Functioning (DIF) Statistics o Model Fit o Local Independence o Dimensionality (factor analyses, etc.) • Test Construction D. ADMINISTRATION • Eligibility • Administration Procedures • Accommodations • Test Security • Data Forensics Program E. REPORTS • Appropriate Uses for Scores and Reports • Reports Provided • Description of Scores • Appropriate Score Uses • Cautions for Score Use F. PERFORMANCE STANDARDS • Setting Performance Standards • Interim Performance Standards Florida Standards Assessments, ITN 2014-47 81 G. SAMPLING • Rationale • Procedure • Representativeness H. SCALING • Rationale • Measurement Models • Scale Scores • Scale Drift • Vertical Scaling • Checking IRT Assumptions (Model Fit, Local Dependence, Dimensionality, etc.) I. EQUATING • Rationale • Pre-Equating • Post-Equating • Development Procedure for Future Forms • Sources of Error Variances to Score Instability • Linking Assessments During Transitioning Years • Linking Assessments to Other Scales for Performance Comparision J. RELIABILITY • Estimating Reliability • Standard Error of Measurement • Student Classification Accuracy and Consistency • Reliability for sub groups in the population • Reliability of year-to-year changes in school means K. VALIDITY • Perspectives on Test Validity • Criterion Validity • Content and Curricular Validity • Construct Validity • Consequential Validity L. CONSTRUCTED-RESPONSE SCORING • Scoring Process • Quality Control • Handscoring Reliability and Validity • Rater Effects M. QUALITY CONTROL PROCEDURES • Quality Control for Test Construction • Quality Control for Non-Scannable Documents • Quality Control in Data Preparation • Quality Control in Production Control • Quality Control in Scanning • Quality Control in Editing and Data Input • Quality Control for Test Form Equating • Quality Control in Scoring and Reporting Florida Standards Assessments, ITN 2014-47 82 N. SUMMARY STATISTICS • Item level statistics • Test level statistics • Review and comparision of raw scores, sub-scores, scale scores, distributions for total and sub-populations across years to monitor the trends across years and sub-populations • Review and comparision of percent proficiency rates across years on different populations • Frequency distributions of student achievement (all students and by sub-group) across years by grades and subject • Scale drift indicators • Correlations of student performance across the various subject areas tested Following each field test or operational administration, differential item functioning (DIF) analyses, to detect possible item bias, will be conducted, preferably using Mantzel-Haenszel method. DIF analyses are conducted for Caucasian, African-American, and Hispanic racial/ethnic groups and by gender. Values for items resulting from these analyses will be included in the item-banking system. Changes in DIF values across administrations will be analyzed and presented in the Technical Report. The contractor will provide a detailed item analysis for each test administration and provide the results in the annual Technical Report. There are several item level statistics, such as pointbiserial, corrected point-biserial, fit, etc. available in the current item bank. Thus, the contractor is responsible for calculating these statistics following each administration and populating the findings into their respective positions in the bank. The Department will work with the contractor to determine the contents of the written report. The contractor will provide several printed copies of the report and also an electronic version in the format determined by the Department. 7.6.3.3 Transitioning from FCAT 2.0 and EOC Assessments to Florida Standards Assessments (FSA) – Linking the Scales As the assessment system transitions to the FSA in spring 2015, the scales of the new FSA will not yet have any meaning attached. Therefore, the Department requires technical assistance from the contractor so that the scales of these assessment systems can be linked through a special and defensible psychometric operation. The contractor will provide a detailed scale linking plan for transitioning from FCAT 2.0 Reading & Mathematics and EOC Algebra 1 & Geometry to FSA. Linking through a special psychometric operation may go beyond employing a simple linking procedure, since each score unit on the two separate scales may have to be linked to each other so that achievement levels can be determined for student and school accountability purposes. The Department may directly engage additional contractors, or the contractor’s subcontractor (if proposed) to triangulate the results. The plan will include alternative options for linking, how the alternatives will be operationalized, scoring plans, and procedures. The plan should include a comprehensive and detailed timeline for all tasks and events. The plan will be reviewed and vetted by Florida TAC members after submission, and may require revision as a result of this review. This review will address the appropriateness and validity of the psychometric procedure selected for linking, as well as the validity, reliability and defensibility of the linked scores. The respondent’s reply should describe available psychometric methods for establishing interim scores for high-stakes student-level decisions, including an explanation of the defensibility of the proposed methods. During the transition years, as student performance on other assessments will be used to determine the achievement of testing standards, the linking/concordant score studies will be conducted by the contractor and concordant scores will be recommended to the Florida Standards Assessments, ITN 2014-47 83 Department. The contractor selected through this ITN process will be expected to secure external legal opinion regarding the defensibility of the linking method to be employed. 7.6.3.4 Implement Achievement Level Standard Setting Meetings The contractor is responsible for facilitating the Department’s process to establish achievement levels and associated cut scores for the Florida Standards ELA/L, Mathematics, Algebra 1, Algebra 2, and Geometry Assessments; achievement levels, passing scores, retention cut for grade 3 ELA/L, graduation standards for ELA/L, and achievement levels and passing scores for the Algebra 1, Algebra 2, and Geometry Assessments in consultation with Florida educators and citizens. The contractor will be responsible for developing a Standard Setting Plan for each standard-setting process utilizing IRT item values and appropriate, reliable, valid, and defensible procedures. The Standard Setting Plan will address the nature of the proficiency level standards, methods for determining the standards, and procedures for validating and analyzing the quality of information reported using the achievement levels. In addition to describing the general standard-setting strategy, the Standard Setting Plan will describe in detail how participants will be identified and how the standard-setting procedure will be implemented, such as the materials to be prepared by the contractor, how test forms will be organized, and how the contractor will analyze the data and present impact results. The contractor will be responsible for organizing and implementing the standard-setting processes, based on the Standard Setting Plans, and for assisting the Department in conducting the standardsetting meetings. Each standard-setting process will involve meetings of standard-setting committees, which will involve Florida teachers and instructional leaders, followed by a reactor meeting involving other Florida education stakeholders. The educator meetings in summer 2015 will result in content-based achievement level descriptions for each grade and recommended cut scores for each test. The Department desires to present proposed cut scores to the State Board of Education in January following each meeting; therefore the final report of activities of each meeting should be available for the Department no later than November following each meeting. The final report of standard setting activities should include in detail process and activities followed during the standard setting, including a description of materials used, statistical background, participants, changing of recommendations through rounds, and participant evaluation of meeting processes. 7.6.3.5 Vertical Scaling The contractor will be responsible for conducting the vertical scaling for the Florida Standards ELA/L and Mathematics Assessments. This scale provides for reporting growth continuously from grades 3 to 10 in ELA/L and from grades 3 to 8 in Mathematics. In other words, each student’s “growth” between adjacent grade levels will be calculated and reported on this scale as a type of “value-added” score. The contractor’s plan for vertical scaling, provided prior to the selection of items and construction of forms for the appropriate administration, should recommend and describe the process of determining hierarchical common content areas across grades, a recommended method for vertical scaling that takes into account potential uses of the scale for NCLB and Florida’s accountability system, the data collection procedures, and how and by whom and by when all of the necessary steps will be completed. The vertical scaling plan must be approved by the Department and presented to the Technical Advisory Committee. After vertical scaling studies are conducted, the contractor will prepare a technical report including a description of the process, statistical background, methodology, the recommended scale, and characteristics of the scale. The final report will be presented to the Technical Advisory Committee. The contractor should anticipate replicating the vertical scaling study three years after the completion of the initial scaling study. This work may occur in the optional renewal period and is contingent on contract renewal. Florida Standards Assessments, ITN 2014-47 84 7.6.3.6 Scaling and Equating Studies As explained in Sections 7.6.1.1 and 7.6.1.2, the Florida Standards Summative Assessments will be pre-equated during test construction and post-equated after each spring test administration. The retake assessments, such as Grade 10 ELA/L or Algebra 1 Retake will be pre-equated only, consisting of a previously-used intact form, or another proposed design that yields a test of similar difficulty to the operational form. In other words, during the time frame of this ITN, there will be several equating activities to put the new test forms on the same scale through a series of intermediate equating procedures (known as equating chains). In this case, equating error from each of these intermediate equating procedures may accumulate to a point where the comparability of scale scores across time is questioned. For this reason, it is essential for the Department to employ practices that ensure comparable scores across time in the presence of equating chains. The contractor is required to submit a research plan to be conducted in spring 2018 to investigate the extent of scale drift on the assessments. The research plan(s) must be approved by the Technical Advisory Committee and the Department, and the resulting report(s) must be submitted to the Department and presented to the Technical Advisory Committee. 7.6.3.6.1 Investigate the Comparability between Devices Delivering Assessments EOC Assessments will be administered online and delivered by several different devices throughout the state since school districts are using many different machines when administering the assessments, such as laptops and desktops. The Department prefers to make sure these devices used in delivering the assessments are not providing any advantage or disadvantage to the performance of students. As a result, there is a need to investigate whether there is an impact of certain devices on the assessment results. The contractor must determine the groups of testing devices, based on the types of devices that work with the contractor’s platform, whose impact will be investigated, create a plan for the study, present the plan to the Department staff and TAC members, and upon approval from the Department, start investigating any impact on item-level, test form-level, and reported score-level. The results of the study and contractor’s recommendations must also be presented to both the Department staff and TAC members. 7.6.3.6.2 Linking Florida Standards Assessments to Other Assessments Measuring the Same Standards The Department intends to compare Florida student performance on FSA to other students’ performance on the same standards from other states, also representing the nation. Thus, contractor must link the scales of Florida Standards Assessments to the scales from other assessments measuring the same standards. By August of 2015, the contractor will complete a comprehensive curriculum mapping study to determine the degree of overlap between Florida’s standards and those in other states in order to determine the degree of alignment. The mapping study results will be analyzed by the Department and contractor’s staff to determine if Florida’s standards are sufficiently like those of other states to ensure a valid assessment for purposes of reporting comparative information or linked data on Florida Standards Assessments. Assuming there is adequate alignment, the contractor will work with the Department to design, plan, and conduct a linking study. The contractor’s plan for linking should recommend and describe the process of determining the linking methodologies, the data collection procedures, and how and by whom and by when all of the necessary steps will be completed. The linking plan must be approved by the Department and presented to the Technical Advisory Committee. After the linking studies are conducted, the contractor will prepare a technical report including a description of the Florida Standards Assessments, ITN 2014-47 85 process, statistical background and methodology, and characteristics of the scale. The final report will be presented to the Technical Advisory Committee. The contractor will also generate data files with linked scores and prepare reports. If a linking study requires collecting data from either a common set of students taking both assessments or from different sets of students taking common set of items, the contractor selected through this ITN will be responsible for scoring the test forms administered to students for linking, including scanning/imaging, processing, handscoring, calibration (generation of item statistics), scoring, and production of student results files including linked scaled scores. The scoring will be conducted after the completion of all scoring and reporting for the operational tests given in Florida during spring of 2016. It is anticipated that the results files from the linking study will be provided to the Department by July 2016. As an outcome of the linking study, the contractor will provide scoring tables displaying how Florida’s scale scores are aligned to the ones of other assessments measuring the same standards. Upon first use of these linking tables in reporting the results of the spring 2017 Florida Standards Assessments, the contractor will provide psychometric services to verify the reporting as well as the reliability and validity of the linked scores for spring 2017. The contractor should anticipate replicating the linking study three years after the completion of the initial linking study. This work may occur in the optional renewal period and is contingent on contract renewal. 7.6.3.6.3 Linking Florida Standards Assessments to International Assessments The Department intends to compare Florida student performance on Florida Standards Assessments to other students’ performance on International Assessments, such as Program for International Student Assessment (PISA) and Trends in International Mathematics and Science Study (TIMSS). Thus, contractor must link the scales of Florida Standards Assessments to the scales from international assessments. By August 2015, the contractor will develop a study plan for the linking Florida Standards Assessments to International Assessments. The contractor’s plan will include, review of the most promising and most feasible methods likely to be used in international linking, discuss the technical feasibility of each method given the nature and design of the Florida Standards Assessments and those of international assessments, such as PISA and TIMSS, propose method(s) for linking the scales of assessments, propose alternative ways for collecting data, explain the process of establishing the linking relationship(s) in detail, and provide a comprehensive and detailed timeline for all tasks and events. The contractor will also share the report with the Department and its TAC members. Upon approval from the Department, the contractor will finalize the plan for the linking and implement all of the activities. The contractor selected through this ITN will be responsible for scoring the test forms administered to students for linking, including scanning/imaging, processing, handscoring, calibration (generation of item statistics), scoring, and production of student results files including linked scaled scores. The scoring will be conducted after the completion of all scoring and reporting for the operational tests given in Florida during spring of 2016. It is anticipated that the results files from the linking study will be provided to the Department by August 2016. As an outcome of the linking study, the contractor will provide scoring tables displaying how Florida’s scale scores are aligned to the ones of international assessments. Upon first use of these linking tables in reporting the results of the spring 2017 Florida Florida Standards Assessments, ITN 2014-47 86 Standards Assessments, the contractor will provide psychometric services to verify the reporting as well as the reliability and validity of the linked scores for spring 2017. 7.7.0 Program Management Respondents should provide their overall description of the approach to manage, implement, and support Florida’s assessment program that specifically addresses the entire scope of work in this ITN. Respondents should identify and describe all resources available to support all program activities. A management plan for the contract should be incorporated into the respondent’s reply. In the event that multiple contracts are awarded, additional program management capacity will be required to coordinate among the other contractors and the Department. Respondents should designate a Senior Program Manager who will have overall, daily responsibility for all processes and deliverables under the full scope of this program. The Chief of the Department’s Bureau of K-12 Assessment will be assigned as the program manager of the contract resulting from this ITN under the overall supervision of the Director of Assessment, Division of Accountability, Research and Measurement. 7.7.1 Provisions Governing the Work of the Contractor The procedural requirements identified in this section govern the work of the contractor. This section establishes specific requirements and general principles by which the program will be managed. The work described in Sections 7.2.0 through 7.6.0 reflects design, development, validation, and implementation requirements of Florida’s standards-based testing program. In preparing replies, respondents should consider the following provisions governing the activities required by these sections. a. All of the processes used by the contractor to complete the work identified in Sections 7.2.0 through 7.7.0 must be approved by the Department. All products must be approved by the Department at milestone stages of development as described in the Production Schedule developed at the beginning of the contract. The Department reserves the right to specify details of tasks and products. b. Design control of all aspects of this program remains the responsibility of the Department. Changes in the design made after the award of the contract will be at the discretion of the Department. c. The Department reserves the right to require changes in materials, produced in any medium, at any stage of the program prior to final printing, production, or deployment. d. Products and processes are modified to some extent for each assessment. For example, modifications may be made to the design, format, or wording of printed and online products from one administration to the next. All such modifications are to be expected as part of the scope of this program and will be at contractor expense. Modifications may also be required for other products and procedures, such as data analyses, file formats, and report designs. Consequently, all printed products developed under this contract are to be only produced in quantities for the particular assessment, unless otherwise specified in writing by the Department. 7.7.2 Contractor Staff and Responsibilities The work described in the ITN constitutes a large, complex, challenging program that requires year-round activity and close attention to overlapping tasks for different years and the impact on project resources. The contractor is expected to provide a sufficient number of highly-qualified personnel to work closely with the Department to manage the contract. Respondents should carefully consider whether they have sufficient and knowledgeable human resources to complete the work activities of the ITN and should demonstrate in their replies that they have sufficient and knowledgeable human resources to complete program work. Companies should consider bidding only if they are committed to assigning the number and quality of staff and staff time required to complete a program of this magnitude. The contractor must assign professional staff members who are qualified, experienced, and capable of providing technical assistance to the Department. Please note the following staff requirements: Florida Standards Assessments, ITN 2014-47 87 a. The contractor will assign at least one (1) full-time senior program manager, four (4) full-time project managers, and two (2) full-time administrative assistants. The full-time senior program manager will concentrate on ensuring quality and timeliness of the full range of services and products required by this ITN. The contractor’s management team is expected to lead project and support staff and to manage implementation activities throughout the contract period. Although this section outlines the minimum staffing requirements for this program, the contractor must provide adequate staff to maintain the highest quality products and must deliver these products according to the Department’s timelines. b. It is anticipated that the four (4) full-time project managers will focus on (a) test development, (b) test administration, (c) scoring and reporting, and (d) computer-based testing. Respondents may propose a modification to this project assignment for the managers. c. Implementing the program requires considerable administrative activities involving the maintenance of schedules, the development of specifications, production of management reports and documents, arrangement of meeting and travel for educators and Department staff, and communication with district and other personnel. Administrative assistants are necessary to serve in a dedicated role as meeting organizers and as direct contacts for the many Florida citizens that may attend assessment meetings throughout the year. The Department requires a single point of contact to discuss each meeting's requirements and venue options and to communicate with participants throughout the travel process, and to address issues that may require late-notice changes in travel plans. d. The managers should be members of the regular organizational staff with experience within the company or institution in coordinating the types of activities identified by this ITN. In their replies, respondents are expected to demonstrate that the individuals identified as managers have sufficient authority across departments within the organization to ensure that the work of the contract has the necessary priority to be completed with the highest quality and on time. e. Respondents should include pertinent staffing charts in their replies that show the regular institutional organization staffing and the proposed organization of program staff for this work. In these charts, the full-time equivalency devoted to work under the contract for each staff member with significant responsibilities will be identified. Persons proposed to fill all key positions will be identified by name. f. Respondents should indicate, by name, the professional personnel to be responsible for major contract activities, with an estimation of the amount of time, as a percent of full-time, each person will devote to each task. This information will be included in a separate chart in the management reply. An appendix to the technical reply should contain one (1) page vitae for each of the professional personnel, including subcontractors, to be assigned to the program, indicating relevant educational background and professional experience. g. In a separate chart(s), respondents should identify the full extent of staff resources to be allocated to the program to conduct activities related to the item and test development; computer-based test system and content support; ancillary materials development; scanning, imaging, data processing, and reporting activities; and scoring, equating, and technical analysis/special studies activities for each year of the program. The chart should identify the major activities by month and the number of staff by category and the amount of time in full-time equivalencies (equivalent person-days) assigned to complete the activities. h. In addition, the contractor must provide for five (5) contractor staff positions funded through the reimbursable category and described in Section 7.7.9.6. Florida Standards Assessments, ITN 2014-47 88 7.7.3 Subcontractors In addition to any subcontractors required in this ITN, the prime contractor may choose to employ one (1) or more additional subcontractors for the completion of these tasks. If the respondent proposes to employ a subcontractor(s), the qualifications and experience of the subcontractor(s) should be documented in the reply at the same level of detail as those of the respondent. The prime contractor will assume responsibility for all services offered in the reply whether or not they are performed or produced by the contractor or by subcontractors. The Department will consider the prime contractor to be the sole point of contact for contractual matters, including payment of any and all charges resulting from the contract. The respondent should describe how the work of all subcontractors will be monitored and guided to ensure adherence to requirements in this ITN, including resulting specifications and schedules. The contractor will provide for quality assurance activities in oversight of the subcontractor’s performance. A separate chart in the reply should identify all of the subcontractors proposed to be involved in the program, the assessment components they will be involved with, and the specific services they will provide. Throughout the life of the contract, any proposed changes in subcontractors must be approved by the Department. The contractor will provide the Department with a qualifications and experience summary for review and consideration for acceptance in advance of a proposed subcontractor’s start of work on program processes or deliverables. As part of the Annual Report, all subcontractors, their services, and Departmental approval status will be listed. A detailed description of the subcontracted services to be provided under the ensuing contract should be included in the reply. All subcontracted services should be documented. Each service (e.g., printing, computer operations, test development, psychometric services, legal counsel, quality control, reporting) should be fully described, including resources, level of service, scheduling, and quality commitments. The respondent should identify the proportion of materials, etc., to be generated by any and all involved subcontractors. No contractor- or subcontractor-provided work related to customer service or to any secure document or data will be provided by a persons or agencies physically located outside of the United States. 7.7.4 Program Management Communication Weekly program management conference calls will be conducted at an agreed upon day and time in order to ensure that all requirements and timelines are maintained and to address needs for problem solving and timesensitive adjustments. The Department may determine periodically that weekly, focused conference calls are required as activities dictate. The contractor will provide a toll-free conference line for all calls between contractor and Department staff required for the effective coordination of the assessment program. To meet timelines for completing tasks, both the Department and the contractor will need to use secure electronic communications, including, but not limited to, a secure FTP site, a website, a teleconferencing site, etc., provided by the contractor in addition to overnight express delivery of materials as necessary. In addition, the Department and the contractor will need access to a secure site that can be used with districts and with subcontractors to transfer secure materials. The contractor and the Department may determine that, for selected meetings, video teleconference meetings will be more time and cost efficient than travel to a common meeting site. In this case, the contractor will coordinate all logistics and provisions for the teleconference. To assist Department staff with communication during on-site visits to the contractor’s offices or other work sites, the contractor will provide access to lockable office space, including a telephone with a speaker, a fax machine, and a computer with Internet access and a local printer. Florida Standards Assessments, ITN 2014-47 89 7.7.5 Program Management Meetings The successful operation of the program will require regular management meetings between the contractor and Department staff. At the Department’s request, the management meetings will take place in Tallahassee, the contractor’s headquarters or other worksite, or in a location convenient and agreed upon by the contractor and the Department. Contractors should anticipate that the first comprehensive program management meeting will take place at the contractor’s headquarters within thirty (30) days of contract execution. The contractor must expect that up to fifteen (15) of their staff will travel to attend approximately two (2) comprehensive program management meetings per year. The contractor will also send all relevant staff to up to four (4) additional annual management meetings that are project specific, such as the scoring and reporting debrief. All relevant subcontractors must also attend these meetings. The contractor will be responsible for the logistics, facilities, and travel costs of their staff and required subcontractors’ staff for all management meetings. These meetings will also involve relevant Department staff. Provisions will be made for electronic participation of any staff member unable to travel to these meetings. In the event that multiple contracts are awarded, the contractor should expect part of the meetings will include all other contractors and Department staff, and that part of the meetings will be reserved for one-on-one meetings between the Department and each contractor to focus on topics and issues specific to each contractor. The cost reply should include funding specifically for Department staff travel for program management meetings and other oversight activities. Program oversight activities include work related to overall program management, management of key processes, including development, test administration, scanning, handscoring, scoring, programming, printing, and reporting. The contractor will be responsible for paying the cost of travel, lodging, and per diem for up to ten (10) Department staff members to attend up to fifteen (15) trips annually. The cost of the trips has been based on travel between Tallahassee and the contractor’s headquarters or between Tallahassee and the contractor’s or subcontractor’s headquarters, scanning/processing sites, or scoring/handscoring sites. Funds set aside for this purpose are described in Section 7.7.9.5. 7.7.6 Program Management Reports Five types of reports are to be prepared by the contractor to facilitate and document program management communications with the Department: (1) the Program Schedule, (2) the Weekly Management and Weekly Action Item Reports, (3) a Technical Report, (4) Annual Program Review, including the Operational Addendum, and (5) the Accounting Report. In addition to written reports, the Department may require the contractor’s participation in presentations to Department senior management or other entities at places and times determined by the Department. 1. The Program Schedule will be developed for each year of the program and will include both high-level management summary information as well as specific dates associated with critical processes, milestones, and deliverables for individual projects listed in the program management plan. The Program Schedule will be derived from the program management plan at the beginning of each test administration year, and will be updated quarterly. The Department expects submission of the Program Schedule in Microsoft Excel. If significant changes occur between quarterly submissions of the program schedule, the Department may request an updated schedule reflecting the changes at any time. 2. The contractor will provide a Weekly Management and Action Item Report to the Department. A template for this report will be provided to the contractor, and the Department reserves the right to modify the design, format, and content of this report at any time. The Weekly Management Report and Action Item Report will report the status of recent and upcoming tasks, highlighting those leading up to deliverables and critical tasks, report progress on program issues, identify unresolved issues, and maintain a list of requests requiring change orders or amendments. The weekly report will be provided to the Department by no later than noon one day prior to the weekly program management conference calls and will reflect the previous week’s work. 3. The contractor will produce a Technical Report and provide the initial draft document to the Department by September 30 of each year of the contract arising from this ITN. The Department will work with the Florida Standards Assessments, ITN 2014-47 90 contractor to identify the specific information to be provided in the Technical Report each year. A proposed report structure is provided in Section 7.6.3.2. 4. The contractor will produce two copies of the Annual Program Review each year to be provided only on CDs/DVDs/jump drives inside plastic sleeves in loose-leaf binders. Along with other materials described below, the Program Review will include a record of the contractor’s services and deliverables in response to contractual/ITN requirements. This review must identify major program accomplishments and successes, as well as areas of the program that require changes and improvement in the upcoming year. The review is intended to be a history of program events and to describe recommended improvements or changes for subsequent years. The annual review will be based on information gathered by the contractor’s program and project management staff throughout the year. The review will address all aspects of program service and products including management, technology, interpretive products, communications, development, test production, test administration, materials shipping/pickup, scoring, and reporting processes. It will also follow the annual administrative cycle, beginning with fall and ending with the following summer. The Annual Program Review will include: – details of meetings held with educators (dates, locations, purpose) – milestones attained/deliverables completed for each project area – problems encountered, root cause analyses, and corrective action(s) taken – performance metrics reports as required by the Department noting changes over time The Annual Program Review will include electronic files of deliverables and documentation on services required by this ITN, beginning with the fall test administration and ending with the summer test administration. It will include: • PDFs of the test and answer books, • PDFs of all screenshots for all computer-based test approved forms, • PDFs of all passage booklets, reference sheets, large print test books, one-item-per-page test books, large print reference sheets, large print response grid instruction sheets, and CBT work folders, • PDFs (untagged and tagged, if applicable) of all interpretive and ancillary products, • Native application files of all interpretive and ancillary products, • PDFs, or other appropriate file formats such as video, of all other products, including memoranda and report templates, • Missing materials and resolution reports, • Final Program Schedule reflecting planned vs. actual dates of task completion, • Weekly Management and Action Item reports, • File Layouts, and • Specifications documents. 5. The collected Accounting Reports will provide a summary of the Quarterly Accounting Reports, including expenditures by program task and level of effort as well as expenditures and balances of Reimbursable Funding Categories identified in Section 7.7.9, including balances resulting from executed change orders and contract amendments. 7.7.7 Meetings Involving Florida Educators and Other Citizens The contractor will be responsible for arranging, organizing, and paying the cost of bringing participants to specific meetings as identified in this ITN. This section identifies points at which meetings with groups of Florida educators and other citizens will be conducted to propose development and/or management guidelines, to draft products, or to review work and products produced by the contractor. The Department will designate Florida educators, and in some instances Florida citizens or consultants from outside the state, to participate in these meetings. Florida Standards Assessments, ITN 2014-47 91 All meetings are facilitated by Department staff unless determined otherwise by the Department. Contractor staff will attend the meetings with Florida educators, as required. The contractor is responsible for on-site meeting organization, support, and management. Generally, the contractor’s manager or team leaders for the program area and an administrative assistant responsible for meeting planning will be present at meetings for overall management and support purposes. The contractor will also be expected to send staff appropriate to assist and participate in each meeting. Generally, for all meetings involving separate subject/grade groups, the contractor will provide the appropriate professional staff to take part in each of the separate subject/grade groups. The contractor will be responsible for all expenses associated with contractor staff travel and participation. All expenses must be in accordance with the Department of Financial Services Guidelines for State Expenditures which prohibits the purchase of food and beverages and other unallowable costs. The Guideline may be accessed through the following link: http://www.myfloridacfo.com/aadir/reference_guide/. The contractor will be responsible for arranging, organizing, attending, and paying travel, lodging and per diem costs required to bring these participants together for the meetings. For meetings to be held in Florida, they will typically be in a central Florida location, a hub city (e.g., Jacksonville), or in Tallahassee. Table 7.7.1 provides information about the number, expected length, approximate number of participants, and estimated cost of such meetings. Table 7.7.7 Meetings of Educators and Other Citizens Table 7.7.7 Meetings of Educators and Other Citizens Meeting Area 1 Item/ Passage Bias English Language Arts/ Literacy (ELA/L), Mathematics 7.3.4.1 1 Oct 2015, 2016, 2017 Oct 2018, 2019, 2020 2 Approx # of attendees/ ind mtg 30 2 Item/ Passage Community Sensitivity ELA/L, Mathematics 7.3.4.1 1 Apr 2015, 2016; May 2017 2 30 Jacksonville or Central Florida 3 Item Content Review ELA/L, Mathematics Test Items 7.3.4.2 1/ subject & grade Oct 2015, 2016, 2017 5 8 Jacksonville or Central Florida 4 Passage Content Review ELA/L 7.3.4.2 1 Aug 2015, 2016, 2017 5 15 Jacksonville or Central Florida 5 Data Review Mathematics 7.3.4.3 1 May or Jun 2015, 2016, 2017 2 9 Tallahassee 6 Standard Setting ELA/L and Mathematics 7.6.3.4 1/ subject Aug 2015 May 2018, 2019, 2020 Oct 2018, 2019, 2020 Aug 2018, 2019, 2020 May or Jun 2018, 2019 N/A 5 300 Central FL 7 Standards Reactor ELA/L and Mathematics 7.6.3.4 1 Aug 2015 N/A 3 30 Tallahassee ITN Ref. Mtgs/ Year Base Contract Florida Standards Assessments, ITN 2014-47 Renewal Days/ Mtg Likely Site Jacksonville or Central Florida 92 Table 7.7.7 Meetings of Educators and Other Citizens Approx # of attendees/ ind mtg 25/subject/ meeting ITN Ref. Mtgs/ Year Base Contract Renewal Days/ Mtg 3 per subject/fi rst year; 2 per subject after first year 2 TBD TBD 3 Spring and Nov 2015, 2016, 2017 Spring and Nov 2018, 2019 2 20 Tallahassee, Jacksonville, Central FL, or site of FERA Sep 2017; May and Sep 2018, 2019, May 2020 Sep 2017, 2018, 2020 5 10/subject per grade (3 grades/cou rses per subject) Tallahassee 5 8 Tallahassee Meeting Area 8 Content Advisory ELA/L, Mathematics 7.3.4.4 9 Technical Advisory Committee Technical 7.6.0 10 Field Test Rangefinding ELA/L, Mathematics 7.5.4.5 2/ subject May and Sep 2015, 2016, May 2017 11 Operational Rangefinder Selection ELA/L, Mathematics 7.5.4.5 1/ subject Sep 2015, 2016, Likely Site Tallahassee, Jacksonville, or Central FL 7.7.8 Communication between the Contractor and Districts Florida has 67 school districts conforming to the boundaries of Florida’s 67 counties. Within each school district is a district coordinator of assessment assigned to work directly with the Department to implement statewide assessment programs. All aspects of test administration and reporting are administered within the districts through the district coordinator of assessment. Other district personnel involved in statewide test administration include one testing coordinator per school and the teachers or guidance counselors who serve as test administrators. While there are some testing sites that are not part of the public school system (e.g., private schools), most communication will still be directed through the district coordinator of assessment. All communication between the contractor and school districts must be directed through or receive prior approval by the Department. Memos required to implement various aspects of the assessment will be prepared by the contractor and approved by the Department prior to release. The contractor will be required to email, mail, or fax memoranda or sets of instructions to district coordinators of assessment as instructed by the Department. Districts will be instructed to communicate directly with the contractor only for the purpose of clarifying meeting travel logistics, ordering additional testing materials, correcting computer-based system problems, or to report problems with the delivery/pickup of materials. For this purpose, the contractor will dedicate for the use of Florida’s district and school personnel a toll-free help desk telephone, operated Monday through Friday 6:00 AM through 8:00 PM, Eastern Time, a toll-free fax number, and a web-based help site available during the same hours cited for the help-desk telephone. This customer service is distinct from that described in Section 7.4.10. Florida Standards Assessments, ITN 2014-47 93 7.7.9 Reimbursable Funding Categories Six funding categories are designated for specified program functions and may be used only for those functions. Use of these funds requires authorization by the Department contract or program manager or program area leads. Authorization may be provided by specific email or other written communication, or by Department approval of a specific plan for a meeting(s) or activities requiring use of these funds. Amounts are designated for administration tasks. Respondents should include these amounts in their cost replies. Monies not expended from these funds will be deducted from final contract payment at the conclusion of the contract during contract resolution. The reimbursable fund balances will be reported by the contractor quarterly with each invoice submission in the Accounting Report. The amounts specified in this section may be increased or decreased by an appropriately negotiated amendment at any time during the lifetime of the contract. The Accounting Report will indicate expenditures from, and the balances remaining, in these funds. The final contract invoice will be accompanied by the final balance sheet for these funds. The amounts allocated for these funds should be identified separately in the respondents’ cost reply. 7.7.9.1 Consultant Services The contractor will provide to the Department technical advice and consultant service, as required, from outside the contractor’s staff. The contractor will recommend and the Department will select the consultant(s) to be used, who may or may not have been recommended by the contractor. A contract between the consultant and the contractor will be required and will include the amount allocated for honorarium, travel, lodging, and per diem as authorized by the Department. Reimbursement for travel will be at rates authorized by Section 112.061, F. S., for travel and per diem. The respondent should include in the reply the amount of $84,375.00 as a fixed amount for the base contract period and $28,125.00 as a fixed amount for each year of the optional renewal period. 7.7.9.2 Contingency Services It is possible that unforeseen circumstances will make it either necessary or desirable to perform tasks not specifically anticipated at this time, for example, data analyses beyond those described herein, unexpected data processing, acquisition of additional materials, or programming for test functionality not yet envisioned. These circumstances cannot be defined in advance, and the Department must be able to request such extra services in a timely fashion so as not to impede implementation of the assessments. In the reply, the respondent should include $84,375.00 as a fixed amount for the full term of the base contract period and will include $28,125.00 as a fixed amount for each year of the optional renewal period. 7.7.9.3 Author’s Alterations and Design Control The Department reserves the right to make changes in materials at any stage of the program prior to final production or delivery. Any changes to the Department approved, production-ready product (e.g., digital proofs or final computer-based test files) that are required due to contractor error will be paid for by the contractor. Otherwise, changes made to the Department approved, productionready product will be paid for as author’s alterations. If any alterations are necessary, the Department will identify in writing to the contractor the alterations to be made. In the reply, the respondent should include $84,375.00 as a fixed amount for the full term of the base contract period and will include $28,125.00 as a fixed amount for each year of the optional renewal period. 7.7.9.4 Provision for Meetings with Florida Educators The respondent will include a fixed amount to cover the costs of travel and per diem for Florida educators, Florida citizens, and others participating in the meetings identified in Section 7.7.7. The contractor will be responsible for the travel reimbursement for participants at rates authorized by Section 112.061, F. S., for travel and per diem. In the reply, the respondent should include $5,491,496.00 as a fixed amount for these meetings with Florida educators during the base contract period and $1,830,498.00 for each year of the optional renewal period. Florida Standards Assessments, ITN 2014-47 94 7.7.9.5 Provision for Meetings with Department Staff A fixed amount will be included in the reply to cover the costs of travel and per diem for the Department of Education staff or others participating in meetings with the contractor on behalf of the Department. Reimbursement for staff will be at rates authorized by Section 112.061, F.S. These funds will not be used to fund travel and per diem of the contractor’s staff. These funds will be expended only with the approval of the Department. In the reply, the respondent should include $438,750.00 as a fixed amount for ten (10) staff members to attend as many as fifteen (15) meetings per year requiring four (4) travel days each during the contract during the base contract period and $146,250.00 for each year of the optional renewal period. 7.7.9.6 Contractor Staff Positions To improve work flow, work quality, and communications, the contractor will fund salary and benefits for five (5) professional positions to work under the direction of the Bureau of K-12 Assessment within the Florida Department of Education. These positions will be three editors and two Systems Project Consultants. The Department will work closely with the contractor to interview and select individuals who are acceptable to both parties. The contractor will employ its standard recruiting, screening, and hiring procedures to secure individuals to fill these positions. The employees will be full-time employees of the contractor. The employees will receive benefits typically provided to the contractor’s employee classification system. The employees will be paid at a rate comparable to similarly skilled and experienced Department employees using the contractor’s employee classification system. The basic duties of the Computer-Based Test Editor (1) and the Materials Editors (2) include: (a) write and edit materials for content, such as items, tests, reports, training materials, brochures, scripts, or on-screen presentations; (b) develop, analyze, and select appropriate information and present it in a form and at a level suitable for the intended audience; (c) review written products and materials for conformity with standards of objectivity, style, and presentation; and, (d) perform a variety of editorial duties related to editorial production. Candidates must possess one (1) year of specialized experience directly related to these duties. Candidates may substitute two (2) full academic years of related graduate level education in lieu of the work experience requirement. The basic duties of the Systems Project Consultants (2) include: (a) participating in weekly meetings and communicating issues between the FDOE and the contractor’s program managers, (b) making suggestions for improvement to the contractor’s procedures to aid FDOE staff in data verification activities, (c) assisting FDOE staff debugging SAS computer programs and analyzing data files, (d) traveling, at the contractor’s expense, with FDOE staff to check scanning at the contractor’s site, (e) assisting in the verification of reports of results using SAS computer programs, (f) assisting in the verification of posting of files and reports on the portal, and (g) communicating issues with District Coordinators of Assessment related to scoring and reporting activities and issues. Candidates must have a thorough understanding of large scale standardized testing programs and data verification techniques. The candidate must also have experience managing data activities such as those required for this project, verifying data files, and using SAS, Excel, and other Microsoft Office products. The respondent should include in the response $1,800,000.00 as a total amount during the base contract period and $600,000.00 for each year of the optional renewal period. 7.7.10 Final Delivery of Materials The contractor agrees to deliver to the Department, upon request, all materials and products in all forms that are developed for and used in conjunction with this program, including test books, answer documents (paper and PDF), final electronic source files of interpretive products, software source code, and ancillary materials, PDFs, and electronic files within 30 days following acceptance by the Department of the final report for the program. Payment of the final program Florida Standards Assessments, ITN 2014-47 95 invoice will not be made until all materials and certification of destruction, as appropriate, are received and approved by the Department and final payment resolution is agreed to by both parties. At the completion of the contract, all data files, including those containing items or graphics, remaining in the possession of the contractor will be destroyed once all handoffs are made to the next contractor. Written verification of destruction will be provided to the Department as part of the final contract report. Destruction will not take place until authorized by the Department. 7.8 DELIVERABLES (See Attached Section 7.8 Deliverables provided in an Excel Spreadsheet.) SECTION 8 – OPENING, EVALUATION AND AWARD 8.0 REPLY OPENING TECHNICAL & PRICE REPLIES WILL BE OPENED BY THE DEPARTMENT'S BUREAU OF CONTRACTS, GRANTS AND PROCUREMENT MANAGEMENT SERVICES PERSONNEL AT 325 WEST GAINES STREET, 332 TURLINGTON BUILDING , TALLAHASSEE, FLORIDA, AS SPECIFIED IN SECTION 3.4 CRITICAL EVENT DATES 8.1 REPLY EVALUATION AND NEGOTIATION PROCESS Using the evaluation criteria specified below, in accordance with Section 287.057, F.S., the Department shall evaluate and rank responsive Replies and, at the Department’s sole discretion, proceed to negotiate with one or more Respondent(s) selected, as follows: A. Selected Respondent(s) will be invited to commence contract negotiations. If necessary, the Department will request revisions to the approach submitted by the selected Respondent(s) until it is satisfied that the contract will serve the Department’s needs. The process will continue until a contract is negotiated and executed. The Department may in its sole discretion, award and enter into contracts with more than one Contractor, if in the best interest of the state. B. The Department reserves the right to negotiate with all responsive and responsible Respondents, serially or concurrently, to determine the best-suited solution. The ranking of Replies indicates the perceived overall benefits of the proposed solution, but the Department retains the discretion to negotiate with other qualified Respondents as deemed appropriate. C. Before award, the Department reserves the right to seek clarifications, to request Reply revisions, and to request any information deemed necessary for proper evaluation of Replies. Respondents may be requested to make a presentation, provide additional references, provide the opportunity for site visits, etc. The Department reserves the right to require attendance by particular representatives of the Respondent. Any written summary of presentations or demonstrations shall include a list of attendees, a copy of the agenda, and copies of any visuals or handouts, and shall become part of the Respondent’s Reply. Failure to provide requested information may result in rejection of the Reply. D. The focus of the negotiations will be on achieving the solution that provides the best value to the state. E. In submitting a Reply Respondent agrees to be bound to the terms of this ITN, however, the Department reserves the right to negotiate different terms and related price adjustments if the Department determines that it is in the state’s best interest to do so. F. The Department reserves the right to reject any and all Replies, if the Department determines such action is in the best interest of the state or the Department. The Department reserves the right to negotiate concurrently or separately with competing Respondents. The Department reserves the right to accept portions of a competing Respondent’s Reply and merge such portions into one project, including Florida Standards Assessments, ITN 2014-47 96 contracting with the entities offering such portions. The Department reserves the right to waive minor irregularities in Replies. Criteria for Evaluation The Department shall evaluate and rank responsive Technical Replies and score them on a scale of 1 to 100 using the following criteria. The Department anticipates awarding the Contract (if any award(s) is made) after negotiations to the responsible and responsive vendor determined to provide the best value to the state. Each Respondent will receive one score for all criteria, other than price, regardless of the number of assessment components it proposes to deliver. This score will constitute the Total Technical Score for the Respondent. Respondents will also receive a separate price evaluation for each assessment component it submits a price for (A-E in Attachment 1). The Department will add the Total Technical Score for each Respondent to the price evaluation score for an assessment component to determine the total score for the Respondent for that assessment component. Respondents will be ranked for each assessment component in this manner. The following criteria will be used to evaluate and rank Replies. A) B) C) D) Executive Summary Qualifications and Experience Technical Plan Management Plan Total N/A 10 points 50 points 15 points 75 points E) Price 25 points 25 points Total Price analysis is conducted through the comparison of Price Replies submitted. The Department shall score responsive Price Replies up to 25 points using the below. Price analysis is conducted through the comparison of Price Replies submitted. The method of awarding points for the Price Reply will be done in accordance with the provisions of Section 287.0572, F.S., which requires use of the present value methodology. The present value discount rate will be used in the computations and evaluation. To determine that discount rate, use the rates identified in Release H.15, Select Interest Rates (Weekly), available online at http://www.federalreserve.gov/releases/h15/. Use the most recent copy of the Federal Reserve Bulletin published at the time of issuance of the ITN. See the “Critical Event Dates” for the release date. The maximum points will be awarded to the lowest acceptable Price Reply. Replies with higher costs will receive the fraction of the maximum points proportional to the ratio of the lowest Price Reply to the higher Price Reply. The fractional value of points to be assigned will be rounded to one decimal place. The criteria for price evaluation shall be based upon the following formula: (Low Price/Respondent's Price) x Price Points = Respondent’s Awarded Points The total maximum number of points that can be earned in the evaluation process is 100 points. 8.2 EVALUATIONS The ITN is designed to assess the most points to the Respondent presenting the best solution for the required services. The Evaluation Committee will consider only those Replies provided after the preliminary administrative review (See SECTION 6.2.1) first completed by the Department’s Bureau of Contracts, Grants and Procurement Management Services. Each member of the Evaluation Committee will be provided a copy of each Technical Reply. Replies will be evaluated on the criteria established in the section above entitled "Criteria for Evaluation" in order to assure Florida Standards Assessments, ITN 2014-47 97 that Replies are uniformly rated. The Evaluation Committee will assign points, utilizing the technical evaluation criteria identified herein and the Procurement Office will complete a technical summary. The Department reserves the right to request oral presentations/seek clarification on any Reply submitted. Information requested and received will be evaluated by the Committee based on the criteria established in SECTION 6.2.2 above. During this stage Respondents will be asked to provide any clarifications needed by the Evaluation Committee to assist in evaluating their Reply. Information received in this stage will be added to the Respondent’s Reply and evaluated as a part of the appropriate section above. 8.3 POSTING OF INTENDED AWARD The intended award will be posted in the Bureau of Contracts, Grants and Procurement Management Services, 325 West Gaines Street, 332 Turlington Building, Tallahassee, Florida and on the VBS and will remain posted for a period of seventy-two (72) hours. (See SECTION 3.4 CRITICAL EVENT DATES.) 8.3.1 Protest of Intended Award Any Respondent who is adversely affected by the Department's recommended award or intended decision must file a written “Intent to Protest” with the Department at the address of posting. See SECTION 4.4 PROCUREMENT PROTESTS/NOTICE OF RIGHTS for protest information. 8.3.2 Inability to Post If the Department is unable to post as defined above, the Department will post a public notice on the VBS. The Department will provide written notification of any future posting in a timely manner. 8.4 AWARD OF THE CONTRACT Services will be authorized to begin when the Contractor receives a fully executed contract from the Department’s Contract Administrator. **** Florida Standards Assessments, ITN 2014-47 98 APPENDIX A: TRANSITION TO COMPUTER-BASED TESTING AND ADDITIONAL TEST ADMINISTRATION DATES All high school and End-of-Course assessments are currently computer-based assessments and are anticipated to remain as such. Grade 11 English Language Arts/Literacy and Algebra 2 End-of-Course Assessments will be implemented as computer-based assessments. Table A.1: Transition to FSA Computer-Based Tests in Florida Grades 3-8: ELA/L and Mathematics Grade PBT PBT PBT PBT Transition to CBT Mathematics PBT PBT PBT Transition to CBT CBT PBT PBT Transition to CBT CBT CBT Mathematics PBT PBT PBT Transition to CBT CBT PBT Transition to CBT CBT CBT CBT Mathematics CBT CBT CBT CBT CBT CBT CBT CBT CBT CBT Mathematics Transition to CBT CBT CBT CBT CBT CBT CBT CBT CBT CBT Mathematics PBT Transition to CBT CBT CBT CBT ELA/L 8 2017-2018 ELA/L 7 2016-2017 ELA/L 6 2015-2016 ELA/L 5 2014-2015 ELA/L 4 2013-2014 ELA/L 3 Subject Transition to CBT CBT CBT CBT CBT Mathematics PBT PBT Transition to CBT CBT CBT Key CBT = Was computer-based test in prior year Transition to CBT = First year of computer-based test in this grade/ content PBT = Administered as a paper-based test Florida Standards Assessments, ITN 2014-47 99 The following dates are for reference only and refer to assessments that are not included as part of this ITN. Dates for the assessments that are included in this ITN are provided in Table 7.2.2. Table A.2: Additional Administration Dates for FCAT 2.0 and Florida EOC Assessments (Assessments not included in this ITN) Florida End-of-Course Assessments (U.S. History, FCAT 2.0 Civics, Biology 1, Algebra 1 Retake, Geometry Retake) Fall Spring Year Reading Reading Spring Science Fall Winter Spring Summer Retake Retake Nov. 30– April 25–May July 11–22, Oct. 5–16, April 11– April 11–15, Sept. 14–25, 2015-16 Dec. 18, 27, 2016 2016 2015 15, 2016 2016 2015 2015 Nov. 28– April 24–May July 10–21, April 17–21, Sept. 12–23, 2016-17 NA NA Dec. 16, 26, 2017 2017 2017 2016 2016 Nov. 27– April 30–May July 16–27, April 16–20, Sept. 11–22, 2017-18 NA NA Dec. 15, 25, 2018 2018 2018 2017 2017 Nov. 28– April 29–May July 15–26, April 15–19, Sept. 10–21, 2018-19 NA NA Dec. 19, 24, 2019 2019 2019 2018 2018 April 13–17, Sept. 9–20, Dec. 2–20, April 27–May July 13–24, 2019-201 NA NA 2020 2019 2019 22, 2020 2020 Nov. 30– April 26–May July 12–23, April 12–16, Sept. 14–25, 2020-211 NA NA Dec. 18, 21, 2021 2021 2021 2020 2020 1 Optional Renewal Period Florida Standards Assessments, ITN 2014-47 100 APPENDIX B CRITICAL AND EXTREMELY CRITICAL ACTIVITIES Final dates for these activities and a final activities list will be determined in the negotiation process of this ITN and included in the final contract. The table below describes the critical and extra critical tasks, the frequency for distribution, and any notes about the approximate delivery dates. Failure to provide these materials by the agreed-upon dates will be cause for the assessment of liquidated damages. Task Activity Critical (C) or Extra Critical (XC) Delivery of practice tests and sample test materials to districts. C Frequency Varies by deliverable Delivery Notes (Approximations) By November/December for assessments that are introduced in the spring, regular sample tests (PDF or computer-based) are delivered the initial year of the assessment or the first year of a format change (i.e., transition from paper-based to computer-based). ITN Ref. 7.3.7.5, 7.4.9.6 Special format print materials (e.g., Braille, large print) are delivered upon request each year. Delivery of the printed End-of-Course test administration manuals and ancillary materials to districts for the paper-based and computer-based administrations. C Prior to each fall administration (see note in 7.4.1 regarding the possibility of 2 manuals for 2014-15 school year) Usually 70 calendar days before each administration. 7.4.1 Delivery of the printed ELA/L and Mathematics test administration manuals and ancillary materials to districts for the paper-based and computer-based administrations. C Prior to each spring administration Usually 70 calendar days before each administration. 7.4.1 Delivery of the printed ELA/L Retake test administration manuals and ancillary C Prior to each fall administration (beginning Usually 70 calendar days before each administration. 7.4.1 Florida Standards Assessments, ITN 2014-47 101 Task Activity Critical (C) or Extra Critical (XC) materials to districts for the paper-based and computer-based administrations. Frequency Delivery Notes (Approximations) ITN Ref. in fall 2015) Delivery of the printed test materials to districts. XC Prior to each administration. Usually 3 weeks before each administration. 7.4.1 Delivery of the preidentification labels to districts. XC Prior to each administration Usually 3 weeks before each administration. 7.4.1 Delivery of the computer-based tests to districts. XC Prior to each administration The first day of the test administration window, unless the system requires pre-loading the assessments. (At that point, they should be delivered in sufficient time for the schools to load the test prior to the administration.) 7.4.9 After each administration Varies by results release deliverable. Initial results delivered as early as two weeks after the end of the administration. For EOC administrations: • RD1-RD3 (state, district, and school reports) must be delivered within two weeks of the last day of the administration window, per statute. • RD4 (hard copies of student results) must be delivered within 1 week of RD1-3. • RD5 (late reporting) should be completed within 2 months of RD4. • RD7 (demographic reports) will be delivered in August/September for the previous school year. For ELA/L and Mathematics (Spring): 7.5.5 Delivery of test results to districts. RD1 – XC Results Delivery (RD) details for 1-7 are provided in Table 7.5.5. This includes print and electronic deliveries. RD2 – XC RD3 – XC RD4 – XC RD5 – C RD6 – C RD7 – C Florida Standards Assessments, ITN 2014-47 102 Task Activity Critical (C) or Extra Critical (XC) Frequency Delivery Notes (Approximations) • • • • • Florida Standards Assessments, ITN 2014-47 RD1-RD3 (state, district, and school reports) delivered between the end of April and the middle of May. RD4 (hard copies of student results) must be delivered before the week of June 8, per statute. RD5 (late reporting) should be completed within 3 months of RD4. RD6 (writing image media) is usually delivered mid-July. RD7 (demographic reports) will be delivered in August/September for the previous school year. 103 ITN Ref. APPENDIX C FLORIDA PUBLIC SCHOOL STUDENT PARTICIPATION ESTIMATES, DISTRICT NUMBERS, AND SPECIAL DISTRICT/SCHOOL NUMBERS STUDENT PARTICIPATION ESTIMATES Table C.1 indicates the anticipated number of students that would be tested in each grade-level test administration. The number of students expected to participate in ELA/L Retake administrations (Table C.2) will be calculated as a percentage of the grade 10 population. Table C.3 indicates the expected test takers for each EOC Assessment. This calculation assumes 200,000 new students for each spring cohort. (Cohort, for the purposes of this ITN, refers to a group of students that enters grade 9 and is enrolled at the high school for a total of four years.) Table C.1 Anticipated Number of Students Participating in Assessment Administrations Table C.1 Anticipated Number of Students Participating in Assessment Administrations Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 2013-14 212,774 201,371 198,642 203,053 210,018 211,085 222,533 206,675 198,131 2014-15 2015-16 2016-17 217,735 225,524 223,753 202,712 207,553 215,105 203,183 204,609 209,564 202,436 207,154 208,854 206,962 206,010 210,865 213,634 210,031 209,364 228,413 229,570 226,066 216,413 218,707 219,248 202,282 205,215 206,493 2017-181 224,786 213,553 217,105 213,364 212,354 214,365 225,364 216,066 207,248 2018-191 225,132 214,786 2019-201 224,893 215,132 1 Optional Renewal Period 215,553 216,734 221,105 219,654 216,864 224,634 215,854 220,268 230,365 231,467 215,364 220,540 206,707 203,211 Table C.2 Anticipated Number of Students Participating in ELA/L Retake Administrations Table C.2 Anticipated Number of Students Participating in ELA/L Retake Administrations Fall Retake Spring Retake 1 ELA/L 65% 45%1 1 As a percent of anticipated Grade 10 enrollment for applicable years from Table C.1. Florida Standards Assessments, ITN 2014-47 104 Table C.3 Anticipated Number of Students Participating in EOC Administrations Table C.3 Anticipated Number of Students Participating in EOC Administrations School Year Spring Summer Fall Winter 2015-16 Spring Summer Fall Winter 2016-17 Spring Summer Fall Winter 2017-181 Spring Summer Fall Winter 2018-191 Spring Summer Fall Winter 2019-201 Spring Summer 1 Optional Renewal Period 2014-15 Algebra 1 200,000 100,000 75,000 56,250 242,188 131,641 98,730 74,048 255,536 141,652 106,239 79,679 259,759 144,820 106,239 79,679 259,759 144,820 106,239 79,679 259,759 144,820 Florida Standards Assessments, ITN 2014-47 Algebra 2 200,000 50,000 12,500 3,125 200,781 50,195 12,549 3,137 200,784 50,196 12,549 3,137 200,784 50,196 12,549 3,137 200,784 50,196 12,549 3,137 200,784 50,196 Geometry 200,000 50,000 12,500 3,125 200,781 50,195 12,549 3,137 200,784 50,196 12,549 3,137 200,784 50,196 12,549 3,137 200,784 50,196 12,549 3,137 200,784 50,196 TOTAL 600,000 200,000 100,000 62,500 643,750 232,031 123,828 80,322 657,104 242,044 131,337 85,954 661,328 245,212 131,337 85,954 661,328 245,212 131,337 85,954 661,328 245,212 105 DISTRICT NUMBERS AND SPECIAL DISTRICT/SCHOOL NUMBERS NOTE: The districts/schools participating in each test administration may vary. For example, some administer only the tests required for graduation. 01-Alachua through 67-Washington 68- Florida School for the Deaf and the Blind (FSDB) 68-0011 Deaf Elementary School 68-0014 Blind Elementary School 68-0012 Deaf Middle School 68-0015 Blind Middle School 68-0013 Deaf High School 68-0016 Blind High School 68-0019 Career Education and Transition High School 69-0011 Okeechobee Youth Development Center 71- Florida Virtual Schools 71-0300 FLVS Full-Time K-8 71-0400 FLVS Full-Time 9-12 71-0500 FLVS Part-Time 6-8 71-0600 FLVS Part-Time 9-12 71-0700 FLVS Part-Time K-5 72-75 University Laboratory (Lab) Schools 72-0011 A.D. Henderson University School & FAU High School 72-0020 FAU/SLCSD Palm Pointe Educational Research School @ Tradition 73-0341 Florida State University School (FSUS) 73-0351 The Pembroke Pines Florida 74-0351 Florida A&M University (FAMU) Developmental Research School 75-0391 University of Florida (UF) Lab School P.K. Yonge Developmental Research School 80- State Colleges with Adult High School Programs 80-0005 Daytona State College (Volusia) 80-0007 Florida State College at Jacksonville (Duval) 80-0011 Indian River State College (St. Lucie) 80-0020 Pensacola State College (Escambia) 80-0025 Seminole State College (Seminole) 80-0026 South Florida State College (Highlands) 97 – Office of Independent Education (Florida Tax Credit Scholarship students) 98 – Ahfachkee School Special Programs (The following special programs are designations within each district. The “XX” is replaced by the two-digit district number.) XX-3518 – McKay Scholarship Students XX-9998 – Home Education Program Students Florida Standards Assessments, ITN 2014-47 106 APPENDIX D SAMPLE TEST DESIGN Table D.1 below is provided as an example of one possible test design that may meet the criteria described in Section 7.3.0. Respondents may propose other designs, and a final design will be discussed in negotiation and agreed upon between the Department and the selected contractor. Table D.1 Example Testing Sessions – Grades 3-8 and High School Table D.1 Example Testing Sessions – Grades 3-8 and High School Content Area Specification Session 1 Session 2 Session 3 English Language Arts/Literacy Time 80 minutes 80 minutes 80 minutes Item Types Multiple Choice and Short Constructed Response Extended Writing in Response to Text Mathematics Time 90 minutes Multiple Choice and Short Constructed Response 90 minutes Item Types Multiple Choice, Gridded/Fill-in Response, Short Constructed Response, and 1-2 Extended Performance Tasks Florida Standards Assessments, ITN 2014-47 Multiple Choice, Gridded/Fill-in Response, Short Constructed Response, and 1-2 Extended Performance Tasks 107 APPENDIX E PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS Appendix E is organized as follows: Part 1 Operational Test Materials Product Specifications & Distribution Requirements Part 2 Florida Department of Education PDF Guidelines for Section 508/ADA Documents Part 3 Ancillary Materials Shipment Schedule & Print Quantities APPENDIX E, Part 1 OPERATIONAL TEST MATERIALS PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS Test Documents – General Considerations The contractor is responsible for producing camera-ready forms of these tests, printing the test documents, and distributing them to school districts. The following requirements apply to al l test documents: • A method to collect demographic information and other special information (e.g., invalidation) o Preferred: Scannable document covers include the student demographic grids and other special coding sections. • A method to track and log secure materials o Preferred: Unique security barcode numbers will be printed on each book containing test questions. • Large Print and Braille (contracted and uncontracted) version of the test. Copies of the regular book (the same form) must accompany the shipment of the large print and Braille materials. Braille notes also accompany the Braille versions. o Preferred: One form of each test (usually form 1) is produced in large print (minimum of 18 point type on 14” x 17” approved paper) and in Braille. • Forms will be spiraled for random distribution. • Documents will be shrink-wrapped in packs of 5 and 25. Quantities smaller than 25 may be desirable and will be approved by the Department when the document size makes lifting and packing difficult or hazardous. • Test documents must be high-quality and durable o Preferred: Test documents are 8 ½” x 11,” saddle stitched or bound; 60# white opaque cougar or approved equivalent paper will be used for answer documents and interior pages of test books. • Test documents should have high-quality, attractive printing and design o Preferred: Covers are printed in one color of ink plus black; Interior pages are printed in one color, some with both scannable and non-scannable ink; Color coding or other identification marks are included on the spine of the documents to identify them when stacked. Test Documents – Specific Considerations: 1. Grades 3-11 ELA/L Test Documents • Up to 40 forms of scannable, sealed test and answer and/or answer documents • Number of document types may vary if different portions of a test are in separate documents (e.g., writing folders) • 1 Reading Passage Booklet as a CBT test accommodation. • 1 standalone CBT Worksheet for each CBT tester. 2. Grades 3-8 Mathematics Documents • Up to 40 forms of scannable, sealed test and answer and/or answer documents • Number of document types may vary if different portions of a test are in separate documents (e.g., performance tasks) • 1 standalone CBT Work Folder for each CBT tester. • One standalone reference sheet per student, as applicable Florida Standards Assessments, ITN 2014-47 108 3. Grade 10 ELA/L Retake • Preferably, one form of a scannable, sealed test and answer and/or answer document • Number of document types may vary if different portions of a test are in separate documents (e.g., performance tasks) • 1 Reading Passage Booklet as a CBT test accommodation. • 1 standalone CBT Worksheet for each CBT tester. 4. End-of-Course Tests • 1 paper accommodation form of the Algebra 1 End-of-Course test and answer book (scannable, with an outside seal). • 1 paper accommodation form of the Geometry End-of-Course test and answer book (scannable, with an outside seal). • 1 paper accommodation form of the Algebra 2 End-of-Course test and answer book (scannable, with an outside seal). • 1 standalone CBT Work Folder for each CBT tester. • 1 standalone reference sheet for each Algebra 1, Geometry, and Algebra 2 tester. 5. Other Accommodations • 1 large print paper accommodation form for each test • 1 Braille form divided into two volumes for each test • 1 one-item-per-page accommodation form for each test 6. Accommodation Standalones • 1 large print reference standalone sheet to accompany each applicable assessment form • 1 large print response grid instructions standalone sheet to accompany each applicable assessment form • 6 pages of blank Braille paper to accompany each Braille assessment form • 1 large print form of the CBT work folder for each CBT Mathematics assessment • 1 large print form of the CBT worksheet for each CBT ELA/L assessment Shipment Schedule and Print Quantities for ELA/L and Mathematics Assessments • All test materials are shipped in TM2 • All test documents may contain up to 64 pages • All grades 3-11 tests will have up to 40 test forms • Grade 10 ELA/L Retake will have one test form • Schools will receive one test form (with standalones, as applicable) per preidentified student • Districts will receive 15% overage of all test materials • For paper-based administration subjects, 10% extra test material is produced Shipment Schedule and Print Quantities for EOC Assessments • All test materials are shipped in TM2 • All test documents may contain up to 64 pages • Spring EOC tests will have up to 4 unique test forms • Fall, Winter, and Summer EOC tests will have one test form each • Schools will receive one test form (with standalones, as applicable) per preidentified student • Districts will receive 15% overage of all test materials • For paper-based administration subjects, 10% extra test material is produced Florida Standards Assessments, ITN 2014-47 109 APPENDIX E, Part 2 FLORIDA DEPARTMENT OF EDUCATION PDF GUIDELINES FOR SECTION 508/ADA DOCUMENTS The following guidelines may be revised throughout the contract to reflect changes in technology or requirements. The contractor will be notified of any changes, as appropriate. In order for PDF documents to be posted on the DOE website or any site that is affiliated with the DOE, they must pass the Full Accessibility check without error in Adobe Acrobat 9.0 or X. DOE uses Adobe Acrobat 9.0 and X to check accessibility of PDF documents. Currently, Adobe Acrobat 8.0 has known issues with the accessibility checker. Although a document may pass the full check, this does not necessarily mean the document is compliant according to DOE Web standards. The following additional checks are required: 1. PDF documents must have the correct reading order. The file must be able to be read by assistive technology in a logical manner. This can be checked in several ways, which include the following items. DOE uses at least two of the following to verify reading order: a. Saving the file as an accessible .txt file and reading it to be sure it is correct b. Using the “reading order tool” in Adobe Acrobat c. If proficient using a “screen reader” (not the Adobe Acrobat “read out loud”), listen to the document d. Using the Reflow view in Adobe Acrobat e. Using the “Order” navigation panel 2. The document must have correct Tab order. Tab order refers to how a user can “tab” through the content. The user should be able to tab through the document in the order it is intended to be read. If the tab order is incorrect, the assistive technology user may jump from one page to another and back and not realize it. Another example of incorrect tab order is the practice of structuring tables or columned lists using tabs. 3. The document must have appropriate “alt text” on all images that have meaning. (Alt text means alternative text that appropriately describes the image for an assistive technology user.) a. Graphs, charts, flowcharts, etc., cannot simply have the word “graph” in alt text. It must contain enough descriptive information for an unsighted user. b. Graphics that do not have meaning need to be tagged as “artifacts” or “backgrounds.” If the primary purpose of an image is decoration or if describing an image will take the listener out of the testing experience and serve more as a distraction to the task at hand, then it would fall under this rule. During regular production rounds, TDC content leads will determine which elements will be tagged as background. c. Putting a blank space for the “alt text” area of an image in an attempt to get the screen reader to skip it is not acceptable in “most” cases, as a screen reader will still say the word “graphic.” Tagging graphics that have no meaning as artifacts or background will cause the screen reader to ignore them. d. Graphs and charts need to be described completely to convey the same information to a listener as the sighted user would get. If the graph is explained in detail in the “content text” of the document, you can refer to that content in the alternate text. For example, if right below a pie chart is a paragraph explaining the chart, then the “alt text” on the chart could simply state “Pie Graph which is explained in detail below.” This is very helpful if the person converting the file to PDF is not the one who created it. e. Alternate text should not be provided on anything except for image tags. f. Links are not “clickable” in alternative text; therefore, including links in alternative text should be avoided. Permission must be obtained from the DOE for special circumstances, such as the approach for tagging the Periodic Table of the Elements in Interpretive Products. 4. If the State Seal, or any other image, is in the header and footer of a document, the first occurrence of the header and footer needs to be tagged and readable by a screen reader. All other occurrences should be tagged Florida Standards Assessments, ITN 2014-47 110 as artifacts or backgrounds. This will let the listener know that there is at least one state seal (or other image) present, but does not repeat on every page that it appears. This rule also applies to headers and footers that are repeated on each page of a document. Only new information in the header and footer selections, such as page numbers, should be tagged. 5. Page numbers need to be tagged in such a way that the screen reader reads them. This is helpful in navigating the document. 6. Tables must be tagged as tables, so they are read correctly. A common problem with tables is that they are being tagged as paragraphs. Tables and table components will be tagged as such: Table headers will have the tags, table data will have tags , and table rows will have tags. The proper order for tagging a table is as follows: , , and
; or , , , and the body section should begin with . To check a table, compare the layout in the normal view against the layout in the View menu and select Reflow. If the layout stays the same and does not shift, the table is properly tagged. 7. Documents greater than 10 pages in length should contain a clickable table of contents, except when it is not feasible. The table of contents should contain the correct tag structure using and tags. For documents that contain a table of contents that is not clickable, the table of contents needs to be made into Bookmarks in the PDF and each table of contents listing needs to be made clickable within the table of contents. The Bookmarks Panel should appear on the left side of the screen for documents that do not contain a table of contents but are greater than 10 pages in length. This Bookmarks Panel will act as the document’s table of contents. During regular production rounds, TDC content leads will determine which elements in TDC documents will be bookmarked. This is needed not only for easy navigation for those using assistive technologies, but it is a convenience to sighted visitors as well, since it allows them to click on an item in the table of contents and be directed immediately to the content of the document to which it refers. 8. Content headings must be tagged as headers and not tagged as paragraphs. Headings need to be tagged with tags (or Heading 1

, Heading 2

, etc.). This helps the listener know what subject matter is contained beneath the header and allows for easier navigation. 9. Files must be saved (in a reduced file size) to Adobe Acrobat version 7.0 with "Fast Web View" enabled for documents tagged in Adobe Acrobat 9.0 or X. (The DOE has discovered that converting files created in Adobe Acrobat 9.0 to Adobe Acrobat 5.0 increases rather than decreases the file size, and converting files created in Adobe Acrobat X to Adobe Acrobat 6.0 or earlier versions distorts the font in interpretive products, so this should be avoided.) Reducing the file size ensures that users who have older versions of Acrobat are able to access the document. This also decreases the file size for quicker download. 10. Links must be functional. a. Hyperlinks must not be broken and must link to the correct page. This includes e-mail addresses that are clickable. b. If a link is split between two lines, both lines need to be linked correctly. c. Hyperlinks need to be standard, which means in blue font and underlined. This allows for sighted users to quickly locate links within the document. 11. Color should not inhibit the readability of text, and color alone is not to be used to convey meaning. Some assistive technologies do not reproduce colored text, and users who are colorblind often cannot differentiate every color. a. If color is used to convey meaning for sighted users, add a symbol in front of the colored items as well. ag this symbol to indicate the meaning of the colored item. Florida Standards Assessments, ITN 2014-47 111 b. Under certain DOE Web Services approved circumstances, the color-coded information may be described using white text that is not viewable to the sighted user. This “invisible” information must be tagged so that it is read prior to the colored text. 12. The following information needs to be noted in the Document Properties: Title–Users can set up a screen reader to read the titles instead of the long file names. Subject–Specific to the document, and may or may not be the same as the document title. Keywords–As with other Web documents, these words are used for search engines and need to be specific to the document’s contents. Language–(Advanced tab) Specified to indicate the primary language of the document. The individual tag properties need to indicate text within a document that may be of a different language (e.g., paragraph in Spanish needs to specify “Spanish” and paragraph in Haitian Creole needs to specify “French”). Author–At this time, the DOE is not requiring the inclusion of “Author;” however, if the name of the author is pertinent to the overall document, it is suggested that a name be included. 13. Lists should be tagged correctly and not tagged as text. The correct order options for list tags are as follows: ,
  • , ; ,
  • , , ; or ,
  • , . If lists have lists within list items, they need to be tagged correctly. Always tag multiple-choice answer choices/options as lists. Sample multiple-choice items appear in Sample Test Materials and in Item Specification books. 14. Forms that need to be filled out should contain the appropriate form fields and check boxes so users may fill them out on the Web. Examples of forms should be indicated as such and do not need to contain form fields and check boxes, because they are samples. The answer grids for sample testing materials are not considered forms at this time for feasibility reasons. Form field properties need to be filled out so that the information that needs to be entered into the form fields is clear. (The "Tooltip" in the form's properties is the portion of the tag that is read by the screen reader.) If two form fields have the same "Name" in a document, then both fields will populate when one of them is filled out; therefore, each form field must have a unique name in the form field's properties. When tagging form fields, it is important to check the accuracy of the Tab order (see list item 2). 15. Pages cannot be blank. They must contain some text, such as “Notes” in the header or the disclaimer, “This page is intentionally blank.” 16. Pages with content in landscape format should be presented in landscape format, so users may read the content without having to print the page. In other words, the Pearson contractor shall rotate landscaped pages inserted sideways so the content appears right side up to the viewer. 17. Words set in bold, underline, or italics may not be used alone to convey meaning. For example, a word set in italics cannot have a coded meaning such as “italics= story title.” a. Add a symbol in front of the italicized, bolded, or underlined information, and tag the symbol to indicate the meaning italics, bolding, or underlining should convey. b. Under certain DOE Web Services approved circumstances, the italicized, bolded, or underlined information may be described using white text that is not viewable to the sighted user. This “invisible” information must be tagged so that it is read prior to the colored text. 18. No text should be entered in the “Actual Text” field in the Touchup Properties box. If that is done, some readers will only read the actual text and then stop instead of reading the entire document. Note: DOE PDF accessibility guidelines were initially effective September 11, 2007. They were revised by DOE Web Services to include Adobe Acrobat 9.0 on January 5, 2010, and by DOE Editorial staff on February 17, 2010, for added clarifications and for the purposes of this document. Some DOE staff are now using Adobe Acrobat X, and the Department is currently in the process of officially updating PDF accessibility guidelines to include Adobe Acrobat X (3/27/12). Florida Standards Assessments, ITN 2014-47 112 APPENDIX E, PART 3 – Ancillary Materials Shipment Schedule & Print Quantities Product Shipment Pages Number of Copies State District Schools Electronic Delivery (508/ADA compliant) Test Administration Manual TM1 Up to 400 25 10% overage 1/20 students Student PreID Labels (Wave 1) TM2 N/A N/A N/A 1/student Student PreID Rosters (Wave 1) Document Count Forms (Preidentified) TM2 N/A N/A N/A TM2 N/A N/A N/A Document Count Forms (NonPreidentified) TM2 N/A N/A N/A 1 per subject 2 per subject/grade 5 per subject/grade Online Comment Forms N/A N/A N/A N/A N/A Yes Security Checklist N/A variable 0 1 1 Yes District Packing List TM1, TM2, TM3 variable 0 1 0 Yes School Packing List TM1, TM2, TM3 variable 0 1 1 Yes District Materials Return Kit TM2 N/A 1 1 0 School Materials Return Kit Return Materials (e.g., envelopes, boxes, special formats) TM2 N/A 1 0 1 TM2 N/A 0 As necessary As necessary N/A 0 As necessary As necessary Bills of Lading Paper Bands TM2 N/A 10 15% over 1/20 student documents Student PreID Labels (Wave 2) TM3 N/A 0 0 1/student (As necessary) TM3 N/A 0 0 1/subject (As necessary) As necessary Yes As necessary (Spring ELA/L and Mathematics only) Student PreID Rosters (Wave 2) (Spring ELA/L and Mathematics only) Misc. Memoranda Florida Standards Assessments, ITN 2014-47 variable 113 APPENDIX F TEST SECURITY REQUIREMENTS, STATUTE, AND RULE Chapter 1008.24 of Florida Statutes and Florida State Board of Education Rule 6A-10.042 establish the requirement that Florida Department of Education tests are to be maintained in a secure manner during development, administration, and scoring in order to preserve the integrity of the tests. When not in use, all test materials are to be kept in secure, locked storage. Individuals who have access to secure test materials are not to copy or otherwise reproduce test questions or reveal test questions verbally or in writing. Persons who are involved in administering or proctoring the tests or preparing examinees for the tests are not to participate in, direct, aid, counsel, assist in, or encourage any activity which could result in the inaccurate measurement or reporting of the examinees' achievement. Examinees' answers to questions are not to be interfered with in any way by persons administering or scoring the tests. Persons violating test security requirements are guilty of a first degree misdemeanor, punishable by a fine of not more than $1,000.00 or imprisonment for not more than 90 days, or both. The security requirements and penalties established by the rule and statute must be provided by the contractor to each person who has access to tests or test questions during the development, printing, administration, or scoring of the tests. A copy of the Statute and Rule is part of this appendix. Restrictions on Printing for Security Purposes The prospective contractors should indicate their ability to comply with the following conditions relative to printing of the required tests. These conditions are necessary as a means of maintaining test security. Lack of compliance with these conditions may subject the reply to rejection if the Department determines it is in its best interest to do so. 1. All test negatives and plates must be maintained under lock and key by the printing supervisor. 2. Unauthorized personnel must not be permitted access to the test negatives, plates, or copies. 3. All plates and negatives must be destroyed by the contractor upon completion of this contract. 4. The Department reserves the right to conduct on-site spot checks of the printing processes. 5. All press pull-ins, trim, and waste material must be shredded at the end of each day's press run by a person authorized to do so by the contractor. 6. Each production run must be made under close supervision of the printing supervisor. THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK Florida Standards Assessments, ITN 2014-47 114 FLORIDA TEST SECURITY STATUTE 1008.24 Test administration and security.— (1) A person may not knowingly and willfully violate test security rules adopted by the State Board of Education for mandatory tests administered by or through the State Board of Education or the Commissioner of Education to students, educators, or applicants for certification or administered by school districts pursuant to s. 1008.22, or, with respect to any such test, knowingly and willfully to: (a) Give examinees access to test questions prior to testing; (b) Copy, reproduce, or use in any manner inconsistent with test security rules all or any portion of any secure test booklet; (c) Coach examinees during testing or alter or interfere with examinees’ responses in any way; (d) Make answer keys available to examinees; (e) Fail to follow security rules for distribution and return of secure test as directed, or fail to account for all secure test materials before, during, and after testing; (f) Fail to follow test administration directions specified in the test administration manuals; or (g) Participate in, direct, aid, counsel, assist in, or encourage any of the acts prohibited in this section. (2) A person who violates this section commits a misdemeanor of the first degree, punishable as provided in s. 775.082 or s. 775.083. (3) A school district may contract with qualified contractors to administer and proctor statewide standardized assessments required under s. 1008.22 or assessments associated with Florida approved courses under s. 1003.499, as approved by the Department of Education in accordance with rules of the State Board of Education. Assessments may be administered or proctored by qualified contractors at sites that meet criteria established by rules of the State Board of Education and adopted pursuant to ss. 120.536(1) and 120.54 to implement the contracting requirements of this subsection. (4) (a) A district school superintendent, a president of a public postsecondary educational institution, or a president of a nonpublic postsecondary educational institution shall cooperate with the Commissioner of Education in any investigation concerning the administration of a test administered pursuant to state statute or rule. (b) The identity of a school or postsecondary educational institution, the personally identifiable information of any personnel of any school district or postsecondary educational institution, or any specific allegations of misconduct obtained or reported pursuant to an investigation conducted by the Department of Education of a testing impropriety are confidential and exempt from s. 119.07(1) and s. 24(a), Art. I of the State Constitution until the conclusion of the investigation or until such time as the investigation ceases to be active. For the purpose of this paragraph, an investigation shall be deemed concluded upon a finding that no impropriety has occurred, upon the conclusion of any resulting preliminary investigation pursuant to s. 1012.796, upon the completion of any resulting investigation by a law enforcement agency, or upon the referral of the matter to an employer who has the authority to take disciplinary action against an individual who is suspected of a testing impropriety. For the purpose of this paragraph, an investigation shall be considered active so long as it is ongoing and there is a reasonable, good faith anticipation that an administrative finding will be made in the foreseeable future. This paragraph is subject to the Open Government Sunset Review Act in accordance with s. 119.15 and shall stand repealed on October 2, 2014, unless reviewed and saved from repeal through reenactment by the Legislature. (5) Exceptional students with disabilities, as defined in s. 1003.01(3), shall have access to testing sites. The Department of Education and each school district shall adopt policies that are necessary to ensure such access. History.—s. 370, ch. 2002-387; s. 1, ch. 2009-143; s. 7, ch. 2013-225. Florida Standards Assessments, ITN 2014-47 115 Florida Test Security State Board of Education Rule 6A-10.042 Maintenance of Test Security. (1) Tests implemented in accordance with the requirements of Sections 1004.93, 1008.22, 1008.29, 1008.30, 1012.55, and 1012.56, Florida Statutes, shall be maintained and administered in a secure manner such that the integrity of the tests shall be preserved. (a) Test questions shall be preserved in a secure manner by individuals who are developing and validating the tests. Such individuals shall not reveal in any manner, verbally or in writing, the test questions under development. (b) Tests or individual test questions shall not be revealed, copied, or otherwise reproduced by persons who are involved in the administration, proctoring, or scoring of any test. (c) Examinees shall not be assisted in answering test questions by any means by persons administering or proctoring the administration of any test. (d) Examinees’ answers to questions shall not be interfered with in any way by persons administering, proctoring, or scoring the examinations. (e) Examinees shall not be given answer keys by any person. (f) Persons who are involved in administering or proctoring the tests or persons who teach or otherwise prepare examinees for the tests shall not participate in, direct, aid, counsel, assist in, or encourage any activity which could result in the inaccurate measurement or reporting of the examinees’ achievement. (g) Each person who has access to tests or test questions during the development, printing, administration, or scoring of the tests shall be informed of specifications for maintaining test security, the provisions in statute and rule governing test security, and a description of the penalties for breaches of test security. (h) During each test administration, school district and institutional test administration coordinators and contractors employing test administrators and proctors shall ensure that required testing procedures are being followed at all test administration sites. Officials from the Department are authorized to conduct unannounced observations of test administration procedures at any test administration site to ensure that testing procedures are being correctly followed. (2) Test materials, including all test booklets and other materials containing secure test questions, answer keys, and student responses, shall be kept secure and precisely accounted for in accordance with the procedures specified in the examination program administration manuals and other communications provided by the Department. Such procedures shall include but are not limited to the following: (a) All test materials shall be kept in secure, locked storage prior to and after administration of any test. (b) All test materials shall be precisely accounted for and written documentation kept by test administrators and proctors for each point at which test materials are distributed and returned. (c) Any discrepancies noted in the number or serial numbers of testing materials received from contractors shall be reported to the Department by designated institutional or school district personnel prior to the administration of the test. (d) In the event that test materials are determined to be missing while in the possession of an institution or school district, designated institutional or school district personnel shall investigate the cause of the discrepancy and provide the Department with a report of the investigation within thirty (30) calendar days of the initiation of the investigation. At a minimum, the report shall include the nature of the situation, the time and place of occurrence, and the names of the persons involved in or witness to the occurrence. Officials from the Department are authorized to conduct additional investigations. (e) In those cases where the responsibility for secure destruction of certain test materials is assigned by the Department to designated institutional or school district personnel, the responsible institutional or school district Florida Standards Assessments, ITN 2014-47 116 representative shall certify in writing that such destruction was accomplished in a secure manner. (f) In those cases where test materials are permitted by the Department to be maintained in an institution or school district, the test materials shall be maintained in a secure manner as specified in the instructions provided by the Department. Access to the materials shall be limited to the individuals and purposes specified by the Department. (3) In those situations where an employee of the educational institution, school district, or contractor, or an employee of the Department suspects a student of cheating on a test or suspects other violations of the provisions of this rule, a report shall be made to the department or test support contractor, as specified in the test administration procedures, within ten (10) calendar days. The report shall include a description of the incident, the names of the persons involved in or witness to the incident, and other information as appropriate. Officials from the Department are authorized to conduct additional investigations. (4) Violations of test security provisions shall be subject to penalties provided in statute and State Board Rules. Specific Authority 1001.02(1), 1008(24)(1) FS. Law Implemented 1001.02, 1008.24 FS. History–New 7-5-87, Amended 10-26-94. Florida Standards Assessments, ITN 2014-47 117 APPENDIX G TEST ADMINISTRATION MANUAL WORKFLOW The production stages for test administration manuals are defined as follows and are first defined for the Department, and then defined for the contractor. This is provided to help the respondent anticipate the level of work required on the review rounds: Table G.1 Test Administration Manual Workflow for FDOE and Contractor for Developing Florida-specific Standalone Products Table G.1 Test Administration Manual Workflow for FDOE and Contractor Round Task FDOE Manuscript (MS) Round Identify which prior administrations’ TAMs to use as a starting point for each section of the current manual. Identify high-level changes, additions, and omissions to the scripts and other sections to be included. Introduce known or potential policy updates. Verify accuracy of all FDOE policies outlined in TAM; apply content; policy updates, where necessary. Review all TAM text by comparing current TAM to most recent and most relevant prior TAM(s); verify that relevant text from each section of the prior TAM is reflected in current TAM. Identify any new naming conventions that need to be incorporated (e.g., changing “calculator” to “approved four-function calculator”); note any global edits. FDOE Round 1 Verify accuracy of links, email addresses, phone numbers, statutes, dates, and schedules. Review instructions in Test Administrator (TA), School Coordinator (SC), and Technology Coordinator (TC) sections to verify consistency (e.g., items packed in DAC box). Confirm TAM accurately indicates which ancillaries will be posted; verify file format(s). Add queries for contractor program team in green text boxes. Include placeholders (text in purple text box) in sections to be addressed at Round 2 (e.g., outstanding policy decisions). File clean-up before returning TAM to contractor: Change internal queries from orange to purple. Remove FDOE internal queries where possible. Florida Standards Assessments, ITN 2014-47 118 Round Task Confirm that all Round 1 edits have been applied. Confirm that queries to contractor from Round 1 have been addressed. Replace all placeholders with content. For assessments with CBT & PBT TAMs: Review instructions in PBT TAM against instructions in CBT TAM for consistency. FDOE Round 2 Compare CBT and/or PBT test form to TAM to o Verify accuracy of scripts (incl. electronic practice test scripts), o Verify instructions, o Verify seal configurations, o Verify page references, o Verify titles, and o Verify tool descriptions. Verify session schedule references in scripts (e.g., Session 2, Day 2); verify session length references in script. Ensure screen shots and instructions are up-to-date. Respond to contractor queries (light blue text boxes); apply checkmarks to approve suggested edits, as needed. Ensure that contractor program team checks have been made; include reminders/queries regarding checks that must be performed; schedule conference call/WebEx, if necessary, to discuss questions/concerns regarding Program Team (PT) verification. Conduct roundtable read-through, comparing to multiple prior TAMs for consistency. Ensure FDOE specialists (e.g., individuals from other offices) verify information (e.g., any updates to students to be tested, any updates to ESE accommodation information) File clean-up before returning TAM to contractor: Change internal queries from orange to purple. Remove FDOE internal queries where possible. Florida Standards Assessments, ITN 2014-47 119 Round Task Confirm that all Round 2 edits have been applied. Confirm that queries to contractor from Round 2 have been addressed. Verify contractor program team checks have been performed by appropriate program team members. FDOE Round 3 Check TOC, headers, footers, pagination, and internal page references for accuracy. Perform a cold read of the entire TAM to identify any final text edits to be made. Verify all FPOs (placeholders) have been replaced with accurate, up-to-date graphics. Verify checklists and ancillaries in appendices for accuracy and consistency (i.e., that forms are approved versions). Review third-party proofreader’s queries/suggested edits; apply checkmarks to approve suggested edits or indicate revised text using verbatim text edits in yellow text boxes. File clean-up before returning TAM to contractor. Remove FDOE internal queries. Round FDOE Final Round Task Confirm that all Round 3 edits have been applied. Confirm that queries to contractor from Round 3 have been addressed. Perform a final visual check of the document. Florida Standards Assessments, ITN 2014-47 120 Round Task Draft new TAM using TAM Information Sheet and MS round FDOE markup in previous TAM. Communicate with FDOE about edits to be applied based on MS round markup. Identify version(s) of previous TAM from which language has been pulled if not the approved final version (e.g., language in x, y, and z sections of CBT TAM pulled form EOC TAM round 3). Include queries to FDOE for sections that require confirmation, revision, or additional information. Contractor Round 1 Insert FPO (placeholder) graphics; insert forms/images from contractor P&D program team when available and note the following: o Graphics should depict information for the applicable admin, such as correct start date and logical start time. o For CBT: Screenshots should be taken using the correct user profile (e.g., CBT Coordinator). o Sample students should have age appropriate date-of-birth fields and grade levels should match the administration. Perform a clean read of TAM and insert queries where text may need to be updated based on comments at debrief, comment form feedback analysis, call center data, etc.; suggest changes; highlight any tentative revisions for FDOE to accept/reject/revise. File clean-up before posting TAM to FDOE: Remove internal queries where possible. Florida Standards Assessments, ITN 2014-47 121 Round Task Apply FDOE edits. If an FDOE edit seems incorrect or inappropriate, contact FDOE to inquire. All edits should be applied unless, during contractor’s review, FDOE is consulted and agrees to leave out edit. Consult appropriate contractor program staff (per Florida Program Team org chart) and acquire approval/sign-off on relevant script, technical, and P&D sections of TAM, including a review of naming conventions for P&D materials. Insert FPO (placeholder) graphics; insert forms/images from contractor P&D program team when available and note the following: o Graphics should depict information for the applicable admin, such as correct start date and logical start time. o For CBT: Screenshots should be taken using the correct user profile (e.g., CBT Coordinator). o Sample students should have age appropriate date-of-birth fields and grade levels should match the administration. Contractor Round 2 Perform a role play to check operational scripts against currently available test forms/books and verify accuracy of instructions, page references, screen shots, tool descriptions, etc. Verify session schedule references in scripts (e.g., Session 2, Day 2); verify session length references in script. Verify the online management system instructions by intended user role. Verify the online management system and computer-based platform screen shots. Verify emails, phone numbers, URLs, dates, and schedules. Verify which ancillaries will be posted on the online management system, and in what file format(s) (check against approved ancillaries matrix). Perform a clean read and provide queries/suggested additional edits. Suggested edits should be indicated using blue comment boxes for FDOE to approve or reject in the next review round (i.e., should not be incorporated into the body of the TAM prior to FDOE approval and should not be made using yellow text boxes, which could be mistaken for FDOE verbatim edits). Indicate content that has and has not yet been verified (e.g., add a text box indicating that school box contents have not yet been verified). File clean-up before posting TAM to FDOE: Remove contractor internal queries. Replace FDOE queries, as necessary, and ensure contractor responses are included. Florida Standards Assessments, ITN 2014-47 122 Round Task Apply FDOE edits. If an FDOE edit seems incorrect or inappropriate, contact FDOE to inquire. All edits should be applied unless, during contractor’s review, FDOE is consulted and agrees to leave out edit. Confirm all FPOs have been replaced with updated artwork. o Graphics should depict information for the applicable admin, such as correct start date and logical start time. o For CBT: Screenshots should be taken using the correct user profile (e.g., CBT Coordinator). o Sample students should have age appropriate date-of-birth fields and grade levels should match the administration. Pearson Round 3 Verify online management system instructions are accurate and up to date. Ensure all program team reviews are complete and that verification is indicated in TAM; where suggested edits arise during PT verification, these edits should be placed in blue comment boxes for FDOE approval prior to incorporation into text. Perform final full review of entire TAM for content, formatting, consistency of information and instructions across sections (e.g., TA, SC, and TC have same instructions and understanding of responsibilities). Consult appropriate contractor program staff (per Florida Program Team org chart) and acquire approval/sign-off on relevant script, technical, and P&D sections of TAM. Verify TOC, headers/footers, pagination, internal page references. File clean-up before posting TAM to FDOE: Remove contractor internal queries. Replace FDOE queries, as necessary, and ensure contractor responses are included. Round Task Confirm all edits have been applied. Confirm that approved versions of all checklists and ancillaries are being used. Contractor Final Round Perform a final, full clean read. File clean-up before posting TAM to FDOE: Remove contractor internal queries. Replace FDOE queries, as necessary, and ensure contractor responses are included. Confirm that when the final version is posted to FDOE, no marks or comments are included except where the copyeditor has made an unrequested edit or where FDOE had a query and contractor is providing a response. Florida Standards Assessments, ITN 2014-47 123 APPENDIX H TEST ACCOMMODATIONS Test Accommodations for Students with Disabilities (General Information) As specified in this appendix, accommodations may be provided to students with disabilities enrolled in public schools with current IEPs or Section 504 plans. Care must be taken to provide a student with only the accommodations permitted for that student. In addition, students with disabilities must be provided the opportunity to participate in practice activities for the tests with appropriate allowable accommodations. Determination of appropriate accommodations in assessment situations for students with disabilities shall be based on the individual needs of each student. Decisions on accommodations shall be made by the IEP or Section 504 team and recorded on the IEP or Section 504 plan. Students with disabilities should be oriented to any test situation through test-taking instruction designed to familiarize them with testing format and procedures. This orientation should take place near the time of testing. Guidelines recommended for making accommodation decisions include: 1. Accommodations should facilitate an accurate demonstration of what the student knows or can do. 2. Accommodations should not provide the student with an unfair advantage or interfere with the validity of a test; accommodations must not change the underlying skills that are being measured by the test. 3. Accommodations must be the same or nearly the same as those needed and used by the student in completing classroom instruction and assessment activities. 4. Accommodations must be necessary for enabling the student to demonstrate knowledge, ability, skill, or mastery. Students with disabilities who are not currently enrolled in public schools or receiving services through public school programs and require accommodations in order to participate in a test administration may have access to accommodations if the following information is provided: • • Evidence that the student has been found eligible as a student with a disability as defined by s. 1003.01(3) (a), F.S., or Rule 6A-19.001(6), FAC, and Documentation that the requested accommodations are regularly used for instruction. For further information, contact the Bureau of Exceptional Education and Student Services at 850-245-0475 or visit the following website: www.fldoe.org/ese/fcatasd.asp. Computer-Based Accommodations for Students with Disabilities Students with disabilities may participate in CBT administrations using computer-based accommodations as indicated on their IEPs or Section 504 plans. CBT accommodations may be included in the category of flexible presentation accommodations for CBT administrations. They may be provided to students along with other allowable test administration accommodations. CBT accommodations must be accessible through the computer-based testing platform that will be used for nonaccommodated students. Accommodations should be administrator-selectable. A. Large Print Font A fixed large print font size (minimum 18-point) that is an enlargement of the regular type is available for students who require it. Computer interface (scrollbar, buttons, cursor/pointer, etc.) is also enlarged accordingly. B. Color Contrast Options for variable font color and background color display preselected font and background color combinations on the computer screen. Specific highlighter colors are also provided for students who require them. The color combinations may include, but are not limited to, the following: • White Text/Black Background (Blue Tracking; Green Highlighter) • Black Text/Beige Background (Green Tracking; Yellow Highlighter) Florida Standards Assessments, ITN 2014-47 124 • Black Text/Light Green Background (Yellow Tracking; Orange Highlighter) Without the use of color contrast, the screen displays the following: • Black Text/White Background (Blue Tracking; Blue Highlighter) C. Zoom Feature allows students to self-select print size and magnification. Zoom levels include: 1x, 1.25x, 1.5x, 1.75x, 2x, 2.25x, 2.5x, 2.75x, 3x, 3.5x, 4x, 5x, 6x, 7x, 8x, 10x. Computer interface (scrollbar, buttons, cursor/pointer, etc.) also zooms accordingly. D. Screen Reader The screen reader provides text-to-speech audio presentation of directions, test items, and answer choices. The Department will provide the contractor with screen reader specifications at the beginning of the contract detailing the various requirements and procedures for read-aloud text and technology. E. Assistive Devices The assistive devices form allows the use of computer-based assistive devices with no other accommodated features enabled for students who require assistive devices but do not require other accommodated CBT forms. Paper-Based Accommodations for Students with Disabilities Students with disabilities may participate in CBT administrations using paper-based accommodations as indicated on their IEPs or Section 504 plans. Paper-based accommodations may be included in the category of flexible presentation accommodations for CBT administrations and may be provided to students along with other allowable test administration accommodations. In order to receive paper-based test materials for any CBT administration, the requirement for a paper-based test OR one or more of the following conditions should be documented on a student’s IEP or Section 504 plan: • The student cannot access a computer. • The student requires an accommodation that cannot be implemented in conjunction with a computer-based administration (e.g., the student requires a specialized setting or uses adaptive furniture that cannot accommodate a computer workstation). • The student requires an accommodated CBT form that is not available. • The student has a flexible scheduling/extended time accommodation that requires testing one session over more than one day. • The student is hospital/homebound. Other conditions, in addition to those listed above, may be considered by the student’s IEP or Section 504 plan team. A. Regular Print A regular print paper version of the test may be requested for a student who requires it. B. Braille A contracted or uncontracted Braille version may be requested for a student who uses Braille materials. Some test items may be altered in format for Braille versions of the test as authorized by the Department. Test items that have no application for the Braille reader will be deleted as authorized by the Department. Student performance standards that cannot be assessed in the Braille format will be deleted according to the requirements of s. 1008.22, F.S. C. Large Print A paper-based large print version of the test may be requested as a unique accommodation for eligible students who are unable to access the computer-based large print or zoom accommodation via the computer-based testing platform. Florida Standards Assessments, ITN 2014-47 125 D. One-Item-Per-Page One-item-per-page materials may be requested as a unique accommodation for paper-based tests for use by students who require one test item on a page, fewer test items on a page, increased space between test items, or true black-and-white print. E. Reading Passage Booklet Eligible students participating in computer-based Reading tests may be provided regular print or large print booklets that contain the reading passages but do not contain the test items. Although Reading Passage Booklets are included in the category of paper-based accommodations, students who require them will test on the computer. Administration Accommodations for Students with Disabilities Administration accommodations must be provided as indicated on student IEPs or Section 504 plans. Listed below are allowable administration accommodations for Florida’s statewide assessments. Not all allowable accommodations are listed. A. Flexible Presentation • Paper-based test materials (regular print, braille, large print) and accommodated computer-based test forms (large print font, color contrast, zoom, screen reader) may be requested for students who require them. • Oral presentation may be provided for all directions and for items/answer choices other than passages and items/answer choices in Reading tests. Reading passages, items, and answer choices may not be read aloud or presented orally, but must be read by the student through visual or tactile means.  Mathematics, science, and history items/answer choices may be read aloud by a test administrator or presented orally via screen reader. • Signed presentation may be provided for all directions and for items/answer choices other than passages and items/answer choices in Reading tests. Reading passages, items, and answer choices may not be signed, but must be read by the student through visual or tactile means. The test administrator may translate using the same method of sign language that the student regularly uses in the classroom, but must be careful not to use signs that might lead the student to the correct response. In such cases, fingerspelling may be used as an alternative. • For oral or signed presentation, the test administrator may sign or read aloud directions, items, and answer choices, other than passages and items/answer choices in Reading tests, to the student in the manner that is regularly used in the classroom. Examples of allowable oral/signed presentation include, but are not limited to, reading/signing items to a group of students, reading/signing every item to a student individually, reading/signing items only when a student requests. • Test directions may be repeated, clarified, or summarized. • A student may be provided with a copy of directions from the administration script that is read by the test administrator. • A student may be allowed to demonstrate that he or she understands the directions (e.g., repeating or paraphrasing). • A student may read aloud test items and answer choices to him/herself. This would require use of a device such as a WhisperPhone®, or would require that the student be tested in a separate setting so that other students are not disturbed. • Verbal encouragement (e.g., “keep working,” “make sure to answer every question”) may be used; however, it may not be used to cue a student regarding correct/incorrect responses. • Students may use magnification devices (e.g., CCTV/video magnifiers, reading loupes, handheld magnifiers). Devices must be used without accessing image-upload features (e.g., devices with a memory card must have the memory card removed during testing). • A student may use a straightedge to maintain or enhance visual attention to test items. Florida Standards Assessments, ITN 2014-47 126 • • • • • Portions of the test may be masked to direct attention to uncovered item(s). Colored transparencies/overlays may be used. Test books may be secured to a work area if no adhesives are used on scannable documents. If necessary, scannable documents may also be secured with adhesives if responses are transcribed into replacement documents. Positioning tools, such as a reading stand, may be used. A student may use a highlighter to highlight key words or phrases in directions, passages, test items, and answer choices. B. Flexible Responding • The student may use varied methods to respond to the test, including written, signed, and verbal responses. (Written responses may include the use of devices such as the Graphic Aid for Mathematics or the geoboard for students using braille.) A test administrator or proctor may transcribe student responses to the format required by the test. Recorded responses must accurately reflect the response of the student, without addition or enhancement by the test administrator or proctor. • A student may dictate responses to a proctor. • A student may use speech-to-text technology to record answers. • A student may indicate answers to test items using the mouse, keyboard, or assistive technology that uses either device. • A student may use a computer switch to indicate answers. • A student may use a computer/alternative keyboard to indicate answers. • A student may use a pointing device to indicate answers. • A student may use other communication devices to indicate answers. • A student may use mathematics grids/guides to organize mathematical computation. • A student may use special paper such as raised line, shaded line, or color-coded (would require that responses are then transcribed by school/district personnel). • Test administrators may check periodically to be sure that students are entering answer choices correctly on the computer or marking in the correct spaces in the test book. C. Flexible Scheduling • A student may be administered a test during several brief sessions, allowing frequent breaks during the test sessions. Between sessions and during breaks (e.g., restroom breaks, lunch breaks), students must be closely monitored to ensure that they do not share responses or change responses to items that were already completed. • A student may use a specific time of day for specific sessions. • A student may be provided extended time to complete a test. Extended time must be provided in accordance with the student’s IEP or Section 504 plan. Extended time is not unlimited time; it should align with the accommodation used regularly in the student’s classroom instruction and assessments. The student is not required to use all of the extended time that is allowed and may end the testing session prior to the expiration of the extended time. • Other considerations for Scheduling accommodations when administering Florida’s statewide assessments to students with disabilities include: • Each test or test session must be completed within one school day. Students who have a flexible scheduling accommodation that allows them to test over more than one day will be required to participate in testing using paper-based test materials. Florida Standards Assessments, ITN 2014-47 127 • For students using paper-based test materials to test over more than one day, the following conditions must be maintained to ensure the validity of the test administration across days:  The student may not be permitted to change his or her responses to items that were completed on a previous day.  The test administrator must closely supervise the administration of the test on an individual basis to ensure that answers from the previous day are not changed and that the student does not preview parts of the test to be answered the following day.  The test administrator must use a paper clip or binder clip to secure the answer documents and prevent the student from reviewing his or her answers from the previous day. ALL CLIPS MUST BE REMOVED AFTER TESTING. Staples or tape should NOT be used on books containing student answers. D. Flexible Setting • A student may be administered a test in a familiar place with a test proctor present, and/or a familiar person who has been appropriately trained may administer the test. Procedures for test security must also be followed and precautions taken in order to ensure that the test remains secure. • A student may be administered a test individually or in a small group setting. A small group should be of a size comparable to the normal instruction group size indicated on the student’s IEP or Section 504 plan. • If a student requires use of an accommodation that may disturb other students, the student should be tested in a separate setting. • A student may use special lighting. • A student may use adaptive or special furniture. • Special acoustics, such as FM systems, may be used to enhance sound or special rooms may be used to decrease auditory distractions. • The opportunity for movement may be increased or decreased. • Stimuli may be reduced (e.g., by limiting the number of items around the student’s desk or computer station). • White noise/sound machines or approved music may be used to reduce auditory distractions. E. Assistive Devices and Tools • Any student who requires software programs to ensure access to the computer and to approved assistive technology (e.g., switch scanning software, specialized on-screen keyboards, mouse emulators) must be able to use these devices with the computer-based testing platform. • A student may use varied assistive devices to access the computer (e.g., alternative keyboards, trackballs, joysticks, switch scanning systems, touch screens, mouth sticks, head wands, eye-gaze or head control systems). • Visual magnification and auditory amplification devices may be used. • Real coins may be used to represent coins depicted in test items. Coins may not be used as counting devices. • For students with visual impairments, an abacus and products such as the Graphic Aid for Mathematics, Math Window®, and the geoboard may be used. These devices are approved substitutes for paper and pencil computation. • Other assistive technology typically used by the student in classroom instruction may be used provided the purpose of the testing is not violated. Implementation of assistive devices must ensure that test responses are the independent work of the student. • Students who use sign language as their primary means of communication may use an English/sign or sign/English translation dictionary. The dictionary must be similar to one used in the instructional setting Florida Standards Assessments, ITN 2014-47 128 • and may not contain definitions of words. The dictionary may contain the sign picture, the word, synonyms, and an index. Manipulative materials, including, but not limited to, counters, base-10 blocks, clock faces, or geometric shapes, are not allowed to be used by any student during the test administration.  FOR BRAILLE MATERIALS ONLY: While the use of manipulatives is not an allowable accommodation, occasionally an item in a Braille test will require the use of manipulatives to give the student a tactile reference. If/when such items occur, test administrators will be provided with the instructions allowing students to use the manipulatives for those specific test items only. This modification is approved by the Department and applies only to students using Braille test materials. Unique Accommodations In accordance with Rule 6A-1.0943, FAC, school districts may request unique accommodations for individual students with disabilities. Unique accommodations usually involve alterations of existing test materials. In addition, they must be regularly used by the student for classroom instruction and must not alter the underlying content of the test. Each unique accommodation must be approved by the Commissioner of Education or a designee prior to its use. Written requests for unique accommodations must be submitted by the district assessment coordinator to the Bureau of K-12 Student Assessment at the FDOE. Accommodation requests must be reviewed by district level staff before being sent to the FDOE. In addition, the signatures of both the district exceptional student education (ESE) coordinator and the district assessment coordinator are required. Administration Accommodations for English Language Learners (ELLs) Districts are required to offer accommodations to students identified as ELLs. Allowable accommodations for ELLs are listed below. The test may be administered with any one of these modifications or a combination of accommodations that are determined to be appropriate for the particular needs of ELLs. However, all testing, with or without accommodations, must be completed during the prescribed testing dates. A. Flexible Setting • ELLs may be offered the opportunity to be tested in a separate room with the English for Speakers of Other Languages (ESOL) or heritage language teacher acting as test administrator. Parents must be informed of this option for students not of legal age and shall be given the opportunity to select the preferred method of test administration. B. Flexible Scheduling • ELLs may take a test session during several brief periods within one school day; however, each test or test session must be completed within one school day. • ELLs may be provided additional time; however, each test or test session must be completed within one school day. C. Assistance in Heritage Language • ELLs may be provided limited assistance by an ESOL or heritage language teacher using the student’s heritage language for all directions and for items/answer choices other than passages and items/answer choices in Reading tests. This should not be interpreted as permission to provide oral presentation of items/answer choices in English or in the student’s heritage language.  The ESOL or heritage language teacher may answer student questions about the general test directions in their heritage language. If the test is administered to a group of students, the teacher may answer questions about directions for the benefit of the group. Questions of clarification from individual students must be answered on an individual basis without disturbing other students.  Mathematics, Science, and History items: The teacher may answer specific questions about a word or phrase that is confusing the student because of limited English proficiency, but is prohibited from giving assistance that will help the student solve problems and answer test items. A student’s Florida Standards Assessments, ITN 2014-47 129 questions must not be answered in a way that would lead the student to infer the correct answer to an item. D. Approved Dictionary • ELLs must have access to an English-to-heritage language translation dictionary and/or heritage language-to-English translation dictionary, such as those made available to ELLs in an instructional setting. However, a dictionary providing definitions written exclusively in the heritage language or in English may not be provided. Florida Standards Assessments, ITN 2014-47 130 APPENDIX I DATA PRIVACY & SECURITY POLICY TABLE OF CONTENTS I. INTRODUCTION II. DEFINITIONS III. PRIVACY OF PERSONAL INFORMATION A. Basic Privacy Protections B. Access to the Department PII IV. INFORMATION SECURITY PROGRAM A. Information Security Risk Assessment B. Security Controls Implementation C. Security Monitoring D. Security Process Improvement E. Breach Remediation F. Organization, Responsibilities and Administration G. Personnel Security Policy Overview V. ENFORCEMENT I. INTRODUCTION Personally Identifiable Information ("PII") on students, including information on their performance on the assessments and limited demographic information, will be collected by the Contractor. Disclosure of PII to the Department and its contractors is authorized by the Family Educational Rights and Privacy Act ("FERPA") for the purposes of— (1) Conducting studies, for or on behalf of the Department and Florida’s school districts, to develop, validate, and administer predictive tests or to improve instruction (34 CFR 99.31(a)(6)(i)(A) & (C)); and (2) Assisting in the evaluation of federal- and state-supported education programs and ensuring compliance with federal legal requirements related to such programs – in particular related to state-level assessments and accountability systems (34 CFR 99.31(a)(3) & 99.35). More specifically, the data will be used by the Department or by one or more of the Department’s Contractors— • to validate, pilot and field test, and improve the assessments; • to report assessment results back to the Department and Florida school districts in a form that is useful to them; • to prepare reports on student performance for the Department and Florida school districts, and the public (PII may not be included in public reports or in reports to states or school districts that were not the source of the PII); • to analyze test results for purposes of accountability, including promotion and graduation decisions for individual students; teacher and school leader evaluations; school accountability determinations; determinations of principal and teacher professional development and support needs; and teaching, learning, and program improvement; and • to carry out studies designed to improve instruction on behalf of the Department and Florida school districts. Access to student data remains the legal responsibility of the Department and Florida school districts, in accordance with FERPA. Florida Standards Assessments, ITN 2014-47 131 This Policy describes, in general, (i) what steps the Department takes to protect PII that is accessed by or provided to the Department or the Contractor; (ii) how that information is used; (iii) with whom the Department shares that information, and (iv) the steps the Department takes to protect the security of that information. Monitoring of the implementation of this Policy is the shared responsibility of the Department and the Contractor. Each of the privacy and security provisions in this Policy is effective upon contract execution. II. DEFINITIONS Capitalized terms referenced herein but not otherwise defined shall have the meanings as set forth below: “Authorized User” means an individual Department or Contractor employee authorized by the Department to access PII maintained by the Department or the Department Contractor. "Destroy" or "Destruction" means the act of ensuring the PII cannot be reused or reconstituted in a format which could be used as originally intended and that the PII is virtually impossible to recover or is prohibitively expensive to reconstitute in its original format. “FERPA” means the Family Educational Rights and Privacy Act of 1974 (codified at 20 U.S.C. § 1232g) and its implementing regulations, as they may be amended from time to time. The regulations are issued by the U.S. Department of Education, and are available at http://www2.ed.gov/policy/gen/reg/ferpa/index.html. "Contractor" means each Department contractor, including any subcontractor(s), that may be required to maintain or handle PII in the course of providing services in support of the Department. "Personally Identifiable Information" (or "PII") means any information defined as personally identifiable information under FERPA. III. PRIVACY OF PERSONAL INFORMATION A. Basic Privacy Protections 1. Compliance with Law and Policy. All PII uploaded to or made accessible to the Department or Contractor will be handled, processed, stored, transmitted and protected in accordance with all applicable federal data privacy and security laws (including FERPA) and with this Policy. 2. Training. Employees of the Department (including temporary and contract employees) and the employees (including temporary and contract employees) of the Contractor are educated and trained on the proper uses and disclosures of PII and the importance of information privacy and security. Such training will include training for new employees and refresher training for current employees. 3. Personnel Guidelines. All the Department and Contractor employees are required to be aware of and work to protect the confidentiality, privacy, and security of PII. The Department, the Contractor, and their respective personnel shall not access PII except to comply with a legal obligation under federal or state law, regulation, subpoena or action by the Department that requires such access, or if there is legitimate need for the information to maintain data systems or to perform required services. The following list provides a general description of the internal policies with which the Department, the Contractor, and their respective personnel are required to comply: – Limit internal access to PII to the Department and Contractor personnel with proper authorization and allow use and/or disclosure internally, when necessary, solely to personnel with a legitimate need for the PII to carry out the educational purposes of the Department. – Allow access to PII by external parties only as authorized by the Department. – Require that materials containing PII in electronic form are stored solely within encrypted data repositories and PII are not available on unencrypted shared drives that are used by other users or on a local drive. PII must be encrypted in transit as well. Florida Standards Assessments, ITN 2014-47 132 – When PII is no longer needed, delete access to PII, in accordance with secure destruction procedures that the Department will provide. – Permit the Department and Contractor employees to store or download information onto a local or encrypted portable devices or storage only when necessary, and to create a written record for retention verifying that the information is encrypted and stored in password-protected files, and that devices containing the information have appropriate security settings in place (such as encryption, firewall protection, anti-virus software and malware protection). – Require that any downloaded materials consisting of PII remain in the United States. – Prohibit the unencrypted transmission of information from the Department, or any other source of PII, wirelessly or across a public network to any third party. B. Access to PII 1. The Department makes the final determination regarding access to PII. 2. Parent Complaints. The Department shall be responsible for handling any complaints from parents (or students 18 and over) regarding the disclosure or use or accuracy of PII. 3. Privacy Administrators. The Department will designate a privacy administrator who is responsible for managing access to PII maintained by the Department or the Contractor by designating authorized users, and for determining the scope of PII to which they have access. The Privacy Administrator shall be responsible for making all administrative decisions regarding access to and use of PII. IV. INFORMATION SECURITY PROGRAM The security of the PII is of critical importance to the Department. The Department's IT Security Program consists of technical, physical and administrative safeguards to protect the PII. The Department's IT Security Program is designed to identify, manage and control the risks to system and data availability, integrity, and confidentiality, and to ensure accountability for system actions. The Department's Security Program includes, and the security program of each Contractor is required to include, the following key general processes which may be more fully described in other materials as necessary: A. Information Security Risk Assessment The Department periodically conducts, and the Contractor is required to periodically conduct, an accurate and thorough assessment of the potential risks and vulnerabilities to the confidentiality, integrity, and availability of electronic, paper, or other records containing PII maintained by the Department and the Contractor; to report such risks as promptly as possible to the Department or other official within the Department designated to be responsible for data privacy and security compliance; and to implement security measures sufficient to reduce identified risks and vulnerabilities. Such measures shall be implemented based on the level of risks, capabilities, and operating requirements. These measures must include as appropriate and reasonable the following safeguards: 1. Administrative Safeguards a. Sanctions: Appropriate sanctions against the Department or contractor personnel who fail to comply with the Department security policies and procedures, with the potential for criminal referral if warranted. b. System Monitoring: Procedures to regularly review records of information systems activity, including maintaining access logs, access reports, security incident tracking reports, and periodic access audits. c. Security Oversight: Assignment of one or more appropriate senior officials within the Department and the Contractor, as applicable, to be responsible for developing, implementing, and monitoring of safeguards and security issues. d. Appropriate Access: Procedures to determine that the access of the Department and the Contractor personnel to PII is appropriate and meets a legitimate need to support their roles in Florida Standards Assessments, ITN 2014-47 133 business or educational operations. Procedures for establishing appropriate authorization and authentication mechanisms for the Department and the Contractor personnel who have access to PII. e. Employee and Contractor Supervision: Procedures for regularly monitoring and supervising the Department and Contractor personnel who have access to PII. f. Access Termination: Procedures for terminating access to PII when employment ends, or when an individual no long has a legitimate need for access. g. Recording Requests and Disclosures: Disclosures of PII to, and requests for disclosures of PII from, third parties -- other than employees of the Department or of the Contractor that provided the PII to the Department -- are recorded. 2. Physical Safeguards a. Access to PII: Procedures that grant access to PII by establishing, documenting, reviewing, and modifying a user’s right of access to a workstation, software application/transaction, or process. b. Awareness Training: On-going security awareness through training or other means that provide the Department and the Contractor personnel (including management) with updates to security procedures and policies (including guarding against, detecting, and reporting malicious software). Awareness training should also address procedures for monitoring log-in attempts and reporting discrepancies, as well as procedures for safeguarding passwords. c. Incident Response Plan: Procedures for responding to, documenting, and mitigating where practicable suspected or known incidents involving a possible breach of security and their outcomes. d. Physical Access: Procedures to limit physical access to PII and the facility or facilities in which they are housed while ensuring that properly authorized access is allowed, including physical barriers that require electronic control validation (e.g., card access systems) or validation by human security personnel. e. Physical Identification Validation: Access is physically safeguarded to prevent tampering and theft, including procedures to address control and validation of a person’s access to facilities based on his or her need for access to the PII. f. Operational Environment: Procedures that specify the proper functions to be performed, the manner in which they are to be performed, and the physical attributes of the surroundings of facilities where PII is stored. g. Media Movement: Procedures that govern the receipt and removal of hardware and electronic media that contain PII into and out of a facility. h. Encryption and Final Disposition of Information: Procedures addressing encryption of all data at rest and in transit and the final disposition of PII. Procedures must include processes for the continued encryption of PII through the time when its secure deletion/destruction has been requested by the Department and the PII is deleted/destroyed. 3. Technical Safeguards a. Data Transmissions: Technical safeguards, including encryption, to ensure PII transmitted over an electronic communications network is not accessed by unauthorized persons or groups. b. Data Integrity: Procedures that protect PII maintained by the Department or the Contractor from improper alteration or destruction. These procedures will include mechanisms to authenticate records and corroborate that they have not been altered or destroyed in an unauthorized manner. Florida Standards Assessments, ITN 2014-47 134 c. Logging off Inactive Users: Inactive electronic sessions shall be designed to terminate automatically after a specified period of time. B. Security Controls Implementation The Department and the Contractor will develop procedures addressing the acquisition and operation of technology, the specific assignment of duties and responsibilities to managers and staff, the deployment of risk-appropriate controls, and the need for management and staff to understand their responsibilities and have the knowledge, skills and motivation necessary to fulfill their duties. C. Security Monitoring In combination with periodic security risk assessments, the Department and the Contractor will use a variety of approaches and technologies to make sure that risks and incidents are appropriately detected, assessed and mitigated on an ongoing basis. The Department and the Contractor will also assess on an ongoing basis whether controls are effective and performing as intended, including intrusion monitoring and data loss prevention. D. Security Process Improvement Based on the Department's security risk assessments and ongoing security monitoring, the Department and the Contractor will gather and analyze information regarding new threats and vulnerabilities, actual data attacks on the Department and the Contractor, and new opportunities for managing security risks and incidents. The Department and the Contractor will use this information to update and improve their risk assessment strategy and control processes. E. Breach Remediation The Department and the Contractor strive to keep PII maintained by the Department and the Contractor secure and to use reasonable administrative, technical, and physical safeguards to do so. The Department and the Contractor will maintain and update incident response plans that establish procedures to follow in case a breach occurs. The Department and the Contractor will also identify individuals within their respective organizations responsible for implementing incident response plans if a breach should occur. If the Department or the Contractor determine that a breach has occurred, when there is a reasonable risk of identity theft or other harm, or where otherwise required by law, the Department or the Contractor, as applicable, will notify affected parties as promptly as possible, and will cooperate as needed to enable compliance with all breach of confidentiality laws. The Department and Contractor employee are required to report as promptly as possible to the Department leadership (or his or her designee) and persons responsible for managing their respective organization’s incident response plan any incident or threatened incident involving unauthorized access to or acquisition of PII of which they become aware. Such incidents include any breach or hacking of the Department’s electronic data system or any loss or theft of data, other electronic storage, or paper. As used herein, "Electronic Data System" means all information processing and communications hardware and software employed in the Department's or the the Contractor's business, whether owned by the Department or the Contractor, or operated by its employees or agents in performing work for the Department or the Contractor. F. Personnel Security Policy Overview The Department and the Contractor shall mitigate the risks posed by users of the Department- or the Contractor-maintained PII by: 1. Performing appropriate background checks and screening of new Department and Contractor personnel, in particular those who will have access to the Department-maintained PII; 2. Obtaining agreements from the Department and the Contractor internal users covering confidentiality, nondisclosure and authorized use of PII, including PII contained in or accessed through the Department; and Florida Standards Assessments, ITN 2014-47 135 3. Providing training to support awareness and policy compliance for new hires and annually for all Department and Contractor personnel. V. ENFORCEMENT The Department and the Contractor will consistently enforce this Policy with appropriate discipline for its own employees. The Department and the Contractor, as applicable, will determine whether violations of this Policy have occurred and, if so, will determine the disciplinary measures to be taken against any director, officer, employee, agent or representative who violates this Policy. The disciplinary measures may include counseling, oral or written reprimands, warnings, probation or suspension without pay, demotions, reductions in salary, or termination of service or employment, as well as criminal referral to law enforcement, if appropriate. Persons subject to disciplinary measures may include, in addition to the violator, others involved in the wrongdoing such as (a) persons who fail to use reasonable care to detect a violation, (b) persons who withhold material information regarding a violation, and (c) supervisors who approve or condone the violations or attempt to retaliate against employees or agents or representatives of the Department or the Contractor for reporting in good faith violations or violators. The Department also may take appropriate actions authorized under contract or by law regarding the Contractor’s failure to comply with the terms of this Policy. It is noted that if the U.S. Department of Education finds that the Contractor has violated FERPA requirements related to disclosure, the Contractor may be debarred from access to PII from the Department for at least five (5) years. Florida Standards Assessments, ITN 2014-47 136 APPENDIX J REQUIREMENTS FOR SCORING PERFORMANCE ITEMS This appendix supplements the ITN description of the procedures required for scoring constructed response items on Florida Standards Assessments in English Language Arts/Literacy (ELA/L) and Mathematics. The ITN requires that student responses for constructed response items ELA/L tests in grades 3 and 10 and Algebra 1 be scored by at least two readers (100% second scored) using imaging technology or other technological applications that are equally responsive to the test administration and reporting schedule. As a quality control measure, 20% of all other constructed response items for ELA/L and Mathematics tests will require at least two readers’ scores. The Department desires to implement processes that are reliable and valid as well as efficient in terms of time and expenditures. Therefore, respondents should, at a minimum, follow the processes described in this appendix. Enhancements to these processes are acceptable when approved by the Department. For example, respondents may propose an alternate method or formula for second reads to ensure quality control of scoring constructed response items. J.1 Overview of Handscoring of English Language Arts/Literacy and Mathematics Constructed Response Items Respondents should submit scoring rubrics to be used in conjunction with other training materials for scoring constructed-response items for all grades and assessments. Field testing of items occurs in the spring of each year with items embedded within the operational tests. Near the end of operational scoring, contractor staff will begin selecting responses from the field test sample that represent a full range of student work. These responses will be presented to educator committees in mid-spring at field test rangefinding meetings. From these meetings, complete training and qualifying sets are created to train scorers who will score a sufficient number of responses to yield reliable statistical data for each constructedresponse item. Every effort must be made to expedite scoring of the field test responses in order to have the item statistics available to construct and produce the following year’s operational tests. Based on the number of constructed response items field tested each spring, the Department will consider an additional fall field test rangefinder meeting for the purpose of preparing training materials for constructed response items that were not scored in the summer. The statistics for these items will be banked for possible use on future operational tests. After all rangefinder meetings are completed, a final scoring material review meeting between the Department and contractor staff will be conducted to finalize training, qualifying, and initial validity materials for scorer training. Depending on the measurement model for the ELA/L assessment, the Department will consider a standalone field test for ELA/L extended-response tasks. Respondents who propose a stand-alone field test should prepare field test sampling specifications detailing the sampling methodology. The Department prefers that such a field test, if proposed, take place in the December time frame of each year of administration. J.2 Conduct Field-Test and Operational Rangefinding Meetings for ELA/L and Mathematics Prior to the scoring of constructed-response items field tested during the spring administrations, the contractor will be responsible for organizing and implementing rangefinding meetings to select training papers for the tasks. The meetings take place in Tallahassee and will involve Florida educators, Department staff, and the contractor’s scoring directors and assistant directors over a period of up to three weeks each spring and each fall for field-test rangefinder meetings and each fall for operational rangefinder meetings. For all educator meetings, the Department and contractor staff will hold concurrent meetings for subjects and grades whenever possible. The contractor’s subject area lead scoring directors, subject/grade scoring directors, and assistant scoring directors will participate in meetings with up to 10 Florida educators and a Department staff member who will serve as chair. Florida Standards Assessments, ITN 2014-47 137 The contractor’s staff will maintain a record of all scores and will keep notes during the rangefinder selection process to document the rationale for assigning scores. Following the meeting, the contractor’s staff will meet with Department staff to finalize the student responses to be used as rangefinders and to identify others that may be used as training, qualifying, and validity responses. The contractor’s staff is also responsible for providing written annotations in a format approved by the Department for the training materials. Requirements for selecting validity sets are identified in the following section. Contractor staff is also responsible for developing as part of the scoring guides any additional scoring notes or criteria required to conduct accurate scoring of the constructed-response tasks. The Department will approve all selections, annotations, or other materials to be used in training. See Section 7.5.4.5 for associated tasks. J.3 Conduct Scoring Material Review Meetings Meetings to review proposed scoring materials for operational tests will be conducted in Tallahassee at least three months prior to the beginning of handscoring training for the spring test administrations. The contractor’s subject area lead scoring director and the subject/grade scoring directors will participate in meetings with Department staff to review scoring criteria and rangefinder papers that were used to score the tasks during field test scoring, or during the previous operational use. The purpose of each review meeting is to review the criteria, supplement the initial set of rangefinder papers with additional papers, if necessary, and to ensure that the contractor’s scoring directors and Department staff share the same, detailed understanding of the scoring criteria for the operational constructed-response tasks. The scoring standards established by the initial field-test rangefinder selection will be maintained during the subsequent review. See Section 7.5.4.5 for associated tasks. J.4 Produce Scorer Training Materials Department staff will work closely with the contractor to prepare scoring materials. Frequent communication between Department and contractor staff can be expected following rangefinder selection and scoring material review meetings to initiate and complete the development of scoring materials. All scoring material will be submitted to the Department for review and approval according to an agreed-upon schedule. Scoring materials must be approved at least four weeks prior to the beginning of training and scoring. The contractor will be responsible for developing a detailed schedule to be included in the handscoring specifications identifying steps and schedules in the development of scoring materials. The contractor is responsible for producing the following scoring materials for each operational and field-test constructedresponse task. • • • • • • Scoring guides Training sets Qualifying sets Validity sets Daily calibration sets Targeted calibration The contractor will develop a system for cataloging and storing all scoring materials developed during the course of the project. Scoring Guides - Scoring guides will be produced for each reader to use during training and scoring. The scoring guides will contain the rubrics, scoring criteria, constructed-response tasks, passages (if applicable), glossary of key terms, and other scoring guidelines designated by the Department and the contractor. The contractor will be responsible for writing annotations for the scoring guides, and the Department will give final approval to all annotations. Training Sets – Contractor must develop a sufficient number of training (practice) sets for ELA/L and Mathematics constructed-response items for scorer training to ensure valid and reliable scores for the duration of the project. The Department will work with contractor staff to determine the number and types of Florida Standards Assessments, ITN 2014-47 138 training sets to be used. The contractor will be responsible for writing the annotations for the training papers for all content areas, and the Department will give final approval to the annotations. The respondent should propose the process for administering the training sets. Validity Papers - Initial validity responses are selected from student responses to field-test administrations and do not appear in any other scoring materials. These responses allow the Department and contractor to monitor scoring accuracy and to maintain a consistent focus on the established rubrics and scoring guidelines. A sufficient number of initial validity responses must be approved in time to be loaded into the electronic scoring system before the scoring session begins. Validity responses are administered to readers online as part of the electronic scoring system and must be indistinguishable from live responses. The validity insertion rate will be agreed upon during the negotiation phase; however, for cost purposes, respondents should assume a validity insertion rate of 1 in 10 for extended ELA/L items and 1 in 25 for all other constructed response items. Once handscoring begins, scoring directors must refresh the validity pool during live scoring by escalating proposed validity responses to the Department for approval. All proposed validity papers for constructed-response items must have two agreement scores assigned. These scores may be generated by two scorers or by one scorer and a back-reading score. The frequency distribution of approved validity papers in the pool throughout the project should mirror the operational frequency distribution to the greatest extent possible. Responses that are approved by the Department will be moved into the electronic scoring system for random distribution to scorers. To avoid validity responses being repeatedly seen by the same scorers, adding and deleting validity papers within the electronic system is a continuous process throughout the project. The contractor will describe the estimated number of readers to be employed and plans for the administration of validity responses. Daily Calibration Sets – In addition to review of selected papers from the Scoring Guide at the start of each shift, daily calibration sets will be sent via the electronic scoring system at least once a day – usually after lunch or break. These 2-3 paper sets will be proposed by scoring directors and approved by the Department. The purpose of daily calibration is to highlight scoring issues or remind scorers of what to look for at a certain score point. Following the scoring of the calibration sets, the scoring directors will conduct a short discussion to further explain the scoring standards therein. Targeted Calibration Sets – Targeted calibration sets consist of 2-3 responses that target an individual’s scoring trend and may be used as individual training/remediation for scorers whose quality metrics are below the standard for any reason. At the completion of scoring, the contractor will securely ship to the Department an electronic file of all scoring materials prepared for and utilized during scoring. J.5 Scoring Directors for Handscoring The contractor will assign its most qualified scoring staff to be scoring directors for Florida’s Standards Assessments. All scoring directors must have worked in scoring director roles on a regular, continuing basis. The project and site scoring directors must be highly qualified to serve in these positions. They must have an appropriate educational background and extensive experience in directing state-level performance scoring projects as members of the contractor’s regular scoring staff. The contractor will submit resumes of all handscoring staff to the Department for approval before assigning them to the project. The contractor will appoint a full-time project director to serve as overall director for Florida Standards Assessment handscoring activities. The project director must be available on a daily basis to discuss issues with the site scoring directors and the Department either in person, by e- mail, by phone or fax throughout the training and scoring sessions. The contractor must provide at least one full-time lead scoring director for each content area with expertise in their respective subject. This contractor staff will be available as liaison for Department content specialists for all handscoring activities. Florida Standards Assessments, ITN 2014-47 139 The contractor will provide a site scoring director at each site who is dedicated solely to scoring of Florida’s tests during the scoring period. Site scoring directors will be on site throughout the training and scoring sessions and will personally assist the scoring directors during the training of team leaders and readers and throughout the scoring sessions. The project director must be a senior scoring manager for the contractor. Each site scoring director must have served as a site scoring director for at least two projects in the past three years involving projects similar in size and scope to Florida’s. The scoring directors who will direct the scoring for each subject/grade combination must be highly qualified to serve in this position. They must have an appropriate educational background and must have had experience serving as scoring directors at this level for previous scoring projects conducted by the contractor. The scoring directors must participate in the field test and operational rangefinder and scoring material review meetings. At least two scoring directors will be required for each grade and subject in order to be available in the event of the scheduling of two shifts or two scoring rooms. For the ELA/L extended constructed response items, it may be necessary to assign three scoring directors for this project. At least one assistant scoring director will be appointed to assist each scoring director in all aspects of directing scoring. The assistant scoring directors must also have an appropriate educational background and must have had experience serving as scoring directors or extensive experience serving as team leaders for previous scoring projects conducted by the respondent. The assistant scoring directors must also participate in all rangefinder meetings. Lead scoring directors, scoring directors, and assistant scoring directors must have their résumés approved by the Department prior to assignment to the project. As stated above, Florida’s handscoring involves selection of rangefinders for field-test tasks at the same time that responses to the operational test are being scored. The scoring director and the assistant scoring director for each subject/grade will work together as a team to ensure that field test rangefinder selection and ongoing direction of operational scoring are both conducted at the highest levels of quality. It may be required that contractor staff assigned to the project be present at field test rangefinding meeting while other contractor staff remains at the scoring site to oversee completion of operational scoring. For each assessment, the project and site scoring directors will ensure that all additional training required for scoring is conducted prior to team leader training. Respondents should describe how they intend for this training to occur. J.6 Team Scoring Leaders The contractor will hire one team leader for every 10 to 12 readers. The team leaders may be especially recruited by the contractor or identified through the reader recruitment process. Team leaders must go through the same screening process as readers. The Department requires that team leaders have previous experience as readers and as team leaders if at all possible. At a minimum, team leaders must be experienced readers. Extensive training sessions approximately four days in length will be conducted for team leaders before readers are trained. The scoring directors and assistant scoring directors for each set of constructedresponse tasks will conduct the training. The team leader training sessions will include the successful completion of a minimum of three training sets and two qualifying rounds. For ELA/L extended constructedresponse writing tasks, Florida currently requires five training sets and three qualifying rounds. The initial training will include the training and qualifying procedures that will be used for the readers. In addition, there will be training on managing interaction with scorers, interpreting scoring reports, completing forms, and following all procedures required by the contractor and the Department. Florida Standards Assessments, ITN 2014-47 140 J.7 Recruit and Hire Readers Respondents should include an analysis of the number of people that must be recruited, hired, and subsequently qualified as readers to complete the scoring within the time required to return reports to districts by the dates designated in the project schedule. The analysis must include the number and percent of people who will be expected to qualify, expected reader-attrition rate during each scoring session, and production required by reader and, if necessary, by shift at each site to complete the work. The respondent should describe the number of team leaders needed for each subject/grade. This detailed analysis should be completed in the reply for the Florida Standards Assessments in English Language Arts/Literacy and Mathematics. Respondents should also indicate expected changes in reader numbers across successive administrations. All readers are required to have earned a bachelor's degree in mathematics or the area of English Language Arts, or some related field. Readers of mathematics constructed-response tasks must have either been certified to teach the level of mathematics being assessed, have a bachelor’s degree in mathematics education or mathematics, or must have completed a degree program with a sufficiently strong emphasis in mathematics to have the mathematics knowledge needed to effectively score responses to mathematics items. Potential readers in Florida may include retired Florida teachers or substitute teachers in the state of Florida provided that they will not teach in Florida’s public schools (or work as a substitute teacher therein) for the remainder of the current school year. The Respondent is responsible for conveying this information and obtaining signed verification that potential readers agree to adhere to this stipulation. The contractor must work with the Department to implement a screening process for hiring readers to satisfy the Department’s requirements. Potential readers must document that they meet the minimal education requirements during the screening process. Respondents should describe the screening process to select scorers. Potential readers with related degrees are those with degrees that involve course work and experience related to the content areas being scored. Potential candidates for training and qualifying to score mathematics shall have degrees in fields such as engineering, computer science, or liberal arts with an emphasis in mathematics. Likewise, for scoring English Language Arts/Literacy, candidates shall have degrees in fields such as English, English education, elementary education, reading education, history, psychology, or journalism. Potential readers will provide information about degree, certification, and course work taken. Minimum college coursework that represents a sufficiently strong emphasis in mathematics needed to score in this area is college-level credit in at least three courses to score at the high school level, two courses for the middle-school level, and one course to score at the elementary level. The contractor will provide for an independent, third-party verification that the potential reader has earned the identified degree. It has been the Department’s experience that obtaining third-party verifications is more expensive and timeconsuming for potential readers holding degrees conferred by foreign universities and degrees conferred by U.S. colleges and universities fifteen or more years ago. The contractor must take this into consideration when hiring potential readers. Potential readers whose degrees cannot be verified by a third party before the start of live scoring must be dismissed. No scores must be entered or maintained in the scoring system from readers with unverified qualifications. Potential readers also include those with a bachelor’s degree or higher who have successfully completed a scoring project in the content area and at the elementary, middle, or high-school level being scored, qualifying and performing at levels required by Florida’s program. Potential readers will ultimately demonstrate their qualification to score through successful completion of training and meeting the qualifying standard on scoring qualification rounds, as indicated in the Handscoring Specifications document. The contractor will document qualifications of readers assigned to be trained in an electronic file (the “Qualification File”) specifying degrees earned; relevant teaching, educational, or work experience; and previous scoring experience relevant to scoring Florida’s assessments. The Qualification Files will not Florida Standards Assessments, ITN 2014-47 141 include personally identifiable information such as name, address, and telephone number. Documentation will be provided to Department staff, as needed, as it is accumulated throughout the hiring and training period. After scoring has begun, the contractor will only maintain Qualification Files for readers who successfully qualify and score student responses to Florida’s assessments. The Department will monitor adherence to the screening process, review reader files as needed, approve acceptable related degrees, and provide general guidance on the screening process. Typically, each scoring site will be monitored by at least one representative from the Department for two to three weeks beginning with training. Prior to and during this on-site monitoring, the Department will provide oversight and monitoring of both potential Team Leader and reader qualifications. This monitoring will include a review for acceptability of potential readers for assignment to the handscoring program, In the month prior to handscoring training, weekly conference calls between the contractor and the Department will provide discussions surrounding site readiness, hiring/staffing, system readiness, and other pertinent topics related to handscoring. After recruitment of potential readers is complete, the Department staff will review a sample of potential readers’ qualifications. Staff review of qualifications will be conducted utilizing a random sample of the Qualification Files. Staff will select additional potential scorer IDs for review as deemed necessary, e.g., for candidates who are having little success in scoring accurately during training. All qualifications for team leaders and a minimum of one in five candidates’ documentation will be reviewed (e.g., for an initial hiring of 150 candidates for training, approximately 30 sets of documentation will be requested). Once scoring training begins, daily conference calls between the contractor and the Department will be conducted for project management purposes. As part of the screening process, all potential readers must sign a statement indicating that they understand the following conditions: • Applicants who do not successfully complete the training and qualifying requirements, they will not be hired as readers. • Hired readers may be dismissed if, after being trained to score, their scoring performance does not meet the requirements of the Department or the respondent acting on behalf of the Department. • Readers who are absent from scoring for two or more consecutive scoring days must repeat the training and qualifying procedures described above. • Applicants must agree that they will not teach or work as a substitute teacher in Florida’s public schools for the remainder of the current school year. Applicants hired for training must also sign an agreement with the Department that they will maintain the security of Florida’s test materials in addition to security agreements required by the contractor. J.8 Training and Qualifying of Readers The contractor will conduct separate training sessions for each constructed-response task or task set. Training is provided separately for each grade level. The scoring director will conduct training with the assistance of team leaders, under the direction of site scoring directors. The purpose of the training is to ensure that each person who scores Florida’s student responses has met the Department's standards for scoring. The training process is essential for ensuring that scores assigned to performance task responses provide valid and reliable information. During training, the trainees will be provided with a scoring guide, highlighters and pens or pencils to make notations on training materials. At the conclusion of training, qualified readers will be taught how to use the alert system to identify students whose responses indicate the need for intervention. The contractor is responsible for developing training procedures in consultation with the Department. The Department will have final approval on all training techniques. To qualify as a reader, candidates must meet the established standards for the qualification sets approved by the Department. Assignment of scorers to items will be done in consultation with Department staff. Florida Standards Assessments, ITN 2014-47 142 J.9 Handscoring Reports The contractor will collect reader performance statistics and provide electronic access to daily and cumulative reader handscoring reports to team leaders and above, including Department staff. The reports must include both real-time and partial- to full-day reports. Real-time reports must include reader inter-rater reliability and validity reports. The same reports are required for both operational and field-test scoring. The Department will determine, in consultation with the contractor, which of these reports will be available to the Department in real-time. The reports listed below are described generically. The contractor and the Department will work together initially to determine the format of this information that should be used for reporting to the Department. An annual review of the format and function of the reports will be conducted and some modifications may be required prior to the next test administration requiring handscoring based upon that review. The Department will have final approval on the design and user access level of each report. At a minimum, reports containing the following information will be produced and will include both numbers and percentages, as appropriate. 1. Cumulative Training Report by Reader. This report identifies, by performance task or performance task sets, by reader, by team, and by site (subject/grade combination), performance on training sets. 2. Cumulative Training Report by Document. This report identifies and summarizes, by performance task or performance task set, by team, and by site, performance on documents. 3. Cumulative Qualifying Report by Reader. This report identifies, by performance task or performance task set, by reader, by team, and by site, performance on qualifying sets. 4. Cumulative Qualifying Report by Document. This report summarizes, by performance task or performance task set, by team, and by site, performance on documents. 5. Cumulative Reader Performance Report. This report includes the reader’s scores on each training set, qualifying set, and validity set; daily inter-rater reliability statistics; and daily score point distribution. This report will be updated daily. The report will be accessed on request of the scoring director or the team leader. 6. Daily and Cumulative Score Point Distribution. This report summarizes, by performance task or performance task set, by team, by site, and by rubric (0-2, 0-4, and 0-6), daily and cumulative distributions of score-point assignments. 7. Daily and Cumulative Validity. This report summarizes, by performance task or performance task set, by team, and by site, daily and cumulative performance on validity sets. 8. Daily and Cumulative Inter-rater Reliability. This report summarizes, by performance task or performance task set, by site, by team, and by rubric, daily and cumulative levels of inter-rater agreement statistics. 9. Score Point Distribution for Standard Curriculum Students. This report summarizes, by team and by site, the cumulative distribution and mean of score-point assignments for standard curriculum students, for writing performance tasks only. 10. Daily and Cumulative Score Point Distribution and Inter-rater Reliability by Reader. This report identifies and summarizes, by performance task or performance task set, by site, by team, by rubric, by reader, daily and cumulative score-point assignment distributions and levels of inter- rater agreement statistics (including perfect, adjacent, and disparate agreement; high/low scoring summaries; and required resolutions). 11. Daily and Cumulative Validity by Reader. This report summarizes, by performance task or performance task set, by site, by team, by rubric, by reader, statistics indicating daily and cumulative Florida Standards Assessments, ITN 2014-47 143 levels of agreement with validity set scores (including perfect, adjacent, and disparate agreement and high/low scoring summaries). 12. Daily and Cumulative Validity by Validity Set. This report summarizes, by site, by team, by validity set, by reader, statistics indicating daily and cumulative levels of agreement with validity set scores (including a listing of validity set scores with readers assigned scores; perfect, adjacent, and disparate agreement; and high/low scoring summaries). 13. Daily and Cumulative Read-Behind Reports. This report summarizes, for image scoring applications, by performance task or performance task set, by team, and by site, the result of online read-behinds by team leaders or scoring directors. 14. Daily and Cumulative Timing Report. This report summarizes, by site, by team, by performance task or performance task set, the daily and cumulative number of readers engaged in scoring and statistics on actual time spent scoring performance tasks or performance task sets (including mean and range). 15. Daily and Cumulative Completion Report. This report, by site and by team, summarizes the daily and cumulative number of student responses read and the proportion of student responses remaining to be read. This report also compares on a daily and cumulative basis the amount of readings completed in comparison to projected completion targets. By the time of the first handscoring planning meeting with the Department, the contractor will provide an annotated list and samples of all reports, including reports used internally by the contractor to monitor the quality and pace of the scoring session. The Department may request that some internal reports be used for Florida and available to Department staff. At the completion of operational and field-test scoring, the contractor will provide the Department with final summaries of all cumulative handscoring reports. The summary reports are to be made available as electronic files and will be posted to a secure transfer file system maintained by the contractor and shared with the Department to access, manage, and transfer files as needed throughout the project. The contractor will produce a technical report that summarizes the score reports and provides details related to the reliability and validity of the field-test and operational handscoring. These materials will be reviewed at an annual handscoring debrief. J.10 Scoring Student Responses The rubrics used for all constructed-response tasks must be approved by the Department and will be used in conjunction with approved scoring guides and additional training materials throughout the handscoring of student responses. Constructed-response items for ELA/L tests in grades 3 and 10 and Algebra 1 will be scored by at least two readers (100% second scored) using imaging technology or other technological applications that are equally responsive to the test administration and reporting schedule. As a quality control measure, 20% of all other constructed response items for ELA/L and Mathematics tests will require at least two readers’ scores. Responses receiving first and second reads of an individual student’s response to an item will be randomly distributed across all readers designated to score that item. Assignment of responses across schools must be randomized to the extent that an individual reader scores responses from several schools within the same time period and does not score responses from the same school in succession. The Department will provide the contractor with a specific set of scoring rules for resolving disagreement between first and second readings with third and, infrequently, fourth readings and for calculating final scores. The following should be noted: • • • Third, independent readings are conducted by team leaders. Complex resolutions and resolutions involving fourth readings and unscorable decisions are resolved by scoring directors. All discrepancies for short-response tasks shall be resolved by a third reader Florida Standards Assessments, ITN 2014-47 144 Attachment 1 ITN 2014-47 Florida Standards Assessments RESPONDENT:_________________________________________________ We propose to provide the services being solicited for ITN 2014-47 Florida Standards Assessments. All work shall be performed in accordance with the specifications of this ITN, which has been reviewed and understood. The below prices are all inclusive. Payment for services will be made based on the deliverables and dates specified. There shall be no additional costs charged for work performed under this ITN, unless changes are made during the negotiation phase. Respondents may submit bids on any individual assessment components (A, B, C or D), any group of assessment components, or to provide all assessment components (A, B, C and D).  The prices on this sheet will be used for evaluation purposes.  All accommodations and deliverable costs should be included in each component price.  In addition, Respondents should complete the pricing per deliverable in SECTION 7.8, for each assessment component, which will be used for negotiation purposes and any resulting contract. PRICING ASSESSMENT COMPONENTS YEAR 1 YEAR 2 YEAR 3 TOTAL ORIGINAL CONTRACT TERM (YEARS 1 - 3) GRAND TOTAL RENEWAL YEAR RENEWAL YEAR RENEWAL YEAR (ORIGINAL TERM + ALL 1 2 3 RENEWALS)* A.  Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 3-8 per schedule in Appendix A. B. Computer-based and paper-based assessments in Mathematics in Grades 3-8 per schedule in Appendix A. C. Computer-based and paper-based assessments in English Language Arts/Literacy in Grades 9-11 per schedule in Appendix A. D. Computer-based and paper-based End-of-Course (EOC) assessments in Algebra 1, Geometry, and Algebra 2 per schedule in Appendix A. E. All assessments (A-D) per schedule in Appendix A. * Evaluation points will be calculated on these costs. Vendor Name: Street Address: City, State, Zip Code: Telephone #: Fax: Authorized Agent (typed): Authorized Agent (Signature): 145 ATTACHMENT 2 DRUG-FREE WORKPLACE (will be considered in case of identical tie replies) Preference shall be given to businesses with drug-free workplace programs. Whenever two or more bids which are equal with respect to price, quality, and service are received by the state or by any political subdivision for the procurement of commodities or contractual services, a bid received from a business that certifies that it has implemented a drug-free workplace program shall be given preference in the award process. Established procedures for processing tie bids will be followed if none of the tied vendors have a drug-free workplace program. In order to have a drug-free workplace program, a business shall: 1) Publish a statement notifying employees that the unlawful manufacture, distribution, dispensing, possession, or use of a controlled substance is prohibited in the workplace and specifying the actions that will be taken against employees for violations of such prohibition. 2) Inform employees about the dangers of drug abuse in the workplace, the business's policy of maintaining a drugfree workplace, any available drug counseling, rehabilitation, and employee assistance programs, and the penalties that may be imposed upon employees for drug abuse violations. 3) Give each employee engaged in providing the commodities or contractual services that are under bid a copy of the statement specified in subsection (1). 4) In the statement specified in subsection (1), notify the employees that, as a condition of working on the commodities or contractual services that are under bid, the employee will abide by the terms of the statement and will notify the employer of any conviction of, or plea of guilty or nolo contendere to, any violation of Chapter 893 or of any controlled substance law of the United States or any state, for a violation occurring in the workplace no later than five (5) days after such conviction. 5) Impose a sanction on, or require the satisfactory participation in a drug abuse assistance or rehabilitation program if such is available in the employee's community, by any employee who is so convicted. 6) Make a good faith effort to continue to maintain a drug-free workplace through implementation of this section. As the person authorized to sign the statement, I certify that this firm complies fully with the above requirements. Vendor's Signature Florida Standards Assessments, ITN 2014-47 146 ATTACHMENT 3 DISCLOSURE STATEMENT PARTNERSHIP OR INDIVIDUAL I hereby certify that I, if an individual, or each of us, if a partnership, doing business as of (Name of Individual or Partnership) , (am)(is) not now involved in nor have I ever engaged in any private business venture or enterprise, directly or indirectly, with the Commissioner of Education, the Deputy Commissioner of Education, any Associate Commissioner of Education, Division Director, or Bureau Chief within the Florida Department of Education. I further certify that neither I, nor any partner, if a partnership, nor anyone acting in my or our behalf has requested that any of the above designated persons or any other employee of the Department of Education exert any influence to secure the appointment of under this proposed agreement. (Name of Individual or Partnership) (1) Signature Signature (1) If partnership, each partner must sign and execute. Signature COMPANY OR CORPORATION I hereby certify that neither I nor any owner, officer, director, or shareholder of (Name of State of Inc.) (Name of Corporation/Company) ,a (1) corporation, licensed to do business in Florida, is presently involved in or has been engaged in any private business venture or enterprise, directly, or indirectly, with the Commissioner of Education, the Deputy Commissioner of Education, any Associate Commissioner of Education, Division Director, or Bureau Chief within the Florida Department of Education. I further certify that neither I nor any owner, officer, director, or shareholder of this corporation or anyone acting on behalf of this corporation or any of its owners, officers, directors, or shareholders has requested that any of the above designated persons or any other employee of the Department of Education exert any influence to secure the appointment of under this proposed agreement. (Company / Corporation) (2) Signature Title (1) If company is not incorporated, insert "not incorporated" in this space. (2) If incorporated, this statement is to be executed by same person who will execute Contract, if awarded. Florida Standards Assessments, ITN 2014-47 147 ATTACHMENT 4 REFERENCES RESPONDENT NAME: _____________________________________ PROVIDE THE FOLLOWING REFERENCE INFORMATION FOR A MINIMUM OF THREE (3) BUSINESSES WHERE SERVICES OF SIMILAR SIZE AND SCOPE HAVE BEEN COMPLETED. BUSINESS NAME: ADDRESS: CONTACT PERSON: PHONE NUMBER: FAX NUMBER: E-MAIL ADDRESS: DATE AND DESCRIPTION OF SERVICES: BUSINESS NAME: ADDRESS: CONTACT PERSON: PHONE NUMBER: FAX NUMBER: E-MAIL ADDRESS: DATE AND DESCRIPTION OF SERVICES: BUSINESS NAME: ADDRESS: CONTACT PERSON: PHONE NUMBER: FAX NUMBER: E-MAIL ADDRESS: DATE AND DESCRIPTION OF SERVICES: Florida Standards Assessments, ITN 2014-47 148 ATTACHMENT 5 MINORITY SUB CONTRACTORS UTILIZATION SUMMARY STATE OF FLORIDA DEPARTMENT OF EDUCATION The Department’s Supplier Diversity initiative strives to ensure the promise of Florida’s future is shared by all of its residents, regardless of race, ethnicity, disability, neighborhood or background. To that end, the Department is dedicated to support, track and increase its small, minority-, women-, and service-disabled veteran business enterprise spending with Prime Contractors and subcontractors. This form was developed to assist in these efforts. The Prime Contractor shall report all small, minority-, women-, and service-disabled veteran business enterprise Subcontractors, identifying the Name, Address, Type of Certification and Dollar Amount on the form below. The Prime Contractor shall submit this form with each invoice submitted for payment, whether or not funds have been spent with a small, minority-, women-, and service-disabled veteran business enterprise subcontractor for the period covered by the invoice. The Office of Supplier Diversity, Florida Department of Management Services will assist in furnishing names of qualified minorities. The Office of Supplier Diversity can be reached at (850) 487-0915; the Internet Web address is http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd . PRIME CONTRACTOR: CONTRACT NO.: CONTRACT TITLE: MBE CONTRACTORS Full Name, Address, Telephone Number State Certified NonCertified NonProfit Dollar Amount Total Amount $ Certified True and Correct by: Submit Report to: _________________________________________ Prime Contractor _________________________________________ Title _________________________________________ Date Mrs. Janice Brown Bureau of Contracts, Grants & Procurement Management Services 325 West Gaines Street 332 Turlington Building Tallahassee, Florida 32399-0400 For additional information, you may call Mrs. Brown at (850) 245-0819, or e-mail janice.brown@fldoe.org Florida Standards Assessments, ITN 2014-47 149 ATTACHMENT 6 VENDOR CERTIFICATION REGARDING SCRUTINIZED COMPANIES LISTS Respondent Vendor Name: _____________________________________________________ Vendor FEIN: ___________________ Vendor’s Authorized Representative Name and Title: ___________________________________ Address: _____________________________________________________________________ City: _____________________ State: _____________________________ Zip: ______________ Phone Number: ____________________________________ Email Address: _ Section 287.135, Florida Statutes, prohibits agencies from contracting with companies, for goods or services over $1,000,000, that are on either the Scrutinized Companies with Activities in Sudan List or the Scrutinized Companies with Activities in the Iran Petroleum Energy Sector List. Both lists are created pursuant to section 215.473, Florida Statutes. As the person authorized to sign on behalf of Respondent, I hereby certify that the company identified above in the section entitled “Respondent Vendor Name” is not listed on either the Scrutinized Companies with Activities in Sudan List or the Scrutinized Companies with Activities in the Iran Petroleum Energy Sector List. I understand that pursuant to section 287.135, Florida Statutes, the submission of a false certification may subject company to civil penalties, attorney’s fees, and/or costs. CERTIFIED BY: __________________________________________________________, Who is authorized to sign on behalf of the above referenced company. Authorized Signature Print Name and Title: _________________________________________ Florida Standards Assessments, ITN 2014-47 150 ATTACHMENT 7 CONTRACT STANDARD TERMS AND CONDITIONS STATE OF FLORIDA DEPARTMENT OF EDUCATION I. II. III. IV. Pursuant to S. 287.058(1), Florida Statutes (“F.S.”): A. Bills for fees or other compensation for services or expenses shall be submitted in detail sufficient for a proper preaudit and postaudit thereof. B. Travel expenses will be reimbursed only if expressly authorized by the terms of the Contract. Bills for any travel expenses shall be submitted in accordance with s. 112.061, F.S. C. The Department may unilaterally cancel this Contract if the Contractor refuses to allow access by members of the public to all documents, papers, letters and materials made or received in conjunction with the Contract that are subject to Chapter 119, F.S., and are not exempt from public inspection by s 119.071, F.S., or by other provisions of general or special law. D. The Deliverables specified in the Contract must be received and accepted in writing by the Department’s Contract Manager before Contractor is entitled to payment. E. To complete this Contract, all services must be performed and/or goods received on or before the date(s) specified in the Contract. F. If this Contract is expressly renewable, it may be renewed for a period that may not exceed three years or the term of the original contract, whichever is longer. The renewal price for the contracted service is set forth in the bid, proposal, reply. Cost for renewal shall not be changed. Renewals shall be contingent on satisfactory performance evaluations by the Department and subject to the availability of funds. Exceptional purchase contracts pursuant to s. 287.057(3)(a) and (c), F.S., may not be renewed. The Contractor shall prepare an invoice for the amount due and mail it to the Department of Education Comptroller after having delivered the products and services required under this Contract to the Contract Manager. The invoice shall set forth details sufficient for a proper pre-audit and post-audit including, where applicable, the products and services delivered and completion dates. Upon receipt of the invoice, the Department of Education Comptroller will request confirmation from the Contract Manager that the delivered products and services are satisfactory and payment is due. If for any reason they are not satisfactory, payment will be withheld until the unsatisfactory condition or conditions are corrected. Upon receipt of the Contract Manager’s approval, the Department of Education Comptroller shall process each invoice in accordance with the provisions of s. 215.422, F.S. A. Contractor agrees to submit invoice within thirty (30) days of the Department’s acceptance of deliverables. It is understood that should Contractor fail to submit invoice within thirty (30) days following the Department’s acceptance of the deliverables, the Department shall not be responsible for payment thereof under this contract or quantum meruit. Section 215.422, F.S., provides that agencies have five (5) working days to inspect and approve goods and services, unless bid specifications or the Contract specifies otherwise. With the exception of payments to health care providers for hospital, medical, or other health care services, if payment is not available within forty (40) days, measured from the latter of the date the invoice is received or the goods or services are received, inspected and approved, a separate interest penalty set by the Comptroller pursuant to s. 55.03, F.S., will be due and payable in addition to the invoice amount. To obtain the applicable interest rate, please contact the Department’s Fiscal s. at 850/245-0401 or Purchasing Office at 850/245-0735. Payments to health care providers for hospitals, medical, or other health care services, shall be made not more than thirty-five (35) days from the date of eligibility for payment is determined, and the daily interest rate is .02740 percent. Invoices returned to a vendor due to preparation errors will result in a payment delay. Invoice payment requirements do not start until a properly completed invoice is provided to the agency. A Vendor Ombudsman, whose duties include acting as an advocate for vendors who may be experiencing problems in obtaining timely payment(s) from a State Agency, may be contacted at 866/352-3776 or by calling the Chief Financial Officer’s Hotline, 800/3422762. As used in this Contract, the term “Deliverable” refers to tangible “commodities”, as defined in s. 287.012(5), F.S., which the Contractor provides pursuant to the Contract and to reports or other tangible or documentary evidence which demonstrate that the Contractor has performed the services required by the Contract. The following provisions govern Deliverables, as applicable: A. Each Deliverable must be physically delivered to the Department’s Contract Manager, or to a person designated by the Contact Manager. If delivery is made to a designee, the Contractor shall give written notice to the Contract Manager of the delivery. A Deliverable is not received until the Contract Manager has physical control of deliveries or has written notice that the designee has physical control. B. In each case in which the approval of a Deliverable is dependent upon tests being conducted by the Department or Contractor, independently or jointly, the Department’s inspection and approval of the Deliverable shall not be subject to the five (5) day provision in s. 215.422, F.S., but shall be governed by the terms and conditions of the acceptance testing plan as stated in Attachment A, until approved in accordance with the plan. C. In each case of a Deliverable of information technology, as defined at s. 287.012(14), F.S., unless specified otherwise in Attachment A, the acceptance testing plan is deemed to include as a minimum the reliable performance of the information technology in accordance with its design specifications in: 1. a test environment that simulates the production environment as much as is reasonably possible; and 2. the production environment for which it is intended for a period of time sufficient for the information technology to have experienced the major foreseeable exigencies of the production functions. D. The Department’s inspection, including testing when applicable, shall determine whether or not the Deliverables appear to be in compliance with the Contract. The Contractor shall be notified in writing of any apparent deficiency. The written notice shall Florida Standards Assessments, ITN 2014-47 151 V. VI. VII. VIII. IX. X. XI. detail the specific action required by the Contractor to correct the deficiency. The Contractor shall timely correct such deficiency and resubmit the deliverable for acceptance. The Contractor represents and agrees that information submitted in support of its requests for payment is the basis of payment and is true and accurate to the best of knowledge of the responsible signatory. A violation of this provision shall subject the violator to the provisions of s. 68.082, F.S., pertaining to false claims against the State, and/or s. 837.06, F.S., pertaining to false official statements. This paragraph applies if this Contract expires in a fiscal year subsequent to the fiscal year in which the Contract is entered. The State of Florida’s fiscal year comprises July 1 through June 30. The Department’s and State of Florida’s performance and obligation to pay under this Contract is contingent upon an annual appropriation by the Legislature. If the Legislature fails to make the necessary appropriation, the Department will determine if there are other unencumbered funds which are available and which can be lawfully expended to pay for the Department’s obligations hereunder. If the Department determines that there are no such funds, the Department shall promptly notify the Contractor. The giving of notice shall be deemed to have cancelled this Contract by mutual consent, with the date of notice being the date of cancellation. Notwithstanding anything to the contrary contained in a State Term Contract, Contractor warrants that all commodities, as defined in s. 287.012, F.S., shall meet the specifications of the Contract and shall be merchantable and fit for the particular purposes intended by the Contract. The Contractor further warrants that as to each Deliverable produced pursuant to this Contract, Contractor’s production of the Deliverable, and the Department’s use of the Deliverable, will not infringe on the copyrights of any third party. This provision applies to each work of authorship in which copyrights subsist pursuant to 17 U.S.C. Sections 102-105 and to each exclusive right established in 17 U.S.C. Section 106. In furtherance of this provision the Contractor additionally warrants that: A. As to each work of software or other “information technology”, as defined in s. 287.012(15), F.S., in which copyrights subsist, the Contractor has acquired the rights by conveyance or license to any third party software or other information technology, which was used to produce the Deliverable; B. As to each image and sound recording incorporated into a Deliverable, the Contractor has acquired the necessary rights, releases, and waivers from the person whose image or sound is included, or from the holder of the copyrights subsisting in the literary, musical, dramatic, pantomime, choreographic, pictorial, graphic, sculptural, motion pictures, audiovisual work or sound recording from which the included image or sound recording was taken. The Contractor further warrants that the Contractor shall not disclose to any third party, without the express, prior, written approval of the Department, any personally identifiable information about any student. This applies to information which came from any record or report of a Florida public education institution or from any education record which is subject to the Family Educational Rights and Privacy Act, 20 U.S.C. Section 1232g. The terms “record a report” and “student” shall have the meanings prescribed in s. 1002.22(2)(c) and (d), F.S. The term “educational record” shall have the meaning prescribed in 20 U.S.C. Section 1232g(a)(4). In the event that the Governor and Cabinet are required to impose a mandatory reserve on appropriations, the Department shall amend this Contract to place in reserve the amount determined by the Department of Education to be necessary because of the mandatory reserve. Such amendments may provide for adjustments in the Deliverable products and services as may be necessary. Intellectual property is subject to following additional provisions: A. Anything by whatsoever designation it may be known, that is produced by, or developed in connection with, this Contract shall become the exclusive property of the of the State of Florida and may be copyrighted, patented, or otherwise restricted as provided by Florida or federal law. Neither the Contractor nor any individual employed under this Contract shall have any proprietary interest in the product. B. With respect to each Deliverable that constitutes a work of authorship within the subject matter and scope of U.S. Copyright Law, 17 U.S.C. Sections 102-105, such work shall be a "work for hire" as defined in 17 U.S.C. Section 101 and all copyrights subsisting in such work for hire shall be owned exclusively by the Department pursuant to s. 1006.39, F.S., on behalf the State of Florida. C. In the event it is determined as a matter of law that any such work is not a "work for hire", Contractor shall immediately assign to the Department all copyrights subsisting therein for the consideration set forth in the Contract and with no additional compensation. D. The foregoing shall not apply to any preexisting software, or other work of authorship used by Contractor, to create a Deliverable but which exists as a work independently of the Deliverable, unless the preexisting software or work was developed by Contractor pursuant to a previous Contract with the Department or a purchase by the Department under a State Term Contract. E. The Department shall have full and complete ownership of all software developed pursuant to the Contract including without limitation: 1. The written source code; 2. The source code files; 3. The executable code; 4. The executable code files; 5. The data dictionary; 6. The data flow diagram; 7. The work flow diagram; 8. The entity relationship diagram; and Florida Standards Assessments, ITN 2014-47 152 XII. XIII. XIV. XV. XVI. XVII. XVIII. XIX. XX. XXI. XXII. XXIII. XXIV. XXV. 9. All other documentation needed to enable the Department to support, recreate, revise, repair, or otherwise make use of the software. The Department reserves the right, at its option, to issue a change order to delete work tasks reducing the total Contract amount by up to 10%. An addition of work tasks within the scope of the Contract, an increase in the total Contract amount, or a decrease of more than 10% of the total Contract amount, shall be implemented only by a Contract amendment signed by both the Department and the Contractor. Pursuant to s. 216.347, F.S., no funds awarded under this Contract may be used for the purpose of lobbying the Legislature, the judicial branch, or a State agency. The Contractor shall grant access to all records pertaining to the Contract to the Department’s Inspector General, General Counsel and other agency representatives, the State Auditor General, the Office of Program Policy and Government Accountability, and the Chief Financial Officer. The Contractor agrees to permit onsite visits by designated Department employees or agents to conduct audits to ensure compliance with Section 20.055, Florida Statutes. These audits may require Department access to records and data, computers and communications devices, and other materials whether owned or operated by the Contractor. Access may include, but is not limited to, user level and/or system level access to any computing or communications device; access to information (electronic, hardcopy, etc) that may be produced, transmitted or stored on the Contractor's equipment or premises; access to work areas; and access to interactively monitor and log traffic on the Contractor's networks. The Contractor must carry general liability insurance, which shall include errors and omissions coverage. The amount of coverage shall be a minimum of $1,000,000 or the aggregate total of all contractual agreements between the Contractor and the agencies and political subdivisions of the State of Florida, whichever is greater. The Contractor shall add the Department as an additional insured on the general liability coverage. The insurance shall cover all of the Contractor’s operations under this Contract and shall be effective throughout the Term of this Contract, as well as any renewals or extensions thereto. It is not the intent of this Contract to limit the types of insurance otherwise required by this Contract or that the Contractor may desire to obtain or be required to obtain by law. The Contractor must submit a Certificate of Insurance indicating coverage for general liability purposes and additional insured coverage, and shall maintain and pay for same throughout the Term of this Contract. A Certificate of Insurance indicating adequate coverage shall be submitted to the Department prior to the time the Contract is entered. Any and all insurance policies shall be through insurers qualified to do business in Florida. The Contractor agrees to provide the Department upon execution of this Contract with a performance bond or other security deposited with the Department in the total amount of the Contract or another amount if specified in the procurement specifications or Attachment A, guaranteeing that the Contractor will perform all work according to this Contract, within the time and price specified in the Contract. A performance bond shall be issued from a surety company, qualified to do business in Florida. The Contractor may not assign or subcontract all or any portion of this Contract without the advance written consent of the Department. In all cases in which the Contractor, with the advance written consent of the Department, assigns or subcontracts, all or any portion of the Contract: A. The Contractor shall monitor the subcontractor or assignee and establish controls to avoid or mitigate risks identified by the Department or the Contractor; and B. The Contractor shall allow the Department to monitor subcontractor or assignee activity and compliance, and the Contractor shall require the subcontractor or assignee to promptly submit to the Department, at the Department’s request, complete and accurate documentation pertaining to the subcontract or the Contract. The Contractor shall coordinate with and assist the Department’s Contract Manager in the performance of the latter’s responsibilities, which include without limitation: A. Monitoring the activities of the Contractor; B. Receiving and reviewing the reports of the Contractor to determine whether the objectives of the Contract are being accomplished; C. Receiving and reviewing the invoices for payment of funds to assure that the requirements of the Contract have been met and that payment is appropriate; D. Evaluating the process used by the Contractor to monitor the activities of any subcontractor or assignee; and E. Accessing, directly, the subcontractors and assignees, as the Contract Manager deems necessary. This Contract may not be modified unless in writing signed by the Department and the Contractor. The Department and the Contractor waive application of the principle of contract construction that ambiguities are to be construed against a contract’s drafter, and agree that this Contract is their joint product. The Department and the Contractor acknowledge that they have had their respective attorneys review and approve this Contract or that they have had the opportunity to do so. This Contract shall be governed by the laws of the State of Florida, and venue for purposes of any action brought to enforce or construe the Contract shall lie in Leon County, Florida. Failure of the Department to declare any default immediately upon the occurrence or knowledge thereof, or delay in taking any action in connection therewith, does not waive such default. The Department shall have the right to declare any such default at any time and take such action as might be lawful or authorized under the Contract, at law, or in equity. No Department waiver of any term, provision, condition or covenant of the Contract shall be deemed to imply or constitute a further Department waiver of any other term, provision, condition or covenant of the Contract, and no payment by the Department shall be deemed a waiver of any default under the Contract. Florida Standards Assessments, ITN 2014-47 153 XXVI. XXVII. XXVIII. XXIX. XXX. XXXI. XXXII. XXXIII. XXXIV Time is of the essence with regard to each and every obligation of the Contractor contained in the Contract. Each such obligation is deemed material, and a breach of any such obligation (including a breach resulting from the untimely performance thereof) shall constitute a material breach. The Contractor shall indemnify and hold harmless the Department, its attorneys, agents and employees, from and against any and all third party claims, suits, debts, damages, and causes of action, whatsoever, whether arising in law or in equity, arising out of or relating to Contractor performance or failure to perform under this Contract. The indemnification shall include reasonable attorney fees and costs incurred by the Department, its attorneys, agents and employees, in the defense of any such claim, suits or causes of action, as aforesaid. This Contract may be cancelled by written agreement of the Department and the Contractor specifically referencing this Contract. Such agreement shall specify the remaining measures necessary to be taken by each party. The Department reserves the right to cancel this contract without cause by giving the Contractor thirty (30) days written notice. Should Contractor fail to perform to Contract terms and conditions, Contractor shall be notified in writing, stating the nature of the failure to perform and providing a time certain (which shall be not less than ten (10) days following receipt of such notice) for correcting the failure. Such failure to perform shall otherwise be dealt within accordance with Rule 60A-1.006, F.A.C. A person or affiliate who has been placed on the convicted vendor list following a conviction for a public entity crime may not submit a bid on a contract to provide any goods or services to a public entity, may not submit a bid on a contract with a public entity for the construction or repair of a public building or public work, may not submit bids on leases of real property to a public entity, may not be awarded or perform work as a contractor, supplier, subcontractor, or consultant under a contract with any public entity, and may not transact business with any public entity in excess of the threshold amount provided in s. 287.017, F.S., for CATEGORY TWO for a period of thirty-six (36) months from the date of being placed on the convicted vendor list. The employment of unauthorized aliens by any contractor is considered a violation of Section 274A(e) of the Immigration and Nationality Act. If the contractor knowingly employs unauthorized aliens, such violation shall be cause for unilateral cancellation of the contract. In addition, pursuant to State of Florida Executive Order No. 11-116, Contractor shall utilize the U.S. Department of Homeland Security’s E-Verify system to verify the employment of all new employees hired by the Contractor during the contract term. Also, Contractor shall include in related subcontracts a requirement that subcontractors performing work or providing services pursuant to the Contract utilize the E-Verify system to verify employment of all new employees hired by the subcontractor during the contract term. Certification Regarding Debarment, Suspension, Ineligibility and Voluntary Exclusion – Lower Tier Covered Transactions A. The prospective lower tier participant certifies, by submission of this proposal, that neither it nor its principals is presently debarred, suspended, proposed for debarment, declared ineligible, or voluntarily excluded from participation in this transaction by any Federal department or agency. B. Where the prospective lower tier participant is unable to certify to any of the statements in this certification, such prospective participant shall attach an explanation to this proposal. The Department may cancel this contract if an attached explanation is not acceptable to the Department or the Federal government. MyFloridaMarketPlace A. MyFloridaMarketplace Vendor Registration Each Vendor doing business with the State of Florida for the sale of commodities or contractual services as defined in section 287.012, Florida Statutes, shall register in MyFloridaMarketPlace, in compliance with Rule 60A-1.030, Florida Administrative Code, unless exempt under Rule 60A-1.030(3) Florida Administrative Code. B. MyFloridaMarketplace Transaction Fee The State of Florida, through the Department of Management Services, has instituted MyFloridaMarketPlace, a statewide eProcurement system. Pursuant to section 287.057(23), Florida Statutes (2010), all payments for commodities and/or contractual services as defined in Section 287.012, Florida Statutes, shall be assessed a Transaction Fee of one percent (1.0%), which the Vendor shall pay to the State, unless exempt under Rule 60A-1.032, Florida Administrative Code. Notwithstanding the provisions of Rule 60A-1.030, et seq., the assessment of a transaction fee shall be contingent upon Federal approval of the transaction fee assessment program and continued payment of applicable federal matching funds. For payments within the State accounting system (FLAIR or its successor), the Transaction Fee shall, when possible, be automatically deducted from payments to the Vendor. If automatic deduction is not possible, the Vendor shall pay the Transaction Fee pursuant to Rule 60A-1.031(2), Florida Administrative Code. By submission of these reports and corresponding payments, Vendor certifies their correctness. All such reports and payments shall be subject to audit by the State or its designee. The Vendor shall receive a credit for any Transaction Fee paid by the Vendor for the purchase of any item(s) if such item(s) are returned to the Vendor through no fault, act, or omission of the Vendor. Notwithstanding the foregoing, a Transaction Fee is non-refundable when an item is rejected or returned, or declined, due to the Vendor’s failure to perform or comply with specifications or requirements of the agreement. Failure to comply with these requirements shall constitute grounds for declaring the Vendor in default and recovering reprocurement costs from the Vendor in addition to all outstanding fees. VENDORS DELINQUENT IN PAYING TRANSACTION FEES MAY BE EXCLUDED FROM CONDUCTING FUTURE BUSINESS WITH THE STATE. Florida Standards Assessments, ITN 2014-47 154
  • . The header section of a complex table (i.e, a table with more than one header row) should begin with