State of Florida
Department of Education
INVITATION TO NEGOTIATE
FLORIDA STANDARDS ASSESSMENTS
BID NUMBER: ITN 2014-47
DEADLINE FOR TECHNICAL QUESTIONS: November 8, 2013 @ 2:00 PM
(There is no deadline for administrative questions)
REPLIES ARE DUE BY: DECEMBER 12, 2013 @ 2:00 PM, EASTERN TIME (ET)
MAIL OR DELIVER REPLIES TO:
Florida Department of Education
Bureau of Contracts, Grants and Procurement Management Services
325 West Gaines Street
332 Turlington Building
Tallahassee, Florida 32399-0400
Attention: Fran Shewan
Phone: (850) 245-9884
Florida Standards Assessments, ITN 2014-47
1
Table of Contents
INVITATION TO NEGOTIATE REGISTRATION
5
1.0
GENERAL INSTRUCTIONS TO RESPONDENT
SECTION 2 – CONTRACT CONDITIONS
2.0
GENERAL CONTRACT CONDITIONS
6
7
7
SECTION 1 - OVERVIEW
SECTION 1 –INSTRUCTIONS
SECTION 3 - INTRODUCTION
6
6
7
3.0
3.1
3.2
3.3
3.4
INTENT
PURPOSE
BACKGROUND
DEFINITIONS
CRITICAL EVENT DATES
7
7
8
8
10
4.0
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9.
4.10
4.11
4.12
PRE-SOLICITATION CONFERENCE:
SITE INSPECTION:
VISITOR’S PASS TO THE TURLINGTON BUILDING
REPLY QUESTIONS & ANSWERS
PROCUREMENT PROTESTS / NOTICE OF RIGHTS
ORAL INSTRUCTIONS / CHANGES TO THE INVITATION TO NEGOTIATE (ADDENDA)
MODIFICATIONS, RESUBMITTAL AND WITHDRAWAL
RESTRICTIONS ON COMMUNICATIONS WITH DEPARTMENT STAFF
CONFIDENTIAL, PROPRIETARY, OR TRADE SECRET MATERIAL
POOR PERFORMANCE NOTICE
WITHDRAWAL OF A REPLY
DISCLOSURE OF REPLY CONTENTS
AWARD
11
11
11
11
11
12
12
12
13
13
13
14
14
5.0
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
5.10
5.11
5.12
5.13
5.14
5.15
5.16
5.17
5.18
5.19
AUTHORIZED TO DO BUSINESS IN THE STATE OF FLORIDA
LICENSED TO CONDUCT SERVICES IN THE STATE OF FLORIDA
OTHER CONDITIONS
IDENTICAL EVALUATION OF REPLIES
DISCLOSURE STATEMENT
SUB-CONTRACTING
CONTRACTUAL OBLIGATIONS
METHOD OF PAYMENT
CONVICTED VENDOR LIST
DIVERSITY IN CONTRACTING
COSTS INCURRED IN RESPONDING
SUBMISSION OF REPLIES BY SUBSIDIARIES OR AFFILIATES
PROHIBITION OF GRATUITIES
INDEPENDENT PRICE DETERMINATION
FINANCIAL CONSEQUENCES
PERFORMANCE BOND
PARTICIPATION IN FUTURE STAGES OF THIS PROJECT
ACCESSIBLE ELECTRONIC INFORMATION TECHNOLOGY
SCRUTINIZED COMPANIES LISTS (AS APPLICABLE)
LIQUIDATED DAMAGES
14
14
14
14
15
15
15
15
15
15
16
16
16
16
16
16
17
17
17
17
6.0
REPLY SUBMISSION
19
Florida Standards Assessments, ITN 2014-47
2
SECTION 4 – SPECIAL INSTRUCTIONS
SECTION 5 – SPECIAL CONDITIONS
SECTION 6 – SPECIAL INSTRUCTIONS – REPLY FORMAT & CONTENT
11
14
19
6.1
6.2
6.2.1
6.2.2
6.2.3
6.2.4
MAIL OR DELIVER REPLIES TO: (DO NOT FAX OR E-MAIL)
REPLY FORMAT INSTRUCTIONS
PRELIMINARY ADMINISTRATIVE REVIEW
TECHNICAL REPLY (PART I) (20 HARD COPIES)
PRICE REPLY (PART II) (20 HARD COPIES)
PRESENTING THE REPLY
SECTION 7 – SCOPE OF SERVICES
7.0
7.1.0
7.1.1
7.1.2
7.2.0
7.2.1
7.2.2
7.3.0
7.3.1
7.3.2
7.3.3
7.3.4
7.3.5
7.3.6
7.3.7
7.3.8
7.3.9
7.3.10
SCOPE OF SERVICES
INTRODUCTION
NOTICE TO RESPONDENTS
PROJECT SCHEDULE
PROGRAM DESCRIPTIONS
EXPECTED NUMBER OF DISTRICTS, SCHOOLS, AND STUDENTS
POTENTIAL TEST ADMINISTRATION DATES
TEST DESIGN AND DEVELOPMENT
TEST CONTENT
TEST CONFIGURATION
DEVELOP ITEMS
REVIEW MEETINGS
DEVELOP FIELD-TEST ITEM SETS AND FORMS
DEVELOP STAND-ALONE ELA/L FIELD-TEST ITEM SETS AND FORMS
CONSTRUCTING TEST FORMS
PRODUCTION OF PRINTED TEST MATERIALS
PRODUCTION OF COMPUTER-DELIVERED PRODUCTS
PRODUCTION OF PAPER-BASED AND COMPUTER-BASED TEST, STYLE GUIDE, AND BRAILLE
PRODUCTION PROCEDURES
7.3.11 PRODUCE SPECIAL FORMATS AND ACCOMMODATIONS FOR STUDENTS WITH DISABILITIES
7.4.0 TEST ADMINISTRATION
7.4.1 PACK AND DISTRIBUTE MATERIALS
7.4.2 MISSING MATERIALS REPORT AND INVENTORY
7.4.3 DISPOSITION AND STORAGE OF MATERIALS
7.4.4 RETRIEVAL OF ANSWER SHEETS AND TEST BOOKS
7.4.5 INNOVATIVE PRODUCT DESIGN
7.4.6 PRINTED PRODUCTS – ADMINISTRATION
7.4.7 TEST ADMINISTRATION ANCILLARY MATERIALS
7.4.8 ONLINE MANAGEMENT SYSTEM
7.4.9 COMPUTER-BASED TEST ADMINISTRATION OVERVIEW
7.4.10 CUSTOMER SERVICE AND TECHNICAL SUPPORT
7.5.0 SCANNING, SCORING, AND REPORTING
7.5.1 SCORING AND REPORTING SPECIFICATIONS
7.5.2 PREPARING SCHOOL AND STUDENT INFORMATION (MSID AND PREID)
7.5.3 SCANNING/EDITING PBTS AND EXTRACTING CBT RECORDS
7.5.4 SCORING
7.5.5 REPORTING
7.5.6 VERIFYING DATA
7.6.0 SCALING, EQUATING, SCORING, AND SPECIAL PSYCHOMETRIC STUDIES
7.6.1 ASSESSMENTS
7.6.2 SCALING, EQUATING, AND SCORING (PSYCHOMETRICS)
7.6.3 TECHNICAL OVERSIGHT AND SPECIAL STUDIES
7.7.0 PROGRAM MANAGEMENT
7.7.1 PROVISIONS GOVERNING THE WORK OF THE CONTRACTOR
Florida Standards Assessments, ITN 2014-47
19
19
19
20
23
23
23
23
23
24
25
25
26
26
27
28
29
30
30
32
32
32
34
35
36
36
39
39
43
43
44
44
44
46
49
52
58
59
59
60
62
63
68
73
77
78
79
81
87
87
3
7.7.2
7.7.3
7.7.4
7.7.5
7.7.6
7.7.7
7.7.8
7.7.9
7.8
CONTRACTOR STAFF AND RESPONSIBILITIES
SUBCONTRACTORS
PROGRAM MANAGEMENT COMMUNICATION
PROGRAM MANAGEMENT MEETINGS
PROGRAM MANAGEMENT REPORTS
MEETINGS INVOLVING FLORIDA EDUCATORS AND OTHER CITIZENS
COMMUNICATION BETWEEN THE CONTRACTOR AND DISTRICTS
REIMBURSABLE FUNDING CATEGORIES
DELIVERABLES (SEE ATTACHED SECTION 7.8 DELIVERABLES PROVIDED IN AN EXCEL
SPREADSHEET.)
SECTION 8 – OPENING, EVALUATION AND AWARD
87
89
89
90
90
91
93
94
96
96
8.0
REPLY OPENING
96
8.1
REPLY EVALUATION AND NEGOTIATION PROCESS
96
8.2
EVALUATIONS
97
8.3
POSTING OF INTENDED AWARD
98
8.3.1 PROTEST OF INTENDED AWARD
98
8.3.2 INABILITY TO POST
98
8.4
AWARD OF THE CONTRACT
98
APPENDIX A: TRANSITION TO COMPUTER-BASED TESTING AND ADDITIONAL TEST ADMINISTRATION
DATES
99
APPENDIX B: CRITICAL AND EXTREMELY CRITICAL ACTIVITIES
101
APPENDIX C: FLORIDA PUBLIC SCHOOL STUDENT PARTICIPATION ESTIMATES, DISTRICT NUMBERS, AND
SPECIAL DISTRICT/SCHOOL NUMBERS
104
APPENDIX D: SAMPLE TEST DESIGN
107
APPENDIX E: PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS
108
APPENDIX F: TEST SECURITY REQUIREMENTS, STATUTE, AND RULE
114
APPENDIX G: TEST ADMINISTRATION MANUAL WORKFLOW
118
APPENDIX H: TEST ACCOMMODATIONS
124
APPENDIX I: DATA PRIVACY & SECURITY POLICY
131
APPENDIX J: REQUIREMENTS FOR SCORING PERFORMANCE ITEMS
137
ATTACHMENT 1: RESPONDENT’S PRICE REPLY
145
ATTACHMENT 2: DRUG-FREE WORKPLACE
146
ATTACHMENT 3: DISCLOSURE STATEMENT
147
ATTACHMENT 4: REFERENCES
148
ATTACHMENT 5: MINORITY SUB CONTRACTORS UTILIZATION SUMMARY
149
ATTACHMENT 6: VENDOR CERTIFICATION REGARDING SCRUTINIZED COMPANIES LISTS
150
ATTACHMENT 7: CONTRACT STANDARD TERMS AND CONDITIONS
151
Florida Standards Assessments, ITN 2014-47
4
State of Florida
Department of Education
Bureau of Contracts, Grants and Procurement Management Services
325 West Gaines Street
332 Turlington Building
Tallahassee, Florida 32399-0400
INVITATION TO NEGOTIATE REGISTRATION
**************************************************************************************************************************
COMPLETE AND RETURN THIS FORM
**************************************************************************************************************************
Reply Number: ITN 2014-47
Title: Florida Standards Assessments
Date & Time Reply Due: December 12, 2013 @ 2:00 PM, EASTERN TIME (ET)
Potential Respondents should notify the Florida Department of Education, Bureau of Contracts, Grants and
Procurement Management Services by returning this Intent to Submit Reply Form as soon as possible after
downloading. Complete the information below and send this sheet only to below e-mail address
(preferred). Form can also be faxed to fax number (850) 245-0719, or mailed to 325 West Gaines Street,
332 Turlington Building, Tallahassee, Florida 32399-0400.
Company Name:
Contact Person:
Address:
City, State, Zip:
Telephone: ( )
Fax Number: ( )
Internet E-Mail Address:
Signed:
Date:
For further information on this process, you may contact Fran Shewan – fran.shewan@fldoe.org at (850)
245-9884.
(Revised 10/06/2012)
Florida Standards Assessments, ITN 2014-47
5
STATE OF FLORIDA
DEPARTMENT OF EDUCATION
INVITATION TO NEGOTIATE
FLORIDA STANDARDS ASSESSMENTS
BID NUMBER: ITN 2014-47
SECTION 1 - OVERVIEW
SECTION 1 –INSTRUCTIONS
1.0
GENERAL INSTRUCTIONS TO RESPONDENT
This section contains instructions explaining the solicitation process and the actions necessary to respond.
General Instructions to Respondent (Form PUR 1001 – incorporated herein by reference) is a downloadable
document which must be downloaded for review. This document need not be returned with the Respondent’s
Reply. Form PUR 1001 may be accessed at http://dms.myflorida.com/business_operations/state_purchasing
under “Documents, Forms, References and Resources”.
In the event of any conflict between Form PUR 1001 and other instructions provided in this document, the
additional instructions in this document shall take precedence over the Form PUR 1001 unless the conflicting
term is required by any section of the Florida Statutes (F.S.), in which case the statutory requirements shall
take precedence.
It is entirely the respondent’s responsibility to examine this ITN, to confirm that the Department’s requirements
are clearly stated, and to submit its reply in a timely, complete, and procedurally correct manner. The
products and services described in this ITN will be procured in accordance with Chapter 287, Florida Statutes
(F.S.), and Chapter 60A-1, Florida Administrative Code (FAC).
THE DEPARTMENT HAS CHOSEN TO USE THE ITN FORMAT FOR THIS PROCUREMENT BECAUSE IT
WANTS VENDORS TO PROPOSE THE BEST METHOD FOR ACHIEVING THE GOAL OF THIS ITN AND
SOLVING THE PROBLEM STATED HEREIN. THEREFORE, ALTHOUGH THE ITN MAY USE
MANDATORY WORDS LIKE “SHALL,” “WILL,” OR “MUST,” AND MAY DEFINE CERTAIN ITEMS AS
REQUIREMENTS, THE DEPARTMENT RESERVES THE RIGHT, IN ITS DISCRETION, TO WAIVE ANY
DEVIATIONS FROM THESE PROVISIONS AND RESOLVE ANY ISSUES IN THE NEGOTIATION PHASE.
THEREFORE, THE DEPARTMENT RESERVES THE RIGHT TO REVIEW THE ENTIRE REPLY TO
DETERMINE IF IT ACHIEVES A LEVEL OF COMPETENCY WORTHY OF FURTHER NEGOTIATIONS;
REGARDLESS OF WHETHER INDIVIDUAL REQUIREMENTS HAVE BEEN ADDRESSED OR NOT.
HOWEVER, VENDORS THAT FAIL TO PROVIDE SIGNIFICANT PORTIONS OF THE SOLUTION OR
ADDRESS SIGNIFICANT PORTIONS OF THE PROCUREMENT MAY STILL BE DEEMED
NONRESPONSIVE. IN ADDITION, THERE IS NO GUARANTEE THAT SUCH DEVIATIONS WILL BE
DEEMED IN THE STATE’S BEST INTEREST OR ANY REPLY CONTAINING THOSE DEVIATIONS
PARTICIPATE IN THE NEGOTIATIONS. USE OF THE TERMS “SHALL,” “WILL,” AND “MUST” INDICATE
THE DEPARTMENT’S INITIAL VIEW OF THE VALUE OF SUCH ITEMS. VENDORS HAVE THE
OPPORTUNITY TO SUGGEST ALTERNATIVES IN THE ITN PROCESS, BUT THERE IS NO GUARANTEE
THAT THE DEPARTMENT WILL AGREE THE DEVIATIONS ARE IN ITS BEST INTEREST OR CREATE
THE BEST VALUE FOR THE STATE.
Florida Standards Assessments, ITN 2014-47
6
SECTION 2 – CONTRACT CONDITIONS
2.0
GENERAL CONTRACT CONDITIONS
Standard terms and conditions that will apply to the contract which results from the solicitation event are
provided in this section. General Contract Conditions (Form PUR 1000 – incorporated herein by reference) is
a downloadable document which must be downloaded for review. This document need not be returned with
the
Respondent’s
Reply.
Form
PUR
1000
may
be
accessed
at
http://dms.myflorida.com/business_operations/state_purchasing under “Documents, Forms, References and
Resources”.
In the event of any conflict between the PUR 1000 form and any other Special Conditions, the Special
Conditions shall take precedence over the PUR 1000 form unless the conflicting term in the PUR form is
required by any section of the F.S., in which case the statutory requirements shall take precedence.
SECTION 3 - INTRODUCTION
3.0
INTENT
The State of Florida Department of Education (hereinafter referred to as the "Department") is soliciting written
replies from qualified vendors to establish a term contract of which the term is anticipated to begin upon
execution of the contract and be effective for 36) months thereafter. Award(s) will be made to the responsible
and responsive vendor that the Department determines will provide the best value to the state.
The resulting contract(s) may be renewed for up to three additional 12 month terms, see Form PUR 1000 for
renewal requirements.
Although this ITN may result in the award of multiple contracts, the term “Contractor” is used throughout this
ITN and refers to each contractor providing the required products and services for each grade and subject for
which the respective contract is awarded.
3.1
PURPOSE
The purpose of this Invitation to Negotiate (ITN) is to solicit replies for activities related to the implementation
of computer-based and paper-based statewide summative assessments aligned to the Florida Standards for
administrations beginning in spring 2015 and continuing through 2016-17 for the base contract with three oneyear optional renewals for the administrations in 2017-18, 2018-19, and 2019-20. Henceforth, in this
document, this test shall be called the FSA (Florida Standards Assessment)
In accordance with Governor Rick Scott’s Executive Order, dated September 23, 2013, this ITN is seeking
Replies that:
A. Provides timely and informative reports of results;
B. Includes costs that are in line with current assessment costs;
C. Ensures that testing time for students is not significantly different from current assessments;
D. Provides for summative assessments to occur as close as possible to the end of the school year;
E. Measures student mastery of the standards taught, including comparability to other states;
F. Includes test quality metrics that are as rigorous as current assessments;
G. Provides results that can be used in conjunction with Florida’s school accountability system;
H. Requires technology parameters that are defined and can be supported, including appropriate
accommodations for exceptional students.
Florida Standards Assessments, ITN 2014-47
7
This ITN addresses four broad assessment components as described below. Respondents may submit bids
on any individual assessment component (A, B, C or D), any group of assessment components, or to provide
all assessment components (A, B, C and D):
A. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 3-8
B. Computer-based and paper-based assessments in Mathematics in Grades 3-8
C. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 9-11
D. Computer-based and paper-based End–of-Course (EOC) assessments in Algebra 1, Geometry, and
Algebra 2
A preferred schedule and plan for transition to computer-based testing is included in Appendix A. However,
depending on the final award for the individual assessments, the Department may vary when tests will be
administered as computer-based or paper-based. The final determination of which tests will be computerbased or paper-based is at the Department’s discretion.
In addition, all Respondents agree, by submitting a reply, that any other state or school district in the United
States may purchase the services provided in the resulting contract, upon the same terms and conditions, and
price, as contained in the contract, or as permitted under the laws of that jurisdiction.
3.2
BACKGROUND
The Florida Comprehensive Assessment Test® 2.0 (FCAT 2.0) and Florida End-of-Course (EOC)
Assessments implement the requirements of Section 1008.22, Florida Statutes (F.S.), requiring assessments
of the academic achievement of Florida’s public school students. The FCAT 2.0/EOC Assessment program is
composed of state-developed criterion-referenced assessments. FCAT 2.0 Reading and Mathematics have
been administered each spring since 2011 to assess students in selected grade levels. FCAT 2.0 Science
was introduced in spring 2012, FCAT 2.0 Writing was introduced in spring 2013, and the Florida EOC
Assessments have been phased in beginning in 2011. Prior to FCAT 2.0, FCAT assessments were
administered each spring since 1998, and included Reading, Mathematics, Writing, and Science. The final
administration of FCAT 2.0 Writing, FCAT 2.0 Reading, and FCAT 2.0 Mathematics will be in spring 2014.
The final administration of the FCAT 2.0 Reading Retake for students with this graduation requirement will be
in spring 2016, and the Florida Algebra 1 and Geometry EOC Assessments will be eliminated after the
summer 2017 administration. (Administration of the FCAT 2.0 Reading Retake and the Florida EOC
Assessments will be handled in another procurement.)
The Sunshine State Standards, which specify challenging expectations for the educational achievement of
Florida students, were revised and approved by the State Board of Education as the Next Generation
Sunshine State Standards (NGSSS). The Reading/Language Arts and Mathematics revised standards were
approved by the State Board in 2007. Science and Social Studies Standards were approved in February
2008. The NGSSS and associated course descriptions for the Florida End-of-Course Assessments are
available at www.flstandards.org. In 2010, Florida adopted the Common Core State Standards for English
Language Arts/Literacy and for Mathematics. As of the release of this ITN, these standards were scheduled
to be reviewed further by Florida’s stakeholders. The degree to which the standards, and by extension the
assessments, will be adjusted is to be determined. Florida plans for the Florida Standards Assessments to be
operational beginning in 2014-15.
3.3
DEFINITIONS
After the award, said Respondent will be referred to as the "Contractor". For the purpose of this document,
the term "Respondent" means a potential Contractor acting on its own behalf and on behalf of those
individuals, partnerships, firms, or corporations comprising the Respondent’s team. The term "Reply" means
the complete response of the Respondent to the ITN, including properly completed forms and supporting
documentation. The term “contract” refers to the agreement between the Department and the Contractor
Florida Standards Assessments, ITN 2014-47
8
resulting from this ITN. “Best value” means the highest overall value to the state based on objective factors
that include, but are not limited to, price, quality, design, and workmanship. A “responsive bid” is a Reply
submitted by a responsive and responsible vendor which conforms in all material respects to the solicitation.
However, for purposes of determining what is material, please refer to Section 1 of this ITN. Use of the terms
“shall,” “will,” and “must,” or the designation of items as “requirements,” does not mandate that such items are
material. The Department reserves the right to waive deviations, in its discretion, and resolve any issues in
the negotiation phase. “Deliverable” means a tangible, specific, quantifiable and measurable event or item
that must be produced to complete a project or part of a project directly related to the scope of services.
Additional definitions/terms as follows:
a.
b.
c.
d.
e.
f.
g.
h.
i.
j.
k.
l.
m.
n.
o.
p.
q.
r.
s.
t.
u.
v.
w.
AL – Achievement Level
Braille – Paper-based accommodation provided for eligible students with disabilities
CBT – Computer-based test; computer-based testing
Cohort – For the purposes of this ITN, refers to a group of students that enters grade 9 and is enrolled at
the high school for a total of four years
District Assessment Coordinator – Primary contact(s) in each school district with whom FDOE
communicates regarding the administration of K-12 statewide assessments and who oversees the
administration of statewide assessments at all participating schools in that district
DAR – District Aggregated Results files used for reporting purposes
DIF – Differential Item Functioning
DSR – District Student Results file used for reporting purposes
DSS – Developmental Scale Score
EOC- End-of-Course – Refers to an assessment that is associated with a specific course
FDOE (DOE) – Florida Department of Education
FPO – Placeholder for a graphic used in a publication
FSA – Florida Standards Assessment, the current name of the assessments to be administered as part of
this ITN
IEP – Individual Educational Plan for students with disabilities (ESE)
IRT – Item Response Theory
Large Print – Paper-based or computer-based accommodation provided for eligible students with
disabilities; large print usually refers to at least 18 point font
MS (Manuscript) – The initial production round when a document is composed and first reviewed
MSID (Master School Identification File) – A file maintained by the Department and available online used
to identify all active schools that will participate in statewide assessment administrations
One-Item-Per-Page – Paper-based accommodation provided for eligible students with disabilities; only
one item appears on each printed page
P&D – Packaging and distribution; often used in reference to the specifications that describe a
contractor’s processes of packaging and distributing all statewide assessment materials prior to each
assessment administration
PBT – Paper-based test; paper-based testing
PreID (Preidentification) – Process for identifying students prior to an administration to print labels with all
of the student’s information (PreID labels) or to create computer-based student records that will be used
for testing
PT (Program team) – Contractor staff members with program-wide expertise who are expected to assist
in developing accurate instructions and providing specific information related to the contractor’s
processes and procedures regarding Florida’s statewide assessments
Florida Standards Assessments, ITN 2014-47
9
x. Rangefinding – Process required for scoring purposes by which student responses to performance tasks
are used to illustrate score points on a rubric
y. RD – Results Delivery (RD1, RD2, etc.)
z. SAR – State Aggregated Results files used for reporting purposes
aa. School Assessment Coordinator – A designated school-level assessment contact who oversees
administration of statewide assessments at that school, training of test administrators and proctors, and
preparing and returning test materials
bb. Seals – Method of maintaining security of test content until a test session begins; paper-based tests may
have one or more paper or plastic seals that students can easily break without tearing the test book, and
computer-based tests may have an electronic means of sealing a test session (e.g., content is locked until
a specific code is entered that allows access)
cc. Section 504 Plan – Plan detailing accommodations for students with temporary disabilities (may not be
ESE)
dd. Section 508 Compliant (ADA Compliant) – Section 508 requires that electronic and information
technologies which are procured or developed, be maintained by federal agencies and be accessible to
people with disabilities, unless it poses an undue burden to do so. Accessibility standards needed to
comply with this law were developed by the United States Architectural and Transportation Barriers
Compliance Board. Section 508 of the Rehabilitation Act of 1973 is in the federal code of regulations (29
U.S.C. § 794(d).
ee. SES (Scaling, Equating, and Scoring) – The set of processes used to ensure that assessment results are
valid, reliable, and accurate
ff. SS – Scale score
gg. SSR – State Student Results files used to check data for reporting purposes
hh. STMs – Sample Test Materials
ii. Test Administrator – A certified educator who administers statewide assessments to students in a testing
room; responsible for reading administration scripts to students and maintaining a secure testing
environment
jj. TAC (Technical Advisory Committee) – A panel of experts who review and respond to proposed
modifications and processes for the assessment program
kk. TAM – Test administration manual
ll. TC (Technology Coordinator) – A designated coordinator at the school or district level responsible for
preparing for computer-based testing and addressing technical issues during testing
mm.TDC – Test Development Center, which is overseen by the FDOE and staffed with editors and Florida
educators; TDC provides quality assurance for all test content and related products, whether customized
or derived from existing products
nn. TM – Test Materials Shipment (TM1, TM2, etc)
3.4
CRITICAL EVENT DATES
These are tentative dates for critical events, and are subject to change.
•
October 25, 2013 – ITN Released
•
November 8, 2013 @ 2:00 PM, ET - Deadline for Technical Questions
•
November 15, 2013 - Department Response to Technical Questions received by deadline
•
December 12, 2013 @ 2:00 PM, ET - Technical & Price Replies are due
Florida Standards Assessments, ITN 2014-47
10
•
December 12, 2013 @ 2:30 PM - Technical Replies will be opened in Room 332 Turlington Building
(Step 1)
•
January 10, 2014 - The Department will evaluate the Technical Replies in Room 1706 Turlington Building
@ 9:00 AM (Step 2)
Price Replies will be evaluated by the Bureau of Contracts, Grants and Procurement Management Services
staff during the evaluation stage.
•
January 28, 2014 - The Negotiation Committee will start negotiations on or about (Step 4)
•
February 13, 2014 BAFO’s due on or about
•
February 25, 2014 – ITA Meeting on or about
The Intent to Award will be posted for 72 hours upon completion of the negotiation meeting and a final
decision by the Department.
SECTION 4 – SPECIAL INSTRUCTIONS
4.0
PRE-SOLICITATION CONFERENCE:
A Pre-Solicitation Conference will not be held.
4.1
SITE INSPECTION:
4.2
VISITOR’S PASS TO THE TURLINGTON BUILDING
Each visitor to the Turlington Building is required to sign in and obtain a Visitor's Pass at the security desk in
the main lobby. Please allow at least 15 minutes prior to Reply due time if hand-delivering the Reply to the
Bureau of Contracts, Grants and Procurement Management Services.
4.3
REPLY QUESTIONS & ANSWERS
Any technical questions arising from this ITN must be forwarded, in writing, to the purchasing agent identified
below. The Department's written response to written inquiries submitted timely by Respondents will be posted
on the Florida Vendor Bid System (VBS) at www.myflorida.com (click on Business, then click on Doing
Business with the state, under Everything for Vendors and Customers, click on the Vendor Bid System, then
Search Advertisement; select the Department of Education in the Agency drop down window and initiate
search), under this Reply number. It is the responsibility of all potential Respondents to monitor this site for
any changing information prior to submitting a Reply.
A Site Inspection will not be held.
Only timely received written inquiries will be “officially” addressed by the Department.
WRITTEN QUESTIONS should be submitted to:
Bureau of Contracts, Grants and Procurement Management Services, Fran Shewan, 325 West Gaines
Street, 332 Turlington Building, Tallahassee, Florida 32399-0400, fshewan@fldoe.org (email preferred),
or via Fax Number: (850) 245-0719. See SECTION 3.4 Critical Event Dates for question due dates.
4.4
PROCUREMENT PROTESTS / NOTICE OF RIGHTS
Pursuant to F.S., Section 120.57(3) (b):
Any person who is adversely affected by the agency decision or intended decision shall file with the agency a
notice of protest in writing within 72 hours after the posting of the notice of decision or intended decision. With
respect to a protest of the terms, conditions, and specifications contained in a solicitation, including any
Florida Standards Assessments, ITN 2014-47
11
provisions governing the methods for ranking bids, proposals, or replies, awarding contracts, reserving rights
of further negotiation, or modifying or amending any contract, the notice of protest shall be filed in writing
within 72 hours after the posting of the solicitation. The formal written protest shall be filed within 10 days after
the date the notice of protest is filed. Failure to file a notice of protest or failure to file a formal written protest
shall constitute a waiver of proceedings under this chapter. The formal written protest shall state with
particularity the facts and law upon which the protest is based. Saturdays, Sundays, and state holidays shall
be excluded in the computation of the 72-hour time periods provided by this paragraph.
Section 120.57(3) (a) provides:
Failure to file a protest within the time prescribed in Section 120.57(3), F.S. or failure to post the bond or other
security required by law within the time allowed for filing a bond shall constitute a waiver of proceedings under
Chapter 120, F.S."
Florida Administrative Code (F.A.C.) Rule 28-110.002(2) defines the term “decision or intended decision,” and
includes the solicitation terms (and any addenda), the award of the contract, and a rejection of all bids.
At the time of filing the Formal Written Protest the protestor must also file a Protest Bond payable to the
Department in an amount equal to one percent (1%) of the estimated contract amount. F.S., Section
287.042(2) (c) and F.A.C. Rule 28-110.005 contain further terms relating to the Protest Bond, including how to
determine the estimated contract amount. In lieu of a Protest Bond, the Department will accept cashier’s
checks, official bank checks or money orders. The bond shall be conditioned upon the payment of all costs
and charges that are adjudged against the protestor in the administrative hearing in which the action is
brought and in any subsequent appellate court proceeding.
The Notice of Protest, Formal Written Protest, and Protest Bond shall be filed with the issuing office as defined
in SECTION 4.3 above.
4.5
ORAL INSTRUCTIONS / CHANGES TO THE INVITATION TO NEGOTIATE (ADDENDA)
No negotiations, decisions, or actions will be initiated or executed by a Respondent as a result of any oral
discussions with a state employee. Only those communications which are in writing from the Bureau of
Contracts, Grants and Procurement Management Services, will be considered as a duly authorized
expression on behalf of the Department until negotiation is awarded.
Notices of changes (addenda) will be posted on the VBS, under this Reply number. It is the responsibility of all
potential Respondents to monitor this site for any changing information prior to submitting a Reply. All
addenda should be acknowledged by the Respondent’s signature and subsequent submission of addenda
with Reply when so stated in the addenda.
4.6
MODIFICATIONS, RESUBMITTAL AND WITHDRAWAL
Respondents may modify submitted replies at any time prior to the Reply due date. Requests for modification
of a submitted Reply shall be in writing and should be signed by an authorized representative of the
Respondent. Upon receipt and acceptance of such a request, the entire Reply will be returned to the
Respondent and not considered unless resubmitted by the due date and time. Respondents may also send a
change in a sealed envelope to be opened at the same time as the Reply. The ITN number, opening date and
time should appear on the envelope of the modified Reply.
Unless specifically requested by the Department, any amendments, revisions, or alterations to Replies will not
be accepted after the closing for the receipt of Replies.
4.7
RESTRICTIONS ON COMMUNICATIONS WITH DEPARTMENT STAFF
Respondents shall not communicate with any Department staff concerning this ITN except for the Department
contact person identified in SECTION 4.3 REPLY QUESTIONS & ANSWERS of this ITN. Only those
communications which are in writing from the Bureau of Contracts, Grants, and Procurement Management
Florida Standards Assessments, ITN 2014-47
12
Services shall be considered as a duly authorized response on behalf of the Department. For violation of this
provision, the Department reserves the right to reject a Respondent’s Reply.
Respondents to this solicitation or persons acting on their behalf may not contact, between the release of the
solicitation and the end of the 72-hour period following the agency posting the notice of intended award,
excluding Saturdays, Sundays, and state holidays, any employee or officer of the executive or legislative
branch concerning any aspect of this solicitation, except in writing to the procurement officer or as provided in
the solicitation documents. Violation of this provision may be grounds for rejecting a Reply.
4.8
CONFIDENTIAL, PROPRIETARY, OR TRADE SECRET MATERIAL
The Department takes its public records responsibilities as provided under Chapter 119, F.S., and Article I,
Section 24 of the Florida Constitution, very seriously. If Respondent considers any portion of the
documents, data or records submitted in response to this solicitation to be confidential, trade secret
or otherwise not subject to disclosure pursuant to Chapter 119, F.S., the Florida Constitution or other
authority, Respondent must clearly mark and identify in its Reply those portions which are
confidential, trade secret or otherwise exempt. Respondent must also simultaneously provide the
Department with a separate redacted copy of its Reply. This redacted copy shall contain the Department’s
solicitation name, number, and the name of the Respondent on the cover, and shall be clearly titled “Redacted
Copy.” The Redacted Copy shall be provided to the Department at the same time Respondent submits
its Reply to the solicitation and must only exclude or obliterate those exact portions which are claimed
confidential, proprietary, or trade secret. The Respondent should also provide two (2) electronic
copies (compact disc (CD), flash drive, etc.) of their Redacted Copy.
Respondent shall be responsible for defending its determination that the redacted portions of its Reply are
confidential, trade secret or otherwise not subject to disclosure. Further, Respondent shall protect, defend,
and indemnify the Department for any and all claims arising from or relating to Respondents determination
that the redacted portions of its Reply are confidential, proprietary, trade secret or otherwise not subject to
disclosure.
If Respondent fails to submit a Redacted Copy with its Reply, the Department is authorized to produce the
entire documents, data or records submitted by Respondent in answer to a public records request for these
records.
4.9.
POOR PERFORMANCE NOTICE
The Respondent should provide for both the Respondent and its employees, subcontractors, and
subcontractor employees, copies of any and all documents regarding complaints filed, investigations made,
warning letters or inspection reports issued, any notice of breach, notice of default, termination notice,
suspension notice, or any disciplinary action initiated or taken under any contract or job performance within
the past seven (7) years. For each instance listed, provide a narrative summary of the contract’s purpose and
scope of work, the Respondent’s performance, including the concerns of the project owner, and any major
adverse findings. In addition, provide the contract or job number, the name of the owner, the term of the
contract, the name, address, and telephone number of the owner’s contract manager. Please also include
any relevant documentation evidencing the performance issues.
The Department reserves the right to seek further information on this matter from the Respondent or to make
inquiries with the project owner. The information obtained from this review may be reflected in the
Respondent’s score or used to declare the Respondent not a responsible vendor.
4.10 WITHDRAWAL OF A REPLY
A Respondent may withdraw a Reply by written notice to the Department on or before the deadline specified
for the receipt of Replies in SECTION 3.4 CRITICAL EVENT DATES of this ITN. Such written notice is to be
Florida Standards Assessments, ITN 2014-47
13
submitted to the Issuing Office at the address specified in SECTION 4.3 REPLY QUESTIONS AND
ANSWERS of this ITN.
4.11 DISCLOSURE OF REPLY CONTENTS
All documentation produced as part of this solicitation shall become the exclusive property of the state and
may not be removed by the Respondent or its agents. All Replies shall become the property of the state and
shall not be returned to Respondent. The Department shall have the right to use any or all ideas or
adaptations of the ideas presented in any Reply. Selection or rejection of a Reply shall not affect this right.
4.12 AWARD
As in the best interest of the State, the right is reserved to award based on all or none, groups of services,
or any combination thereof, to a responsive, responsible Respondent. As in the best interest of the state,
the right is reserved to reject any and/or all Replies or to waive any minor irregularity in replies received. In
addition, the Department reserves the right, in its discretion, to correct deviations during the negotiation phase.
Conditions which may cause rejection of Replies include, without limitation, evidence of collusion among
Respondents, obvious lack of experience or expertise to perform the required work, failure to perform, or meet
financial obligations on previous contracts.
SECTION 5 – SPECIAL CONDITIONS
5.0
AUTHORIZED TO DO BUSINESS IN THE STATE OF FLORIDA
Foreign corporations and foreign limited partnerships should be authorized to do business in the state of
Florida. Domestic corporations should be active and in good standing in the State of Florida. Such
authorization and status should be obtained by the Reply due date and time, but in any case, must be
obtained prior to contract execution. For authorization, contact:
Florida Department of State
Tallahassee, Florida 32399
(850) 245-6053
5.1
LICENSED TO CONDUCT SERVICES IN THE STATE OF FLORIDA
If the services being provided require that individuals be licensed by the Florida Department of Business and
Professional Regulation or any other state or federal agency, such licenses should be obtained by the Reply
due date and time, but in any case, must be obtained prior contract execution. For state licensing, contact:
Florida Department of Business and Professional Regulation
Tallahassee, Florida 32399-0797
(850) 487-9501
5.2
OTHER CONDITIONS
Other conditions which may cause rejection of replies include, without limitation, evidence of collusion among
Respondents, obvious lack of experience or expertise to perform the required work, failure to perform or meet
financial obligations on previous contracts.
5.3
IDENTICAL EVALUATION OF REPLIES
Whenever two (2) or more replies which are equal with respect to price, quality, and service are received, the
Department will determine the order of award using the criteria established in 60A-1.011, F.A.C. The "DrugFree Workplace Program Certification" can be found as Attachment 2.
Florida Standards Assessments, ITN 2014-47
14
5.4
DISCLOSURE STATEMENT
The Disclosure Statement Form (Attachment 4) should be signed and submitted with the Reply.
5.5
SUB-CONTRACTING
This contract or any portion thereof shall not be sub-contracted except as permitted herein, with the prior
written approval of the Department. No sub-contract shall, under any circumstances, relieve the Contractor of
its liability and obligation under this contract; and despite any such sub-contracting the Department shall deal
through the Contractor, which shall retain the legal responsibility for performing the Contractor obligations.
The Prime Contractor shall report all Minority Subcontractors, identifying the Name, Address, Type of
Certification and Dollar Amount on the Utilization Summary form, attached as Attachment 5. A list of
subcontractors shall be provided to the Department’s contract manager upon execution of the Contract. The
Prime Contractor shall submit the Utilization Summary form with each invoice submitted for payment. The
form must be submitted with all invoices, regardless if funds have not been spent with a Minority
Subcontractor for the period covered by the invoice. The Office of Supplier Diversity, Florida Department of
Management Services, will assist in furnishing names of qualified minorities. The Office of Supplier Diversity
can be reached at (850) 487-0915; the Internet Web address is
http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd/.
5.6
CONTRACTUAL OBLIGATIONS
The Department’s Contract Standard Terms and Conditions are incorporated in this ITN as Attachment 6 and
will govern the relationship between the Department and the Contractor. A Reply submitted by the successful
Respondent(s) shall be incorporated into the final contract(s). The terms and conditions of this ITN shall
control notwithstanding any statement to the contrary by the Respondent, unless such terms and
conditions are modified by the Department during the negotiation phase.
5.7
METHOD OF PAYMENT
Compensation and payment will be made in accordance with the terms and conditions of the contract.
5.8
CONVICTED VENDOR LIST
A company placed on the Convicted Vendor List may not submit a Reply or be awarded a contract to provide
any goods or services pursuant to Rule 60A-1.006 F.A.C. The “Convicted Vendor List” is published at:
http://www.dms.myflorida.com/business_operations/state_purchasing/vendor_information/convicted_suspen
ded_discriminatory_complaints_vendor_lists/convicted_vendor_list.
5.9
DIVERSITY IN CONTRACTING
The State of Florida is committed to supporting its diverse business industry and population through ensuring
participation by minority-, women-, and service-disabled veteran business enterprises in the economic life of
the state. The state of Florida Mentor Protégé Program connects minority-, women-, and service-disabled
veteran business enterprises with private corporations for business development mentoring. We strongly
encourage firms doing business with the State of Florida to consider this initiative. For more information on
the Mentor Protégé Program, please contact the Office of Supplier Diversity at (850) 487-0915.
The State is dedicated to fostering the continued development and economic growth of small, minority-,
women-, and service-disabled veteran business enterprises. Participation by a diverse group of Vendors
doing business with the state is central to this effort. To this end, it is vital that small, minority-, women-, and
service-disabled veteran business enterprises participate in the State’s procurement process as both
Contractors and sub- contractors in this solicitation. Small, minority-, women-, and service-disabled veteran
business enterprises are strongly encouraged to contribute to this solicitation.
Florida Standards Assessments, ITN 2014-47
15
Information on Certified Minority Business Enterprises (CMBE) and Certified Service-Disabled Veteran
Business Enterprises (CSDVBE) is available from the Office of Supplier Diversity at:
http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd/.
5.10 COSTS INCURRED IN RESPONDING
This ITN does not commit the Department or any other public agency to pay any costs incurred by the
Respondent in the submission of a Reply or to make necessary studies or designs for the preparation thereof,
nor to procure or contract for any articles or services.
5.11 SUBMISSION OF REPLIES BY SUBSIDIARIES OR AFFILIATES
A Respondent, its subsidiaries, affiliates, or related entities is limited to one (1) Reply. Submission of more
than one (1) Reply per activity by a Respondent may cause the rejection of all Replies submitted by the
Respondent. In the alternative, the Department my decide, in its sole discretion, which Reply to
evaluate and consider. A subsidiary or affiliate of a prime Respondent may also be included as a
subcontractor in another Respondent’s Reply.
5.12 PROHIBITION OF GRATUITIES
By submission of a Reply, the Respondent certifies that no elected or appointed official or employee of the
state of Florida has or will benefit financially or materially from this procurement. Any contract arising from this
procurement may be terminated by the Department if it is determined that gratuities of any kind were either
offered to or received by any of the aforementioned officials or employees from the Respondent or its agents
or employees.
5.13 INDEPENDENT PRICE DETERMINATION
A Respondent shall not collude, consult, communicate, or agree with any other Respondent regarding this
procurement as to any matter relating to the Respondent’s Reply.
5.14 FINANCIAL CONSEQUENCES
The Contractor shall submit to the Department for payment the above deliverables and those proposed and
accepted by the Department that are satisfactorily completed in accordance with the terms and conditions set
forth in SECTION 7.0 SCOPE OF SERVICES and SECTION XXXXX DELIVERABLES. Payment shall be
delayed for any deliverables that are not satisfactorily completed until all deficiencies are corrected and the
services have been accepted by the Department. In addition, delays in providing completed deliverables will
subject the Contractor to damages.
5.15 PERFORMANCE BOND
The Contractor shall supply to the Department a Performance Bond in the amount of ten percent (10%) of the
amount of the annual award. The surety shall be in a form acceptable to the Florida Department of Education,
such as a bond, cashier’s check, certified check or money order. A Surety must be authorized to do business
in the state of Florida. The Performance Bond shall be executed and furnished to the Department within ten
(10) calendar days prior to the Contractor beginning work under the contract. The performance bond must
state that it includes coverage of liquidated damages assessed against the contractor. A performance bond is
not required from Florida State Universities.
Cap: For each calendar, contract or fiscal year which corresponds to the period of the performance bond
specified, the cumulative total of liquidated damages for which the Contractor(s) may be liable shall not
exceed, annually, ten percent (10%) of the total contract amount at the time of assessment of liquidated
damages.
Florida Standards Assessments, ITN 2014-47
16
5.16 PARTICIPATION IN FUTURE STAGES OF THIS PROJECT
As stated in Chapter 287.057 F.S.
17(a)c A person who receives a contract that has not been procured pursuant to subsections (1) through (3)
to perform a feasibility study of the potential implementation of a subsequent contract, who
participates in the drafting of a solicitation or who develops a program for future implementation, is
not eligible to contract with the agency for any other contracts dealing with that specific subject
matter, and any firm in which such person has any interest is not eligible to receive such contract.
However, this prohibition does not prevent a vendor who responds to a request for information from
being eligible to contract with an agency.
5.17 ACCESSIBLE ELECTRONIC INFORMATION TECHNOLOGY
Respondents submitting Replies to this solicitation must provide electronic and information technology resources
in complete compliance with the accessibility standards required by Section 282.601-282.606, F.S., and Rule 608.002, F.A.C. These standards establish a minimum level of accessibility.
5.18 SCRUTINIZED COMPANIES LISTS (as applicable)
Section 287.135, Florida Statutes, requires that at the time a company submits a bid or proposal for a contract
for goods or services of $1 million or more, the company must certify that the company is not on either the
Scrutinized Companies with Activities in Sudan List or the Scrutinized Companies with Activities in the Iran
Petroleum Energy Sector List. Both lists are created pursuant to section 215.473, Florida Statutes.
Replies of $1 million or more should include the attached Scrutinized Companies Lists Form (Attachment 6)
to certify the respondent is not on either of those lists. The Form should be submitted with the Technical
Reply.
5.19 LIQUIDATED DAMAGES
Accurate and timely delivery is imperative and, as a result, the contract will include the following provisions for
the completion of the Work Tasks/Activities and Interim Deliverables and Final Deliverables identified in this
ITN.
The Contractor(s)‘s failure to complete Work Tasks/Activities and Interim Deliverables and Final Deliverables
both correctly and on time will result in substantial injury to the Department and the State of Florida, but the
amount of damages resulting from such injury cannot be calculated with certainty. Each such failure to
complete Work Tasks/Activities and Interim Deliverables and Final Deliverables both correctly and on time is
hereinafter referred to as a default. Defaults shall be deemed corrected on the date that the Work
Tasks/Activities and Interim Deliverables and Final Deliverables have been correctly completed. For each
default, the Contractor(s) shall be liable to the Department for liquidated damages and not as a penalty, as set
forth below:
a. Tasks/Activities and Interim Deliverables: For each default on performance and completion of Work
Tasks/Activities and Interim Deliverables (see SECTION 7.8), it is agreed that for each and every day
work remains uncompleted beyond the time set for its completion the Contractor(s) shall be liable to the
Department in a specified amount of three thousand dollars ($5,000.00) per business day as liquidated
damages for such delay, and not as a penalty, from the date of notice by the Department until the default
has been corrected.
THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK
Florida Standards Assessments, ITN 2014-47
17
b. Final Deliverables: For each default on completion of Final Deliverables (see SECTION 7.8), the
Contractor(s) shall be liable to the Department in a specified amount per calendar day from the date of
notice by the Department until the default has been corrected. The specified amount of damages is
defined below:
Schedule of Liquidated Damages
For Final Deliverables
Number of business days
Liquidated Damages
elapsed after the default
First
$5,000.00
Second
$15,000.00
Third
$30,000.00
Fourth
$45,000.00
Fifth
$60,000.00
Sixth
$75,000.00
Seventh/ subsequent days
$125,000.00 per day
c. Cap: For each calendar, contract, or fiscal year which corresponds to the period of the performance bond
specified in ITN Section 5.15, the cumulative total of liquidated damages for which the contractor may be
liable shall not exceed, annually, ten percent (10%) of the total contract amount at the time of assessment
of liquidated damages.
d. Collection: At its option, the Department shall collect liquidated damages by making claims against the
performance bond, from time to time, until the bond has been exhausted or by deducting the liquidated
damages from contract payments to the contractor. In the alternative, the Contractor may make payment
to the Department in the amount of such liquidated damages.
e. Deadline Adjustments:
(1) In the event the contractor wishes to propose a change of deadline for a Critical Work Task, the
contractor may propose the change to the Department, in writing. The Department has no obligation
to agree to the change, but shall notify the contractor, in writing, as to whether or not it agrees to the
proposed change. If the Department agrees, the modified deadline shall become the effective
deadline for purposes of assessing liquidated damages.
(2) In the event the Department wishes to change a deadline for an Extremely Critical Work Task to a
date which shortens the contractor’s time for meeting the deadline, it shall notify the contractor of the
new deadline in writing in the form of a contract amendment. If the contractor determines that by
exercise of every reasonable effort, the contractor will be unable to meet the new deadline, the
contractor shall offer a proposed shortened deadline which the contractor can meet by the exercise of
every reasonable effort.
(3) In the alternative, the contractor may propose a contract amendment in which the additional
personnel hours needed to achieve the shortened deadline are compensated by a price adjustment.
(4) If the parties are unable to reach agreement, the Department shall determine the deadline. However,
the issue of whether liquidated damages apply to the contractor’s failure to meet the shortened
deadline shall be reached as follows. The parties shall submit to binding arbitration the issue of what
date (earlier than the original deadline) the contractor could have completed the Extremely Critical
Work Task on time and correctly by exercising every reasonable effort. The date determined through
arbitration shall be the deadline for purposes of assessing liquidated damages. Arbitration shall be
conducted pursuant to the procedures of the American Arbitration Association, subject to the Florida
Arbitration Code, Chapter 682, F.S.
Florida Standards Assessments, ITN 2014-47
18
f.
Waiver: The Department, in its sole discretion, may waive the imposition of liquidated damages or a
portion thereof in a given instance. Such waiver, in any instance, shall not constitute a waiver in any
future instance, nor establish any right on behalf of the contractor to a waiver.
g. Department Approval: In situations where in the contractor must obtain the Department’s approval of an
activity or product before the contractor can complete a Critical Work Task, the contractor shall be
responsible for providing a reasonable time for the Department to complete its review and for the
contractor to correct any deficiencies. In the event the Department’s review is not conducted within a
reasonable time, as described in Section 7.7.4, that shall be grounds for the contractor to propose a
deadline adjustment pursuant to Subsection 5.19(e)(1), above. The parties’ contract managers may
agree, in writing, in advance, as to the reasonable time for the Department’s review of a specific activity or
product.
SECTION 6 – SPECIAL INSTRUCTIONS – REPLY FORMAT & CONTENT
6.0
REPLY SUBMISSION
Both Technical and Price Replies will be received as specified in SECTION 3.4 CRITICAL EVENT DATES.
All Replies and associated forms should be signed and dated in ink by a duly authorized representative of the
Respondent. The overall Reply should be written in a concise manner, which is conducive to effective
evaluation and selection.
All Replies and related documents submitted in response to this ITN shall become the property of the state.
6.1
MAIL OR DELIVER REPLIES TO: (Do Not Fax or E-Mail)
Florida Department of Education
Bureau of Contracts, Grants and Procurement Management Services
Attn: Fran Shewan
325 West Gaines Street
332 Turlington Building
Tallahassee, Florida 32399-0400
6.2
REPLY FORMAT INSTRUCTIONS
This section contains instructions that describe the expected format for the Reply. All Replies submitted
should contain two parts and be marked as follows:
PART I TECHNICAL REPLY NUMBER ITN 2014-47
(One Separately Sealed Package for Technical)
PART II PRICE REPLY NUMBER ITN 2014-47
(One Separately Sealed Package for Prices)
THE SEPARATELY SEALED PACKAGES MAY BE MAILED TOGETHER IN ONE ENVELOPE OR BOX.
6.2.1 Preliminary Administrative Review
The absence of any of these documents may result in a determination that the Reply is nonresponsive and the Reply shall not be evaluated. The Reply forms furnished must be used when
submitting the Reply. Forms are to be filled out in ink or typewritten. Administrative Review information
shall be included in the Technical Reply portion (Part I) of the Reply and consist of the following:
All Addenda
DUNS Number
Florida Standards Assessments, ITN 2014-47
19
Disclosure Statement Form (Attachment 3)
Reference Form (Attachment 4) – Provide at least three (3) references, which demonstrate efforts
comparable to the one described in this ITN. The Department reserves the right to contact the references
regarding the services provided. Any information provided will be subject to the requirements of the
Florida Public Records Law.
Transmittal Letter – Provide a Transmittal Letter (on Company Letterhead) that contains the following:
a statement confirming that the person signing the Reply is authorized to represent the Respondent
and bind the Respondent relative to all matters contained in the Respondent's Reply
the company’s federal tax identification number
the company’s DUNS number
a statement confirming that the Respondent has read, understands, comply and agrees to all
provisions of this ITN, unless changed during the negotiation process. By submitting a Reply,
the Respondent agrees to the terms and conditions of the ITN notwithstanding any statement
in the Reply to the contrary. However, the Department will attempt to resolve any
disagreements in the negotiation sessions.
a statement confirming that the Respondent is authorized to conduct business in Florida in
accordance with the provisions of Chapter 607, F.S. In lieu of such statement, the Respondent
alternatively should confirm that authorization to do business in Florida will be secured prior to the
award of the contract.
a statement confirming that the Respondent is registered on the MyFloridaMarketPlace website in
accordance with the provisions by the State of Florida. In lieu of such statement, the Respondent
should alternatively confirm that registration authorization will be completed prior to the award of the
contract.
a statement confirming that the Respondent has electronically registered a valid W-9 with the
Department of Financial Services (DFS). DFS is ready to assist any vendors with questions, and
vendors must submit their W-9 forms electronically at https://flvendor.myfloridacfo.com. Contact the
DFS Customer Service Desk at (850) 413-5519 or FLW9@myfloridacfo.com with any questions. In
lieu of such statement, the Respondent should alternatively confirm that registration will be completed
prior to the award of the contract.
6.2.2 Technical Reply (Part I) (20 hard copies)
(Do not include price information in Part I)
The Respondent should submit one (1) original in hard copy, nineteen (19) hard copies and twenty (20) in
electronic format (compact disc (CD), flash drive, etc.) copies in Microsoft Word 5.0 or higher, or
Adobe Acrobat of the Reply which is to be divided into the sections described below. The Department
expects all Replies to be in this format. Failure of the Respondent to follow this outline may result in the
rejection of the Reply, or result in a lower evaluation score. Any rejection of a Reply, or other adverse affect
on a Respondent’s Reply, due to its failure to properly organize its Reply shall be at the sole risk of the
Respondent. The Technical Reply should be submitted in a separate sealed package marked "TECHNICAL
REPLY FOR ITN 2014-47".
1. EXECUTIVE SUMMARY
The Executive Summary should be written in non-technical language to summarize the Respondent’s
overall capabilities and approaches for accomplishing the services specified herein. Evidence should
include, but not be limited to, respondent’s mission, date of founding, size, and experience. This section
should also demonstrate the Respondent’s clear understanding of the Department’s goal to securely and
reliably develop, administer, score, and report Florida’s assessments in an efficient and cost-effective
manner. The Executive Summary should be limited to no more than six (6) pages.
Florida Standards Assessments, ITN 2014-47
20
2. QUALIFICATIONS AND EXPERIENCE
The respondent should provide a description of qualifications and prior experience performing tasks
similar to those required in this ITN. The discussion should include a description of the respondent’s
background and relevant experience that qualifies it to provide the products and services required by this
ITN.
The respondent should document the contracted services for previous assessment projects similar to the
one described in this ITN and any other projects the respondent believes will document its corporate
capability. For each, the documentation should include a description of the services and products
delivered; the contract period; and the name, address, and telephone number of a contact person for each
of the contracting agencies.
If subcontractors are to be used for any portion of the work activities, the respondent should name the
subcontractor(s) and document the experience and qualifications of the subcontractor(s) in performing
tasks identical or similar to those they will be asked to do. A separate chart in the proposal may be used to
identify all of the subcontractors proposed to be involved in the project and the services they are expected
to provide. The respondent also should provide examples of materials that demonstrate the quality of the
work done by the respondent on similar projects.
The proposal should include information sufficiently descriptive to permit the proposal evaluation
committee to consider the experience and capability of the prime contractor as well as the subcontractors.
As part of the proposal evaluation, the proposal evaluation committee will contact one or more of the
agencies for whom the respondent previously had been engaged to obtain information and
recommendations. Committee members may also depend upon and use their own personal knowledge of
the respondent, the subcontractors, and the proposed personnel as they evaluate each proposal.
3. TECHNICAL PLAN
The technical plan should provide a description of the respondent’s proposal to meet the requirements of
this ITN. This discussion should encompass all of the requirements of Section 7.1.0 through Section
7.6.3.6.3 and associated deliverables of this ITN and should be organized and numbered in accordance
with the organization and numbering system used in Sections Section 7.1.0 through Section 7.6.3.6.3.
The technical plan should describe the methodologies, quality management standards and best practices
that will be employed in managing and completing the requirements. The descriptions should be
thorough, clear, yet as concise as possible. The Respondent should also identify where significant
difficulties may be anticipated and describe any specific techniques proposed to be used to resolve the
difficulties.
TECHNICAL APPROACH
The Respondent should explain the approach, capabilities, and means to be used in accomplishing the tasks
in the Scope of Services, and where significant development difficulties may be anticipated and resolved. Any
specific techniques to be used should also be addressed.
TECHNICAL APPROACH ORGANIZATION OF THE INVITATION TO NEGOTIATE
Section 7.1 of the ITN states the purpose of the ITN; describes the current Florida statewide assessment
program and modifications to the program that will be implemented under the ITN; and provides information
for preparing replies.
Sections 7.2 through 7.7 describe the scope of work for the years covered by this ITN. References are
made in Sections 7.2 through 7.7 to the appendices, which provide detailed information about printed
products, reports, shipments, and other elements of Florida’s assessment program. Each section includes a
description of products and services to be provided.
Florida Standards Assessments, ITN 2014-47
21
Section 7.2.0: Program Descriptions. An overview of the standards-based assessments requiring services
described in this ITN.
Section 7.3.0: Test Development. A description of the test development tasks from item writing through the
production of final test forms.
Section 7.4.0: Test Administration. A description of the test administration tasks, including the delivery and
return of test materials, the production of test administration ancillary materials, providing customer service
and technical support, needs for the online management system, and administering computer-based
assessments.
Section 7.5.0: Scanning, Scoring, and Reporting. A description of the scanning, scoring, and reporting
tasks from preparing school and student information through the reporting of results. The psychometric
aspects of scoring are covered in Section 7.6.0.
Section 7.6.0: Scaling, Equating, Scoring, and Special Psychometric Studies. A description of the
scaling, equating, scoring, and special psychometric studies and tasks.
Section 7.7.0: Program Management. A description of how the work of the contract should be managed,
including meetings, staff, and reimbursable funding categories.
SYSTEM DEMONSTRATION (ELECTRONIC DEMONSTRATION)
The Respondent should submit two (2) copies with the Technical Reply, in electronic format (compact
disc (CD), DVD, flash drive, etc.) a demonstration (demo) of the proposed system. The system demo
provided should not exceed 25 slides or 30 minutes of presentation.
Do not include pricing information with the Technical Reply or the demo. Any inclusion of prices/discounts in
the Technical Reply or demo may be grounds for deeming the Respondent’s Reply nonresponsive.
After the Evaluation Phase is completed, the Department will notify the Respondents invited to participate in
Negotiations. Only those Respondents entering into Negotiations will be allowed and required to present an
on-site oral presentation of the proposed operational system. The Respondent’s on-site presentation will be
conducted during the Negotiation meeting. The actual date, time and duration of Negotiation
meetings/presentations are to be determined.
4. RESPONDENT'S MANAGEMENT PLAN
The management plan should describe the organization and document its capacity to provide the
products and services required by this ITN. This discussion should specifically encompass the
requirements of Section 7.7.0 through Section 7.7.10 and associated deliverables of this ITN and should
be organized and numbered in accordance with the organization and numbering system used in Section
7.7.0 through Section 7.7.10. The descriptions should be complete, clear, and concise.
The management plan should identify by name the managers and professional personnel responsible for
this project. Charts showing the organizational structure and the incorporation of proposed project staff
into the structure should be included, as well as the amount of time in FTE person-days that each person
will devote to the major tasks and activities.
The Respondent should include a description of the methodology to be used to manage the project work,
including control of costs, assuring quality, identifying and minimizing risk, and maintaining schedules;
Vitae should be included as an appendix to the proposal and should include relevant educational
background and professional experience. Personnel assigned by the contractor should demonstrate
qualifications and competencies of the position to which assigned.
It is assumed that the contractor may use outside printers for some materials, such as preprinted forms.
Printers will be documented as subcontractors, and the management plan should identify the proportion of
Florida Standards Assessments, ITN 2014-47
22
materials to be printed by the contractor and by outside vendors. Procedures for quality control and
security during printing should be described.
6.2.3 Price Reply (Part II) (20 hard copies)
The Respondent should submit one (1) original hard copy, nineteen (19) hard copies and twenty (20)
electronic format (compact disc (CD), flash drive, etc.) copies in Microsoft Excel 5.0 or higher. The
Respondent’s price information should be submitted on the form provided in this ITN. Prices submitted in the
Price Reply form, Attachment 1, will be used to evaluate the price component of the evaluation. Payment for
services provided under the contract will be made based on the deliverable schedule submitted, Section 7.8,
The Price Reply and deliverable schedule should be submitted together in a separate sealed package marked
"PRICE REPLY FOR ITN 2014-47".
6.2.4 Presenting the Reply
The Reply should be limited to a page size of eight and one-half by eleven inches (8½" x 11"). Type size
should not be less than a 12 point font. The Reply shall contain a table of contents, be typed single-spaced
and have separate parts, each clearly labeled including page numbers. The information to be addressed in
each part is described in the above sections. The absence of information or the organization of information in
a manner inconsistent with the terms of this ITN may result in the rejection of the Reply, or a reduced
evaluation score. Bindings and covers will be at the Respondent's discretion; however, elaborate
notebooks/hard back binders are discouraged.
Unnecessarily elaborate special brochures, art work, expensive paper and expensive visual and other
presentation aids are neither necessary nor desired.
All Respondent materials should be packaged so that each box of materials shipped to the Department does
not exceed 25 pounds.
SECTION 7 – SCOPE OF SERVICES
7.0
SCOPE OF SERVICES
7.1.0 Introduction
This ITN addresses four broad assessment components as described below. Respondents may submit bids
on any individual assessment component (A, B, C or D), any group of assessment components, or to provide
all assessment components (A, B, C and D):
A. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 3-8
B. Computer-based and paper-based assessments in Mathematics in Grades 3-8
C. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 9-11
D. Computer-based and paper-based End–of-Course (EOC) assessments in Algebra 1, Geometry, and
Algebra 2
The Department will continue to phase in computer-based assessments in Grades 3-8, transitioning
completely to computer-based by 2017-2018 according to the preferred schedule in Appendix A. However,
depending on the final award for the individual assessments, the Department may vary when tests will be
administered as computer-based or paper-based. The final determination of which tests will be computerbased or paper-based is at the Department’s discretion.
This ITN will specify a complete package of products and services so that the FSA will be developed,
administered, scored, and reported separately from other assessments aligned to Florida’s Next Generation
Sunshine State Standards under a contract currently held by NCS Pearson. This ITN does not include the
Florida Standards Assessments, ITN 2014-47
23
Science, Social Studies, FCAT 2.0 Reading Retake, and Florida EOC Assessments aligned to the Next
Generation Sunshine State Standards that will be covered in a separate procurement.
Although this ITN may result in the award of multiple contracts, the term “Contractor” is used throughout and
refers to each contractor providing the required products and services for each grade and subject for which the
contract is awarded.
7.1.1 Notice to Respondents
1. This ITN provides for implementation of Florida’s standards-based assessment program. The contract will
include development, psychometric services, administration, scoring, and reporting of Florida Standards
Assessments in English Language Arts/Literacy and Mathematics for the test administration cycles for
2014-2015 through 2016-17 with possible extension of services in three one-year renewal periods, 201718 through 2019-2020. Replies will be evaluated on the basis of the respondent’s qualifications and
experience and the technical and cost replies for the tasks described herein for the entire span of the
ensuing contract.
2. This ITN defines the requirements for implementing Florida’s standards-based assessment program. The
ITN and the selected contractor’s reply, together with clarifying documents, define the work to be
conducted under contract. These documents will be incorporated into the contract resulting from the state
standards-based assessment program project award. Because Florida’s standards-based assessment
program is technical and complex, it is possible that a responsive reply may not totally or clearly reflect
ITN requirements in all details. If the reply of a respondent selected as a result of the bidding process is
inconsistent with the ITN, the requirements of the ITN will prevail; the selected contractor will be expected
to perform all ITN requirements without an increase in cost above the proposed cost. Furthermore, by
submitting a bid the respondent is agreeing to be bound by the terms of the ITN notwithstanding any
language in the reply to the contrary. Any discrepancies between the terms of the ITN and the reply will
be resolved in the negotiation process at the Department’s discretion, and when it is in the state’s best
interest.
3. The Department reserves the right to substitute or modify tasks on a generally equal basis. Substitutions
or modifications will be made prior to the selected contractor's beginning significant efforts to complete the
task. All design changes, substitutions, or modifications will be given to the contractor in writing. It is not
the intent of this provision to increase the scope of work of the contract but to protect the Department as
design changes to improve the assessment become known during the implementation of development
and administration activities. The Department also reserves the right to amend the contract resulting from
this bid competition to add, modify, or delete designated tasks, products, and/or services related to the
core activities of this project.
4. In developing the solution pursuant to this procurement, a respondent may propose to provide a custom built
solution, a solution comprised of pre-existing, proprietary materials or products that are the intellectual
property of the respondent or third parties, or a combination thereof. A respondent may propose that all
intellectual property developed be owned solely by the Department, owned jointly by the Department and the
respondent, or, at a minimum, that the Department be granted a perpetual and non-exclusive license for the
use, reproduction and modification of the solution for use in Florida. The Department reserves the right to
request alternative intellectual property arrangements or combinations of such arrangements during
negotiations with respondents, in order to obtain a reply that provides the best value to the state. All
permissions or licenses required for the use of materials or products that are the intellectual property of a third
party must be obtained by the respondent. All materials, products, and licenses must be transferred to the
Department at the end of the contract period such that the Department can maintain and sustain the solution
beyond the end of the contract period.
5. The selected contractor will work closely with the Department and with groups of Florida educators to
complete all tasks. All procedures, processes, and products used by the contractor to complete contract
work must be approved by the Department. Throughout the contract period, the contractor will confer with
Florida Standards Assessments, ITN 2014-47
24
the Department on a continuing and consistent basis and will be involved in frequent face-to-face
meetings with the Department, as necessary.
6. Several reimbursable funding categories identified in Section 7.7.9 are set aside for specific functions
during the course of the project. Expenditures in these categories will be reconciled on a quarterly basis.
7. The dates for reporting assessment results necessitate a restricted time period to complete the work of
scoring activities. Respondents are expected to propose efficient and innovative procedures to meet
these required reporting dates and are expected to demonstrate in their replies that they have the
capability and are willing to commit the human and technological resources needed to complete the
scanning, student response file processing, scoring, and reporting tasks in the required time period.
8. The fundamental purpose of Florida’s standards-based assessment program is to provide accurate
information on student achievement. The contractor will utilize every means required to ensure that
information created by the project is accurate. The selected contractor will be responsible for correcting
at the contractor’s expense any errors arising from activities that are the responsibility of the contractor.
This may involve activities such as conducting analyses to identify the cause and extent of errors;
reprogramming or reproducing products or other materials; replacing data files; reproducing reports; and
shipping replacement products or reports to the Department or districts using expedited shipping services.
9. News releases or public announcements pertaining to this ITN, a contract award resulting from
the ITN, or work completed under a contract will not be made without prior written approval from
the Department.
7.1.2 Project Schedule
The deliverables schedule presents a list of major project activities and the dates for completion where these
can be determined. Many of the deliverable dates are contingent upon finalized administration dates and will
be determined during the negotiation process. Appendix B indicates critical and extremely critical tasks and
provides general guidelines for the timeframes for completion, but these are also contingent on finalized
administration dates. The dates provided in 7.2.2 are subject to change during the negotiation phase.
Respondents should provide a detailed schedule that incorporates the activities included in this ITN for the
entire life of the contract. The schedule will indicate essential steps for the completion of project tasks,
initiation and completion dates, and task responsibility. Tasks and responsibilities will include an explanation
of work effort required and full-time equivalent staff (FTE) committed to this work. The reply should also
provide total annual FTE and identify individuals and/or staff roles that will work full-time on this
project. Respondents should be aware that the schedules for each project year overlap. To accurately
estimate required annual resources, schedules for preceding and succeeding years need to be considered.
Following the contract award, the schedule will be further developed into a working document that will be
updated weekly, indicate all task completion dates, and be revised as needed with Department approval (See
Section 7.7.6). A draft work schedule, based on a sample provided by the Department, will be prepared by the
selected contractor for submission to the Department one week prior to the initial planning meeting, which will
be held no later than one month after the execution of a contract for the project. The schedule will be
submitted in hard copy and electronic format using Department-approved spreadsheet software that permits
sorting of the file by start or end date and major task and subtask groupings.
7.2.0 Program Descriptions
The selected contractor will provide the Department with a criterion-referenced achievement test, aligned to
and measuring mastery of the Florida Standards in English Language Arts/Literacy (ELA/L) and Mathematics
proficiency for administration to Florida public school students. For purposes of responses to this ITN, the
Florida Standards are to be considered those adopted by the Florida State Board of Education in July, 2010.
Any changes to the standards following additional Florida public review scheduled for fall 2013, and resulting
changes to the assessment will be addressed through the amendment of the contract that may be awarded as
a result of this ITN.
Florida Standards Assessments, ITN 2014-47
25
The primary purpose of the assessment is to provide student academic achievement and learning gains data
to students, parents, teachers, school administrators, and school district staff. This data is to be used by
districts to improve instruction; by students, parents, and teachers to guide learning objectives; by education
researchers to assess data; and by the public to assess the cost benefit of the expenditure of taxpayer dollars.
The assessment must be designed to:
1. Assess the achievement level and annual learning gains of each student in ELA/L and Mathematics.
2. Provide data for making decisions regarding school accountability, recognition, and improvement of
operations and management, including schools operating for the purpose of providing educational
services to youth in Department of Juvenile Justice programs.
3. Identify the educational strengths and needs of students and the readiness of students to be promoted to
the next grade level, to graduate from high school, and to succeed in college and in careers.
4. Assess how well educational goals and curricular standards are met at the school, district, and state level,
and at the national level, to the extent possible.
7.2.1 Expected number of districts, schools, and students
The expected number of students, shipping destinations, and schools are provided in Appendix C as a basis
for preparing replies. Respondents should calculate the quantities of materials needed to implement the test
administrations on the basis of these estimates. If the total number of tests administered across all grades
and test administrations exceeds the number of tests described in this appendix by more than ten percent
(10%), the Department will execute a formal change to the contractual agreement.
Appendix C also lists the 67 school districts and approximately 13 special districts. For the school districts and
the special districts combined, the contractor will ship testing materials to approximately 80 separate
destinations. In the 67 school districts and the special districts combined, approximately 4,000 separate
administration sites (including private schools, community colleges, etc.) will be involved in the test
administrations, and the contractor will expect that number to increase by as much as five percent (5%)
annually, calculated based on the previous year. Some of this increase is due to legislative changes that
increase the number of students participating in Florida’s statewide assessment program. If the increase of
administration sites is beyond the anticipated five percent (5%), the incremental cost increase will be handled
in an amendment to the contract.
7.2.2 Potential Test Administration Dates
Potential projected starting dates for future assessment administrations relevant to this ITN are shown in Table
7.2.2 below for illustrative purposes only. The respondent’s reply should propose starting dates for spring
administrations that occur as late in the school year as possible. Replies should take into account the scoring
of constructed- and extended-response items, data verification activities, and reports of results by the required
dates (Appendix B).
THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK
Florida Standards Assessments, ITN 2014-47
26
The FSA EOC Assessment administrations will be at the same time as the Florida EOC Assessment
administrations. These dates are also provided, for reference, in Appendix A.
Table 7.2.2 EXAMPLE: Potential Test Administration Dates for Florida’s Standard-Based Assessments
Table 7.2.2 EXAMPLE: Potential Test Administration Dates for Florida’s Standard-Based Assessments
Year
Grades 3-11 ELA/L
Grades 3-8 Mathematics
Grade 10 ELA/L Retake
2014-15
March 2-13, 2015
N/A
2015-16
February 29-March 11, 2016
October 12-23, 2015
February 29-March 11, 2016
2016-17
March 6-17, 2017
October 10-21, 2016
March 6-17, 2017
2017-181
March 5-16, 2018
October 9-20, 2017
March 5-16, 2018
2018-191
March 4-15, 2019
October 8-19, 2018
March 4-15, 2019
2019-201
March 2-13, 2020
October 7-18, 2019
March 2-13, 2020
1 Optional
Renewal Period
Algebra 1 EOC
Geometry EOC
Algebra 2 EOC
April 7-25, 2015
July 13-24, 2015
September 14-25, 2015
November 30-December 18, 2015
April 4-22, 2016
July 11-22, 2016
September 12-23, 2016
November 30-December 21, 2016
April 3-21, 2017
July 10-21, 2017
September 11-22, 2017
December 4-22, 2017
April 2-20, 2018
July 16-27, 2018
September 10-21, 2018
December 3-21, 2018
April 8-26, 2019
July 15-26, 2019
September 9-20, 2019
December 2-20, 2019
April 6-24, 2020
July 13-24, 2020
7.3.0 Test Design and Development
All services related to test design, item development and review, item banking, and test production for the FSA
are the responsibility of the contractor selected under this ITN. Respondents may propose a customized
assessment, or products and services derived from existing products developed and published by the
contractor, or developed as part of other initiatives. It is understood that by the time the contract is executed
for the services requested in this ITN, test items may have already been developed for assessments that will
be operational in 2014-2015. However, the Department expects to be closely involved in the development and
review of new test items to be field tested and developed in subsequent years. To the extent possible, the
Department also expects to be involved in any test construction activities for 2015 assessments still to be
completed after contract execution. The Department’s primary goal with respect to test design is ensuring that
the assessment measures student mastery of the Florida Standards, and accomplishes this in the most costeffective way without significantly increasing testing time beyond that of current Florida assessments. A
sample test design is provided in Appendix D.
The Department has established a Test Development Center (TDC), staffed with editors and Florida
educators, to provide quality assurance for all test content and related products, whether customized or
derived from existing products. The selected contractor will work closely with the Department and TDC staff to
ensure that the test philosophy, design, item specifications, and product quality are aligned with Florida’s
expectations. The Department and TDC will also coordinate closely with the selected contractor to conduct
Florida Standards Assessments, ITN 2014-47
27
rangefinding meetings with educators in order to generate training materials for the scoring of performance
tasks. While the selected contractor will be required to coordinate all meeting activities and materials, the
Department and TDC will identify invitees and assist in facilitating all meetings. Required meetings are
referenced throughout the ITN. See Section 7.7.7 for a summary of details about educator/citizen meetings.
Test development procedures should include review and revision of proposed test and item specifications,
review of test blueprints, review and evaluation of items available for use on the test (item development plan),
field testing for needed items to complete the desired test design, and final operational test creation. All
phases of test development shall use accepted validity, reliability and other testing principles including
Universal Test Design. A plan for complete data review of field-tested items should be delineated. The
Department must approve the items and the test forms before they become operational. Respondents who
propose solutions derived from existing products that are used by other states must take into consideration the
requirement for including state educators in the quality control aspect of item development and scoring of
performance tasks. The extent of educator involvement will be discussed during negotiations and agreed
upon prior to contract execution.
To the greatest extent possible, the Department is seeking a solution that provides comparability of results
with other states. Such comparability could be obtained by a number of acceptable means. The assessment
could be one that is also administered in other states, or the design could include embedding of a small
number of norm-referenced items within each criterion-referenced test form. A design that uses embedded
norm-referenced items would not be expected to yield student-level comparative data, rather in this case, the
Department would expect reporting at an aggregate level. The final design will be discussed during
negotiations and agreed upon prior to contract execution.
7.3.1 Test Content
1. The FSA will consist solely of test items as described in the previous section. If an existing “shelf product”
test includes additional components beyond those described above, it will be necessary to remove these
items.
2. If the test being proposed is to be derived from an existing product, respondents should:
a. Describe the general content being measured by the proposed test.
b. Provide full-length inspection copies of the test and test administration manual for each grade level.
c. Provide a test blueprint that governs construction of the test, including the number of items per
objective and reporting category.
d. Describe the objectives being measured by the test.
e. Describe the taxonomic classification system used to classify each test item and the proportional
assignment of items within the taxonomic classification system.
f.
Describe the range of content difficulty included on each level of the test including the extent to which
the items at each level assess content that may be difficult for some students.
g. Describe the method by which text complexity and grade-level appropriateness was determined for
reading passages
h. Describe how grade-level appropriateness of item content for each level of the test was determined
i.
Describe the extent to which the items included at each level assess higher order thinking skills. If
the taxonomic classification system (See e. above) includes a thinking skills dimension, please
include this information in that description.
j.
Describe the manner in which the test is configured into subparts for both test administration and
score reporting purposes, including how reporting categories are related to the taxonomic
classification system.
Florida Standards Assessments, ITN 2014-47
28
k. Provide an indication of the minutes of testing time required for the separate content sections.
3. The respondent should describe the special materials or manipulatives required or permitted for correct
administration of the proposed test (e.g., online rulers or Department-approved calculators). Detailed
descriptions of or access to each required item should be provided with the technical reply.
4. Reading passages should expose students to grade-level text of appropriate complexity. Texts should
reflect a balance of authentic literature (stories, drama, and poetry) and informational text/literary
nonfiction, which would include passages that address the literacy standards requirement for history/
social studies, science, and technical subjects that students are expected to read. Selections should
include a combination of single texts and paired passages of sufficient complexity for close, analytical
reading which would allow students to discern and locate evidence from the text to support assertions in
multiple-choice items or writing responses. Respondents may propose passages from three sources,
including public domain, permissioned, and commissioned sources. Solely for cost comparison purposes,
respondents may assume a test design that includes 1/3 of each passage type, but respondents may
propose solutions that best meet the primary requirements of the ITN. The final test design will determine
the proportion and genre percentages for passages.
5. English Language Arts/Literacy items aligning to each passage may include multiple-choice items as well
as short, constructed responses to text-dependent questions. One extended response item should be
drawn from a paired or multiple-text passage set. Extended writing prompts should allow students to
produce opinions/arguments, informative/explanatory responses, and narratives.
6. The first tests to be administered by the selected contractor under this ITN will be the spring 2015 ELA/L
and Mathematics tests in Grades 3 – high school.
a. The selected contractor will be required to provide stand-alone computer-based calculator
applications for 4-function, scientific, or graphing calculator use on computer-based tests, depending
on final test design. These computer-based calculator applications will be made available to districts
and schools for downloading to local computers and for online access and practice throughout the
year.
b. For any Florida test produced for students, the selected contractor will provide braille, large-print,
regular-print, and one-item-per-page versions at each grade level for students with disabilities. The
contractor must also provide computer-based accommodations for students with various disabilities,
as described in Section 7.3.11.2. A demonstration of these accommodations may be included in the
respondent’s presentation.
c. The selected contractor will also print and ship Braille and large print sample test materials upon
district request for students who will require these paper-based accommodations. These sample test
materials are considered “generic,” meaning once they are created by the contractor, there is no
expectation that they will be updated during the life of this contract.
7.3.2 Test Configuration
This section provides information on the configuration for all test documents and related materials for both
print and computer test delivery.
7.3.2.1 Test and Answer Document Design
Respondents should describe the proposed document design and layout for each grade and
subject test, for computer-based and paper-based tests, as described for each year in Appendix A.
The selected contractor designs and produces camera-ready art for the test forms that are used to
print test books and answer books or prepare computer-based test items in delivery-ready format.
Contractors are expected to provide the necessary human and other resources necessary to meet
extremely critical deadlines for delivery of documents and the computer-based test system and
files.
Florida Standards Assessments, ITN 2014-47
29
Respondents have been requested to submit bids for a paper-based and computer-based format
for each assessment. However, even if a computer-based assessment is the accepted solution,
any tests offered on the computer must also be prepared for use in a paper-based format (Braille,
large print, regular print, one-item-per-page) as noted in these tables. In this instance, the paperbased format will only be offered by special request for up to five percent (5%) of the students and
can be used at the Department’s discretion.
To the extent that a respondent’s design solution allows for some customization, the Department
will consider replies that include:
1. Security-controlled booklets containing only FSA Reading passages and no test items for the
computer-based tests to be available by special request for up to five percent (5%) of the
students. See Section 7.3.11.1.
2. Blank work folders for all students taking the FSA Mathematics computer-based tests. The
work folder consists of four pages for students to do scratch work, the fourth page containing a
full-page grid.
3. Blank worksheets for all students taking the FSA ELA/L computer-based tests. The worksheet
consists of two pages for students to make notes.
4. A copy of the appropriate reference sheet, if proposed by the respondent and accepted by the
Department, may also be provided for students taking the assessment for which it is proposed.
5. Work folders, worksheets, and reference sheets/periodic tables are not numbered or
inventoried by the contractor; they are destroyed by the school after results arrive.
For respondents who do not offer such customization, replies should address how test design and
presentation eliminate the need for these considerations.
The Department may choose to modify the design of any state-specific test documents/files prior to
any test administration, and reserves the right to change this configuration beyond the
specifications but within the scope of the contract, if necessary, through change orders or contract
amendments. Appendix E provides additional details concerning these materials.
7.3.3 Develop Items
For items developed after contract execution, it is the responsibility of the selected contractor to ensure that
the quantity of items developed and submitted for review annually is sufficient to create operational test forms
for each administration. Respondents should propose item and passage counts by item type, standard or
cluster, as appropriate, needed to be developed annually in order to meet these needs. The contractor must
identify and recruit a sufficient number of highly-qualified writers for passages and items to ensure variety and
quality of development. Resumes for these writers should be provided and approved by the Department prior
to contract execution. Note that no active Florida educators will be approved as item writers.
The contractor must conduct in-depth training of passage/item writers at least annually. Department content
experts will attend the passage/item writer training to provide input and oversight for the process.
The selected contractor will be responsible for internal reviews to ensure item clarity, content accuracy,
adherence to Test Item Specifications, psychometric properties, difficulty, readability requirements, and
potential bias or insensitivity. It is unacceptable to submit to the Department proposed test items at any time
that contain blatant errors, such as violation of Test Item Specifications, spelling errors, or grammar mistakes.
The Department will not review any submissions with these types of obvious errors even if the impact creates
schedule challenges for the contractor, for which the contractor will be responsible.
7.3.4 Review Meetings
For test items developed for use on assessments following the spring 2015 administration, the selected
contractor will be responsible for organizing and convening meetings for the purpose of reviewing assessment
Florida Standards Assessments, ITN 2014-47
30
items for content and bias and community sensitivity issues by Department-appointed review committees
composed of Florida educators and other citizens. A detailed list of meetings to be conducted under this
contract and the funding mechanism for these meetings are provided in Sections 7.7.7 and 7.7.9 of the ITN.
For each meeting, the selected contractor will be required to maintain original forms of all committee member
test security and non-disclosure agreements. These will be made available to the Department on an asneeded basis.
The meetings will be held in a Florida location that is easily accessible (hub cities) to the majority of
participants and capable of handling the logistical requirements of a large-scale, multi-committee meeting.
The selected contractor will prepare assessment items and all other materials required to conduct the
meetings. Separate content review meetings will be conducted for each subject/grade combination.
If more than one contract is awarded as a result of this ITN, the Department will coordinate with the
contractors to determine the most efficient method of planning and conducting the review meetings. The
respondent may propose recommended solutions in the event of the award of multiple contracts.
7.3.4.1 Bias and Sensitivity Committee Reviews
The selected contractor is responsible for coordinating all logistics and for preparing materials for the
bias and community sensitivity review committees. The committees are normally composed of
representatives of Florida citizens and educators from various backgrounds who review each newly
developed test item. The primary purpose of each committee is to consider whether the subject
matter and language used is free of potential bias and acceptable to Florida students, parents, and
other community members.
Respondents should propose methods for conducting these reviews.
7.3.4.2 Item Content Review Committees
For the subject/grade combination content review meetings, each item should be projected on a
screen and changes should be made to items onscreen throughout the review process. The selected
contractor will be responsible for supplying the computers, projection devices, and other hardware
and software needed to project and modify items onscreen. The contractor will also provide
calculators, reference materials such as Test Item Specifications, textbooks relevant to the grade and
subjects being reviewed, dictionaries, and thesauri for each room as is appropriate to the nature of
the meeting. Print copies of the items being reviewed will be provided to each participant in securitycontrolled notebooks.
At the conclusion of each meeting, Department and contractor staff will review and evaluate changes
and ideas recommended by each committee. The Department will have final approval of all changes
to be made.
After these meetings, the selected contractor will conduct an editorial and content review of the items
to ensure that changes have been applied correctly and that all editorial and content issues are
corrected. This process is referred to as the “Cleans” review process. The Department will review
and have final approval of all items.
The selected contractor’s development project manager, the development team, and necessary
assistants will participate in the assessment item content review meetings to manage the meetings
and to be responsible for organizing, distributing, secure material tracking, and data entry functions
required for meetings. Arrangements will be required for collecting and keeping secure materials
locked in secure storage when not in use and for copying materials during the meeting.
7.3.4.3 Data Review Committees
Once items have been field tested, the contractor is responsible for convening a meeting to review
the statistical data on each item from field testing to identify any issues from a psychometric
perspective. This Data Review Committee may include state content experts, grade-level teachers,
Florida Standards Assessments, ITN 2014-47
31
accessibility and accommodations experts, and psychometricians, as appropriate. The primary
purpose is to ensure that all dichotomously-scored items and tasks behave as expected, include all
correct and appropriate responses, and are fair for all students. This committee may review
differential item functioning (DIF) statistics to ensure that an item did not unintentionally favor or
disfavor a particular subgroup of students. Additional considerations include determining whether an
item can be used as is or needs to be revised and, if so, whether it will be returned to the field test
pool for further testing. The contractor will be responsible for applying any recommended edits to
items and will submit the revised items for final Department approval.
7.3.4.4 Content Advisory Committees
Of particular importance for the design and development of Florida assessments are the Content
Advisory committees. The members of the ELA/L and Mathematics Content Advisory committees
advise the Department about the scope of the assessments for these areas. Their recommendations
may include: standards to be assessed; the number of benchmarks, passages, and items to be
addressed per grade level; the item types recommended for each assessed standard; and the
content of the Test Item Specifications documents.
7.3.5 Develop Field-Test Item Sets and Forms
For designs using embedded field testing, it is the responsibility of the selected contractor to group the
developed items into item sets that will be included in the operational forms of the test for each content area.
In making the decision as to which items will be field-tested, the selected contractor’s field-test proposals must
consider the areas of greatest need in the item bank, the range of content and rigor included in the items, and
the number of items available to be field tested.
7.3.6 Develop Stand-Alone ELA/L Field-Test Item Sets and Forms
Depending on the measurement model for the proposed ELA/L assessment, the Department will consider a
stand-alone field test for ELA/L extended constructed-response tasks. Respondents who propose a standalone field test should prepare field test sampling specifications detailing the sampling methodology. The
Department prefers that such a field test, if proposed, take place in the December time frame of each year of
administration. A field test administration is not listed in other sections of this ITN (schedules, test
administration manuals, etc.); however, if it is determined during negotiations that a stand-alone field test will
take place, all test administration and scoring schedules and deliverables will be agreed upon prior to contract
execution.
7.3.7 Constructing Test Forms
This section describes the requirements for constructing test forms, including the creation and maintenance of
test construction specifications. The test construction process must be supported by sophisticated computer
software, preferably a component of the item banking system that will generate test statistics and related
curves. The construction of operational forms must be completed in time to meet the requirements of this ITN
for delivery of test materials to districts.
7.3.7.1 Test Construction Specifications
The specific processes, guidelines, requirements, and schedule to be followed in constructing all
forms of the assessments are to be provided to the Department as detailed, expert-developed, and
referenced specifications. The test construction specifications must also include information about
the criteria for item selection, content representativeness, design of the assessment, evaluation
criteria for reviewing statistical characteristics of items, and desirable assessment characteristics. A
section of the specifications must address considerations and requirements for those assessments
that are pre-equated, if applicable, as well as a section for unique requirements of constructing
computer-based assessments. Careful and extensive development of the creation of linking or
anchor item sets must be included with details about statistical and content requirements for selecting
sets of items and criteria for annual refreshing/replacement of portions of the anchor item set.
Additionally, the specifications must include necessary information regarding item context/position
Florida Standards Assessments, ITN 2014-47
32
based upon previous use. The selected contractor must also retain assessment construction records
during the lifetime of the contract. These records must include assessment summaries (item IDs and
item/assessment statistics) of the various versions of proposed assessments and reasons for
decisions made regarding each version, and Department approval documentation for the final version
of each assessment.
7.3.7.2 Test Construction System
The contractor will provide a computer-based test construction system that interfaces with the item
banking system to select and sequence items, generate test curves, and provide statistical summary
tables and test curves upon demand. The system proposed by the respondent should be fully
described in the reply including technical specifications, screen shots and access to a demonstration
version of the system.
7.3.7.3 Construct the English Language Arts/Literacy, and Algebra 1, Algebra 2, and Geometry
Retake Forms
Grade 10 English Language Arts/Literacy and Algebra 1, Algebra 2, and Geometry retake tests will
be administered under this contract beginning in fall 2015. Respondents should propose pre-equated
retake forms that can be scored for reporting on accelerated timelines. Solutions for retakes that do
not include performance tasks will be considered, even if performance tasks are included in the
operational forms. Overall test characteristics must mirror those of the operational assessments;
however, the Department will discuss the design of the retakes during negotiation and a decision will
be finalized prior to contract execution.
7.3.7.4 Prepare Camera-Ready or Screen-Ready Test Forms
For each grade or subject test, the selected contractor is responsible for preparing camera-ready
documents or screen-ready files after the test items have been selected, composed in forms per
grade/subject, and reviewed. Preparing test documents for publication includes developing test book
configurations, test maps, specifying page layouts, proofing composed PDFs, and completing digital
proofs in conjunction with the Department and external reviewers, as appropriate to the selected
contractor’s proposed solution.
7.3.7.5 Interpretive Products
The selected contractor will develop interpretive products (IP) to interpret or describe test items for
specific subject areas or to explain test results. The contractor will be expected to propose sample
items, review publication PDF and digital rounds, provide alternate text for ADA-compliant files,
create or refine annotations for student responses, and ensure coherence and consistency with the
operational test. Interpretive products follow the production and review guidelines outlined in Section
7.3.10. Required products include the following:
Understanding Florida’s Assessment Reports
This publication describes the various reports of test results distributed to students, schools, and
districts. An explanation and an image of each type of report are provided. The booklet also includes
information about the test content areas and a glossary of technical terms. The contractor will deliver
the electronic ADA-compliant PDF for posting to the DOE website so files can be posted to coincide
with the first results delivery (RD1).
Sample Test Materials (STMs)
STMs will be delivered for all assessments, including assessments that are primarily computerbased, and include 10-20 developed sample items in a sample test booklet for students with a
separate answer key booklet for teachers. Both documents are produced in ADA-compliant PDF files
for unsecure web posting and in large print and Braille accommodated versions. The printed
accommodations are delivered to schools each fall so students can practice answering various item
types. The ADA-compliant version is delivered at the same time so all students have the same
opportunity for practice. Respondents should propose designs for sample test books and answer
Florida Standards Assessments, ITN 2014-47
33
keys for the ADA-compliant PDF version and for printed large print and Braille accommodated
versions.
English Language Arts/Literacy and Mathematics Scoring Guides
The contractor will be expected to post electronic publications that provide information to help
educators understand the scoring of constructed-response items included on student reports. These
publications provide information about the test design, standards and benchmarks, item types, and
how items are scored. They also provide scoring rubrics and the annotated papers used to train
scorers for the constructed-response items. Each sample item from the scoring papers is displayed
as it appeared on the test and responses for each score point, images of student responses, and
annotations explaining why the response received a particular score are provided.
Computer-Based Test (CBT) Practice Tests
Respondents should specify a plan for delivering computer-based practice tests for all computerbased assessments, including accommodated computer-based assessments to be reviewed and
revised as needed in conjunction with Department and external reviewers. The purpose for these
practice tests is to familiarize students with the computer-based testing system and to provide
students with an opportunity to practice answering various item types.
All practice tests will include items similar to test format and content. The practice session may
combine subject items or use separate practice sessions for each subject area as the Department will
determine. Practice sessions will include approximately six (6) to twenty (20) items per subject.
Review and development of the student tutorials/practice sessions must follow the Department’s
procedures for forms review, outlined in Section 7.3. In addition, the Department reserves the right to
request updates to the practice sessions to reflect any changes in CBT platform functionality or
necessary changes to the practice questions throughout the life of the contract. The contractor must
anticipate that these updates may occur and must be prepared to provide these updates at no
additional cost. (These would not qualify as “author’s alterations” defined in Section 7.7.9.3.)
Additional information about the computer-based format and practice tests is provided in Table
7.4.9.6.
Test Item Specifications
Respondents should propose a detailed plan for producing and reviewing test item specifications for
ELA/L and Mathematics grades and subjects in consultation with the Department and Florida
educators. The specifications govern the writing and review of passages, prompts, stimuli, and test
items and generally identify the item/task requirements and constraints for measuring each standard.
The publications may include introductory information and appendices as necessary to fully explain
how the tests and items are developed. The test item specifications may include clarification of the
standard being assessed, specific content and item limitations, and item formats by which each
standard will be assessed. Sample items may be included for each item type. Item writers are the
primary audience for these publications, but Florida educators also use these materials, so ADAcompliant PDF files will be posted to the Department website. These publications will not be
distributed in print. Respondents may propose approximate document lengths and document design.
In conjunction with the review of the test item specifications, an external review panel of Florida
educators will review the test design and philosophy contained in each manuscript and will make
recommendations for improvements to the item specifications for each assessment.
7.3.8 Production of Printed Test Materials
1. The selected contractor is responsible for producing all versions of the materials and products developed for
this project, including test forms.
2. The selected contractor is responsible for proofreading documents during all stages of production. The
contractor should propose well-defined procedures to ensure drafts are checked by computer for spelling and
Florida Standards Assessments, ITN 2014-47
34
proofed for complete accuracy by qualified proofreaders from the respondent's staff before they are sent to
the Department for review. Qualified persons within the respondent's organization and independent editors,
who will be responsible for proofreading, should be named in the reply and their resumes attached. The
respondent should include a provision for the cost of changes made during the final proof stage of production
as “author’s alterations” to be authorized by the Department.
3. The selected contractor will design and implement a procedure for assessing the quality of printing. The
procedure must include on-site quality control plans to ensure 100% correct printing, sealing, and collating test
documents. In their replies, respondents should explain what arrangements they have reached with printers
to establish technology or procedures to prevent miscollating test and answer books. Printing subcontractors
that produce miscollated test and answer books may not be approved for subsequent test administrations.
Procedures for ensuring printing quality must include plans for proofing all materials before and after printing.
The selected contractor will be responsible for replacing misprinted or otherwise defective materials at the
contractor’s expense within a time period acceptable to the Department.
4. In addition to the quality of printing, the Department requires stringent test security procedures to be followed
during all stages of document production and printing. In their replies, respondents should explain the security
plan to be implemented while test documents are in production and at the printer. See Appendices F and I for
these restrictions.
5. If errors occur during production, the selected contractor will be responsible for providing to the Department
within 30 days of the event a root-cause analysis, providing details on the cause and steps for improvement.
7.3.9 Production of Computer-Delivered Products
1. The selected contractor is responsible for producing all versions of computer-delivered products developed for
this project, including test forms. These products include all information, text, and graphics that will be
presented to computer-based test users.
2. The Department recognizes that the development, review, and approval processes for computer presentation
of information and materials parallels similar work for printed tests and products.
3. The respondent should specify its suggested plan for computer-based-test production with the objective of
ensuring that industry-leading quality is attained. The plan for the test content review should include steps
required to develop computer-based tests and practice tests to be reviewed and revised as needed prior to
approval in conjunction with Department and external reviewers, as appropriate to the respondent’s proposed
solution.
4. The functionality of CBT forms (e.g., tools, reference sheets, navigation) must be checked carefully to ensure
that errors have not been introduced, and that the functionality remains consistent across forms and
administrations. The Department will provide a checklist that can be used as a starting point for the
functionality review, but the selected contractor must be prepared to devote significant time to thoroughly
check each form.
5. A final review round will include an “end-to-end” check of the product, including checks to ensure user access
to the product and accurate navigation and functionality. The respondent should include a provision for the
cost of changes made following approval of a computer-based-test form as “author’s alterations” to be
authorized by the Department (see Section 7.7.9.3).
6. Qualified persons within the respondent's organization and independent editors, who will be responsible for
proofreading, should be named in the reply and their resumes attached.
7. The Department requires stringent test security procedures to be followed during all stages of document
production and printing. In their replies, respondents should explain the security plan to be implemented while
test documents are in production. See Appendix F for these restrictions.
Florida Standards Assessments, ITN 2014-47
35
8. If errors occur during production, the selected contractor will be responsible for providing a root-cause
analysis to the Department within 30 days following the event, providing details on the cause and steps
for improvement.
7.3.10 Production of Paper-Based and Computer-Based Test, Style Guide, and Braille Production Procedures
1. The selected contractor will be responsible for all aspects of production for the CBT system, secure webbased reporting site, and printed materials, including formatting, graphics, and key entry. For each
publication, the contractor will submit product specifications that identify printing plans, type size and
style, ink and paper color, paper quality, and layout. Printing examples that show type size and style will
be included. The contractor will provide copies of the final electronic files for products.
2. The selected contractor will also print any additional materials needed to implement the project, such as
transmittal memoranda, labels for packing and packing lists.
3. The selected contractor will be responsible for all aspects of production for publishing printed products,
including formatting, graphics, and key entry. For each publication, the contractor will submit printing
plans that identify type size and style, ink and paper color, paper quality, and layout. Printing examples
that show type size and style will be included. The contractor will present these plans annually. The
Department expects attractive, quality printed materials at reasonable cost. Guidelines for the quantities
of each product to be printed are indicated in Appendix E.
4. Editorial specifications must include editorial and stylistic considerations for the development of test
questions and the layout of test pages. The selected contractor is responsible for updating and revising
the specifications annually as style and layout requirements may change. The contractor will work with
the Department to identify the components of the specifications that need revision and will revise and print
copies for all users.
5. The selected contractor will submit plans that identify screen layouts and navigation elements, font style
and size requirements, background and foreground color palettes, and requirements for graphics. The
contractor will identify and propose a branding plan to identify and provide consistency for Florida
assessment components delivered via computer. The Department expects an attractive, high quality,
engaging, and functional presentation. The contractor will present this information, as well as the
proposed scheduling and process for review rounds, in a specifications document that addresses
computer-based presentations.
7.3.11 Produce Special Formats and Accommodations for Students with Disabilities
7.3.11.1 Paper-Based Accommodations
For all assessments, the selected contractor will provide Braille and large print versions for visually
impaired students. For all assessments, the selected contractor will also provide one-item-perpage/one-sided accommodated versions as unique accommodations. For test documents, the
selected contractor will create the electronic files in the correct format needed to produce these
special formats. Although multiple forms of the tests may be constructed for each administration,
only one form for each test will be converted to Braille, large print, and one-item-per-page versions.
Large print documents will be printed in a minimum of 18-point type on approved paper and bound
using agreed upon methods. Reformatting of documents may be necessary to meet these
specifications. The selected contractor is responsible for having Braille materials proofed by an
independent party that includes a certified Braille reader. The Department may also employ the
services of a Braille proofreader. See Appendices E and G for more details and specifications on
special formats of test documents.
1. Respondents should include proposed subcontract publishers of Braille and large print materials
in their response. This publisher will produce the accommodated versions of the test and
answer books and other documents at the contractor’s expense. Test administrator notes and
Florida Standards Assessments, ITN 2014-47
36
scripts to accompany each administration’s Braille test versions will also be developed by the
contractor.
2. While most blind students in Florida require contracted Braille, there will be some students who
require uncontracted Braille. The selected contractor will be responsible for providing either
format based on district requests. Approximately 50 copies of the Braille versions and up to 400
copies of the large print version may be required for each grade level and subject area per
administration. Test administrator notes and scripts to accompany each administration’s Braille
test versions will be provided by the contractor as a supplement to the manual, and will undergo
the same iterative review process as other printed materials.
3. If requested by districts, the selected contractor will also provide large print and/or Braille
versions of sections of the test administration manuals. These publications will be produced so
that they will be delivered to districts along with the shipment of the regular print versions of
these products as identified in Appendix E. A maximum of five (5) each per format may be
produced for each administration.
4. The selected contractor will maintain and update an internal document that identifies which
person, by position and name that will perform each task in the Braille materials process. Each
task must be described in detail and the document must be accessible online with other
downloadable materials.
5. On request, the selected contractor will develop and deliver one-item-per-page documents to
districts. Approximately 25 copies of the one-item-per-page version may be required for each
grade level per administration. The need for these specialized documents will be determined by
the Department through its unique accommodations requests process and the Department will
provide an approved list identifying students needing this accommodation to the contractor.
6. The one-item-per-page documents will be packed and shipped by the selected contractor for
identified students. These documents will be printed on 8 ½” x 11” paper, three-hole punched,
and placed in a binder so that individual pages may be removed as necessary. Because these
are also used for students who require true black and white print, the selected contractor will
provide for the Department’s approval a sample of the materials to be used that demonstrates
the paper quality and font/style. The document package will include the appropriate scannable
answer document for student responses and the contractor will include directions for
administration and management of this test version. The contractor will be responsible for
transcription of one-item-per-page student responses.
7. The respondent’s reply should describe the processing of large print, Braille, and one-item-perpage versions of answer documents, including handling and transcription during the scoring
process. All Braille, large print, and one-item-per-page materials require separate packaging but
must be shipped so that they arrive in districts at the same time as the materials for regular
students.
8. The contractor will also deliver paper-based regular print or large print Reading Passage
Booklets for computer-based Reading tests. The booklets, which contain reading passages but
do not contain test items, will be for use by eligible students who need to utilize handwritten
strategies in order to respond to test items on the computer. See Section 7.3.2.1.
7.3.11.2 Computer-Based Accommodations
Florida’s computer-based tests must also be available in paper-based format as an accommodation
at all grades and subjects. The use of this format will be dictated by a student’s IEP or Section 504
plan. The Department anticipates that the number of students taking a paper-based test (PBT)
based upon allowable accommodations will not exceed five percent (5%) of the total population for
each grade and subject tested on computer, and the selected contractor will produce this quantity five
Florida Standards Assessments, ITN 2014-47
37
percent (5%) of paper-based tests. For these students, paper-based accommodations described in
Section 7.3.11.1 must be available.
The reply should address the capability of the computer-based system to provide the following
special accommodations for students and to handle any additional requirements related to the use of
these accommodations.
1. Screen Reader
The respondent’s reply should include a description of a screen reader for computer-based
tests. The respondent should propose a method for reading text within the computer-based
system that does not require additional, proprietary software installed on the user’s computer.
The screen reader should be able to read all text and/or text descriptions of graphics or pictures,
including graphics or pictures used as answer options. The system should also have the
capability to use modified text if necessary (e.g., reading an algebraic equation with grouping
symbols), and allow user control of the reading speed and volume. The screen reader should be
designed for use by both sighted and visually impaired students. The Department recognizes
that the contractor may have different types of screen readers for these two purposes, but this
difference should be transparent to the user. The Department must approve the usability and
voice of the screen reader. In addition, the contractor must ensure that the pronunciation of the
screen reader is accurate, consistent across forms and subject areas, and would be
recognizable to students. If the screen reader does not have accurate, recognizable
pronunciation, it must be modifiable to meet the Department’s standards. In addition, the
contractor must check the pronunciation on every form to ensure it is correct, is tracking
properly, and is understandable prior to it being provided to the Department for review. The
Department prefers that the contractor adhere to the Department’s screen reader specifications
to ensure consistent performance of the screen reader. However, the Department will review
screen readers that meet the intent of this section and the needs of the Department.
2. Increased Font Size (Large Print and Zoom)
The respondent’s reply should describe available options on the CBT system to enlarge font
size, graphics, and CBT interface (i.e., buttons, tools, cursor, scrollbar) for students with
disabilities and describe how these options are set for a student. See Section 7.4.9.6 for large
print and zoom size requirements.
3. Font and Background Color (Color Contrast)
The respondent’s reply should describe available options on the CBT system for changes to the
font color and background color in order to optimize the ability of students with visual
impairments to read the screen and describe how these options are set for a student. See Table
7.4.9.6 for the font/background color combinations.
4. Assistive /Adaptive Devices
The respondent’s reply should describe the ability of the CBT system to interface with
assistive/adaptive devices. The reply should include a description of specific devices that may
be used and known devices that are not supported, as well as any special requirements
necessary for use, and should describe how options for use are set for a student.
5. English-to-Heritage Language Translation Dictionary
The respondent’s reply should describe the provision on the CBT system of an on-screen
English-to-heritage language translation dictionary for English Language Learners, if this can be
made available. The accommodated student should be able to select a word and be presented
with a translation sensitive to the context of the text. The language translation dictionary should
be available in Spanish and Haitian Creole. The respondent’s reply should describe the highlevel tasks involved, note the general timelines required, including the major/essential
deliverables and services required.
Florida Standards Assessments, ITN 2014-47
38
6. English-to-Sign Language Translation Dictionary
The respondent’s reply may describe the provision on the CBT system of an on-screen Englishto-sign language translation dictionary, if this can be made available. The accommodated
student should be able to select a word and be presented with a translation sensitive to the
context of the text. The respondent’s reply should describe the high-level tasks involved, note
the general timelines required, including the major/essential deliverables and services required.
7.4.0 Test Administration
The contractor is responsible for all aspects of test administration described in this section.
7.4.1 Pack and Distribute Materials
1. The contractor is responsible for all arrangements and costs associated with packing, distributing, and
retrieving physical materials. There must be 100% accounting for all secure materials (e.g., test books,
test and answer books) distributed to and returned by districts using barcode labeling systems, as well as
for any physical media (e.g., CDs) used to transmit electronic files. The contractor must guarantee that
distribution procedures are accurate and make corrections to the system in the event of errors.
Respondents’ replies should include descriptions of the procedures they will use to complete these tasks.
Respondents should identify carriers they propose to use and the procedures for delivery and retrieval of
test materials. The Department must approve all carriers.
2. The contractor will prepare packaging and distribution (P&D) specifications for each test administration.
The contractor will use the information in the specifications to program packaging systems and train
personnel involved in the processes. The packaging specifications will include the contractor’s
procedures for packing and distributing materials to districts and retrieving materials from districts. The
specifications will include a description of how the materials are packaged, how quantities of materials will
be calculated, examples of packing and inventory lists for boxes sent to districts and schools, methods
used for distributing and retrieving materials, a description of the procedures used to inventory materials
as they are returned, as well as any special instructions or administration-specific information. The
contractor will also include samples of all memoranda required in the delivery/pickup process. For
reference, the Department will provide the contractor with the most recent packaging and distribution
specification document at the initial contract management meeting. Proof of Department approval for
each specification document will be evidenced through email confirmation of approval.
3. The number of districts and special schools that serve as districts may change slightly during the life of
the contract. Appendix C provides information about the number of districts, schools, and students that
will receive test materials. Respondents should anticipate shipping materials to district assessment
coordinators at approximately eighty (80) separate sites. Materials will be shipped to a central district
location and distributed to the schools by the district.
4. Several school districts have contracts for services with school sites that are not geographically located
within the school district itself. School district personnel supervise the sites and the district assessment
coordinator provides training for site-based test administrators related to the administration, including
security, packing, and shipping. To enable prompt receipt and return of test materials, the contractor may
be required to ship materials directly to and retrieve materials directly from these remote sites. The
Department will work with the district assessment coordinators and the contractor to identify and approve
the remote sites to which the contractor will be permitted to ship and retrieve materials without going
through the district office. For the purpose of preparing replies, respondents should expect that up to
twenty-five (25) such locations might be identified.
5. At the beginning of the contract, the Department will provide the contractor with a data file containing a list
of the districts and schools (names and identification numbers). The contractor will be responsible for
maintaining this list, updating it, and providing a copy of the data file to the Department at the end of the
contract or at other times within the contract period on request.
Florida Standards Assessments, ITN 2014-47
39
6. The contractor will collect preidentification information from districts for each administration via the online
management system and assume that preidentification will be used for 100% of the school population.
These files will be used to load student data into computer-based testing systems, as well as produce
barcode PreID labels and rosters for paper-based tests. The contractor will establish a system to ensure
that the labels/documents delivered to districts contain all requested data, reflect the options selected by
districts, are accurate, and are printed at a level of quality that permits accurate scanning and precludes
the possibility of smudging, smearing, and/or flaking. Contractors will provide quality control throughout
the printing process to ensure the quality of label/document printing. The contractor will provide a
separate roster of students using the same sort order used to prepare the student labels/documents. This
roster will be used by districts and schools to verify the accuracy of certain information that must be
correct before the label/document can be used.
7. A list of the primary materials to be shipped for each administration, the quantities to be packaged for
schools, districts, and the Department, and other packaging specifications is provided in Appendix E.
8. The contractor’s specifications for packing, distributing, and receiving materials will include the following
provisions:
a. The contractor will shrink-wrap materials in quantities specified by the Department. Some materials
will not be shrink-wrapped (e.g., Test Administration Manuals). All secure materials will be shrinkwrapped.
b. Schools within a district will be ordered by school number on all lists and for shipping purposes,
unless the district has requested that materials be sorted in route code order, in which case the
materials will be sorted by route code, by school number.
c. The contractor will package and ship materials by district size as determined by the Department,
beginning with the largest district. The largest district will receive materials at the beginning of the
delivery window; the smallest district will receive materials last. District rank by size information will
be updated annually by the Department.
d. The contractor will label the boxes of test materials with the message “TO BE OPENED BY
ASSESSMENT COORDINATOR ONLY.” District boxes will be differentiated from school boxes by
color (e.g., white district boxes and brown school boxes). Boxes containing the district/school
packing lists, materials return kits, and ancillary materials must appear different (e.g., different color,
colored stripe) from all other district and school boxes for easy identification. School boxes will be
labeled with the name/number of the district and name/number of the school in at least a 16 point
font. Label design must be approved by the Department. Only boxes directed to the district-level
staff, such as boxes containing district overage, will be labeled with the name of the district
assessment coordinator. The contractor will label district boxes (that contain overage and other
materials for the district) on all four sides and the top, and number boxes as “Box 1 of X,” where X is
the total number of boxes sent to that district. The contractor will label school boxes on all four sides
and the top, and number boxes as “Box 1 of X,” where X is the total number of boxes sent to that
school. If materials for more than one administration or type of material are shipped during the same
window, the contractor must have a way to distinguish these shipments from each other (e.g., colorcoded label on boxes). Boxes containing secure test materials must never indicate secure content
on the outside of the boxes.
e. For districts receiving more than one pallet in a shipment, a map listing the identity and location of
boxes on each pallet will be provided electronically and will be provided at the time of delivery as a
hard copy to the district assessment coordinator to assist with distribution of boxes in the district.
f.
The contractor must have a method to collect route code information from districts and package
materials accordingly. If district assessment coordinators provide internal district delivery route code
Florida Standards Assessments, ITN 2014-47
40
information, the contractor will use this information to build pallets for districts to match district
delivery route codes.
g. The contractor will pay charges on all materials shipped to and from each district. The contractor will
make arrangements for and pay for shipment if, due to a delivery error, a district is asked to ship
materials to another district. The contractor must use an overnight delivery service for such
shipments. The contractor is not responsible for any costs schools may incur in shipping test
materials from their schools to the district office, unless the need to ship is the result of a packaging
error by the contractor. The contractor must also have a method for districts to order additional test
and ancillary materials, as needed, in the online management system. The contractor must be aware
that requests for additional materials are often critical and such shipments must be shipped overnight
for early-morning delivery, on request.
h. The contractor must receive Department approval before shipping materials to districts. Approval will
be provided after the Department has received and proofed examples of printed products and
reviewed sample shipments, as described in the packaging and distribution specifications. Approval
is sent via an Approval to Ship form.
i.
Secure materials will be packaged sequentially in ascending barcode order (for example, if a school
receives 200 secure documents and 100 documents fit in each box, Box 1 will contain the documents
with barcode numbers 1-100, and the document with barcode #1 will be at the top of the documents
contained in Box 1 and the document with barcode #100 will be at the bottom of the documents
contained in Box 1. Box 2 will contain the documents with barcode numbers 101-200, and the
document with barcode #101 will be at the top of the documents contained in Box 2 and the
document with barcode #200 will be at the bottom of the documents contained in Box 2.). Packing
lists must list all box contents in order as they are packaged with the first item on the list being the
first item (on top) in the first box.
j.
The contractor will package materials for each shipment by school in returnable boxes (doublewalled) suitable for multiple uses. The contractor will provide additional boxes statewide for the
return of materials that are available as an additional order for districts.
k. No box will be packed to weigh more than 25 pounds, and boxes will be of the dimensions, or similar
to, those indicated in Table 7.4.1. If test materials require different box dimensions than those listed
below, the specifications for boxes will be approved by the Department during the negotiation phase.
Table 7.4.1 Box Dimensions*
Table 7.4.1 Box Dimensions*
BOX SIZE
MATERIALS
9.5” x 12” x 13”, Double-walled
Regular Print
9.5” x 12” x 6.5”, Double-walled
Regular Print
19” x 15” x 4.5”
Large Print/Braille
19” x 15” x 6”
Large Print/Braille
19” x 15” x 9”
Large Print/Braille
* Box height is dependent upon the quantity of materials in each shipment.
l.
Upon completion of the delivery of each shipment, the contractor will provide proof of delivery to the
Department indicating, but not limited to, a description of the materials, date/time of delivery, and the
name of the staff member accepting the delivery. Proof of 100% delivery will be used for invoicing
purposes.
Florida Standards Assessments, ITN 2014-47
41
m. The contractor will provide fifteen percent (15%) materials overage to each district for paper-based
test materials and computer-based ancillary materials (work folders, etc.). The Department and
districts will decide how materials overage will be distributed to districts and schools for each
shipment. Districts may select from four (4) distinct algorithms for overage distribution:
All fifteen percent (15%) overage to the district
All fifteen percent (15%) overage to the school
Ten percent (10%) overage to the district and five percent (5%) overage to the school
Five percent (5%) overage to the district and ten percent (10%) overage to the school
n. The contractor must anticipate that some districts will require an additional pickup after the specified
pickup date. This may occur in up to 15 districts for as many as 500 boxes for each spring
administration, and in up to 10 districts for as many as 200 boxes for each retake and End-of-Course
administration.
o. Large-print, Braille, and one-item-per-page materials are to be packaged and labeled separately and
delivered to districts during the same delivery window as the shipment of other test materials in Test
Materials Shipment 2 (TM2).
p. Preidentification labels and rosters will be packaged with test materials (included in TM2). If
applicable, Wave 2 labels will be shipped separately.
q. It may be necessary for the contractor to ship test materials to districts/schools and the Department in
multiple shipments per testing window, not including requests for additional materials. The number of
shipments may vary depending on the administration. The respondent should refer to the
approximate shipment dates provided in Appendix B. The contractor must be prepared to deliver
some materials on alternate schedules per district request (e.g., to avoid conflicts with spring break or
closures) and must have a method to effectively track these requests and ensure that shipping
companies meet the revised dates. Shipping companies must never contact districts directly to
arrange alternate delivery/pickup dates unless agreed upon by the Department.
r.
The contractor will prepay charges on return shipments from districts. Return labels, prepaid postage
labels, instructions, and other materials needed for return will be provided in a Materials Return Kit
shipped in TM2.
s. Districts will return answer documents using overnight delivery, if deemed necessary to meet
reporting deadlines. Materials will be returned according to a pickup schedule developed by the
Department and the contractor that allows schools enough time to package materials and also meets
scoring and calibration deadlines.
t.
The contractor will be responsible for mailing or shipping any miscellaneous materials to the
Department and districts by overnight delivery service or other means, as requested. The contractor
is required to secure the services of shippers who will provide inside delivery and unload large
shipments onto loading docks. All carrier vehicles must display the carrier name, and districts must
be made aware of what carrier name to expect prior to the pickup or delivery date. Districts may or
may not have a loading dock or the appropriate equipment to support the delivery and pickup of
materials (e.g., shrink-wrap; hand-trucks). In this event, the contractor’s shipper must be prepared to
provide such services/materials. District staff is not expected to assist in loading or unloading
materials, but will be available to provide instructions to the carrier. Carriers must communicate with
district staff face-to-face when delivering or picking up materials, never leave materials unattended,
and be prepared to protect materials in the event of inclement weather.
u. The contractor must develop procedures to monitor the receipt of all materials and develop error logs.
The date materials are received and any errors made by districts in packaging and completing forms
Florida Standards Assessments, ITN 2014-47
42
must be documented. When problems arise, the contractor must contact districts and the Department
to resolve them. Error logs will be identified by school and district, and will document all failures to
follow the established procedures and, if appropriate, how the errors were resolved. The error logs
will be delivered to the Department within three weeks after the scheduled receipt of materials from
all districts.
v. The contractor will use a barcode numbering system to verify all secure materials returned after
testing. The contractor will provide a written report to the Department documenting the check-in of all
secure materials. f the contractor’s system for barcode verification is demonstrated to be unreliable
(e.g., materials not scanned properly), each document will be required to be scanned twice, creating
independent data files that can be matched for scanning errors. Beyond these measures, the
contractor will be required to manually check answer documents to ensure 100% accuracy of secure
document check-in.
w. The contractor will include in its check-in procedures a method of checking for and retrieving used
answer documents erroneously packaged with used and unused materials, invalidated materials, or
test books. These “orphan” answer documents will be processed and scored under extended
timelines and individual reports will be generated and shipped to the districts.
x. The contractor will produce statistical reports summarizing responses to the online test administrator,
school coordinator, technology coordinator, and district coordinator comment forms. A summary,
analysis, and list of recommended actions derived from the comment form responses, as well as a
data file containing all responses, will be delivered to the Department no later than two (2) months
after completion each administration. Contractor staff will be responsible for summarizing the openended responses received from district and school staff. This summary must be a categorized report,
not simply a list of open-ended responses.
7.4.2 Missing Materials Report and Inventory
The contractor will prepare a missing materials report for all secure materials, based on the scanning that is
completed during materials check-in. The report must be produced after all materials have been scanned,
including as many orphan and late-returned documents as possible, and must not reflect materials that are not
missing (e.g., contractor has verified against the soiled/destroyed documents list). The contractor must
implement quality control measures to ensure the accuracy of the reports. The contractor will produce a
preliminary missing materials report for Department review. The Department will review and approve the
preliminary missing materials reports for distribution to districts. The Department requires ten working days to
review preliminary reports prior to approval. Reports will be prepared for each affected school listing the title
of the missing products and the corresponding barcode numbers. School reports will be delivered to district
assessment coordinators with a district-level summary and memorandum, and a state-level summary report
will be delivered to the Department. Any missing materials returned by districts will be recorded in the missing
materials inventory maintained by the contractor. The contractor will deliver a final summary report of missing
materials to the Department by the date indicated in the Deliverables Schedule.
7.4.3 Disposition and Storage of Materials
After testing, the contractor will inventory and store unused test and answer books, manuals, and other
materials for a period of at least six (6) months. During this time, the contractor will be required to ship
quantities of these materials, as necessary, to the Department and districts.
Replies should include a plan for secure destruction of excess test materials that requires Department
approval. However, the contractor will store 100 copies of each test and answer book for each administration
throughout the life of the project. In addition, the contractor will store some digital record of the CBT
assessments as they appeared to the students for that administration. Any materials that may be used in
subsequent assessments will be stored by the contractor throughout the life of the project. Additionally, six (6)
months after the contract ends, the subcontractors and print vendors will be required to destroy all electronic
files and used print copies according to the test security requirements approved by the Department.
Florida Standards Assessments, ITN 2014-47
43
On verification of the individual barcode numbers of all secure materials returned by the districts and
acceptance by the Department of accurate results files, the contractor will store used documents containing
student responses, at contractor expense, for a period of two (2) years. Likewise, all student CBT response
files must be electronically archived for a two-year period. At the end of the two-year period, the contractor
will ship or destroy the materials according to instructions from the Department. The contractor must submit a
letter to the Department requesting permission to destroy specific materials. This process is already welldefined by the Department and will be shared at the initial contract management meeting. Test security
requirements will be maintained throughout the destruction process. It may be necessary to retain a small
number of used documents for a longer time period due to Department security investigations. At the end of
the contract, the contractor will be expected to ship these materials to the new contractor.
Destruction of secure documents and CBT student response files must be requested in writing and authorized
by the Department. The contractor and all subcontractors must submit certificates of destruction that describe
in writing the specific items/files destroyed.
7.4.4 Retrieval of Answer Sheets and Test Books
In addition to the document retrieval specified in Section 7.4.1, the contractor will periodically be required to
retrieve up to 1200 used test and answer books per grade, per administration from storage. Likewise,
individual student response files from the computer-based testing system for up to 1200 students per grade,
per subject, per administration may need to be retrieved. The contractor must maintain information related to
warehouse location for each school’s used materials to allow for retrieval as may be required for Department
investigations or audits. In some cases, retrieval of answer documents will require the contractor to conduct a
manual verification of machine scoring. The contractor will be responsible for costs associated with retrieval,
review, and possible delivery of these materials to the Department. In addition, if errors are found, the
contractor may be required to re-score and re-report these documents.
7.4.5 Innovative Product Design
New, innovative designs are required for all Florida-specific products developed by the contractor. Designs for
publications will include the organization, format, and page or screen layouts for administration manuals,
reports of assessment results, information publications, and other materials. Designs will also be produced for
other products, as appropriate. The contractor will produce all graphics, charts, and illustrations for the
products for which it is responsible and share designs and graphics with other contractors, as required by the
Department. The design process will include furnishing conceptual art, design development, and final cover
designs. The contractor will produce all graphics, charts, and illustrations and will secure the full range of
copyrights and/or permissions that allow for production in print, secure web, and unsecure web format.
The contractor must arrange for personnel with professional expertise and experience in the areas of
visual/graphic arts and document/software/web design to work on the design and development of documents
and computer/online interfaces. If the contractor does not have qualified, professional personnel devoted
primarily to visual/graphic arts and document/web design on staff, a subcontractor or subcontractors must be
identified to provide these design services. Respondents should indicate in their replies the qualified
personnel/subcontractors who will provide these professional services and include their vitae or other
documentation of expertise and experience in the reply.
7.4.6 Printed Products – Administration
This section provides guidelines, including descriptions of and duration of review rounds, for all printed
products required for test administration services in this ITN. Contractors must provide high-quality, carefully
edited materials for each round of review by the Department.
7.4.6.1
Stages of Development of Printed Products – Administration
The contractor is responsible for producing the first written draft and all subsequent versions of all
materials and products developed for this project, and for submitting documents for Department
review. The review period for the Department is seven (7) work days, but longer review times will be
required if multiple documents are under review simultaneously. For each review round, the contractor
Florida Standards Assessments, ITN 2014-47
44
will post an electronic version (PDF) by 9:00 a.m. ET on the first scheduled day of Department review.
If materials are delivered or posted after 9:00 a.m., the review schedule will be revised accordingly.
Documents for CBT-related materials will be posted to a secure website for review in their presentation
format.
The delivery of administration materials is a critical task and workflow must be planned carefully. The
overlap of reviews could create a burden on both the Department and contractor staff. Excessive
burdens can be avoided with sufficient staffing, advanced planning, and an early start. The contractor
must propose a detailed materials development schedule for managing the workflow and volume of
documents to be reviewed in each seven (7) day period in a format approved by the Department. The
contractor must develop a schedule for the production of test administration manuals that follows the
review schedule of test books/answer documents or computer-based test forms by one round. The
schedule must consider what is reasonable given the Department’s staffing constraints. All proposed
schedules must be approved by the Department. The number of documents to be reviewed in any
seven (7) day period must be negotiated with and approved by the Department. The Department is
committed to reviewing products submitted by the contractor as efficiently as possible.
The contractor is responsible for proofreading all documents during the production stages. The
contractor will propose well-defined procedures to ensure that all drafts are checked by computer for
spelling and proofed for complete accuracy by qualified proofreaders before they are sent to the
Department for review. The contractor will submit a written report by the proofreaders indicating
proposed results and actions. Before moving to digital proof stage, the Department will review and
approve the proofreaders’ report. Respondents should indicate in their replies the qualified
personnel/subcontractors/independent editors who will be responsible for proofreading and include
their vitae or other documentation of expertise and experience in the reply.
Test Administration Manuals
For respondents proposing solutions derived from other products, the Department understands that
certain test administrations must be standardized, however, there are certain Florida requirements
that must be met. Replies from such respondents should indicate which aspects of test
administration are considered customizable, and a final determination will be agreed upon during
the negotiation phase. For instance, the Department may require the production of test
administration manuals specifically for Florida that include the standard information, or the
production of Florida-specific supplementary publications that complement the contractor’s more
generic publications. The following information details the Department’s expectations for Floridaspecific stand-alone products.
For each year of the contract (Fall – Summer administrations), the following test administration
manuals will be produced: One (1) ELA/L and Mathematics manual for paper-based tests; One (1)
ELA/L and Mathematics manual for computer-based tests; One (1) ELA/L Retake manual (beginning
with the first retake administration in 2015); (1) End-of-Course manual. Administration requirements
and scripts for all assessed grade levels/subjects will be included in the manuals. Instructions for
Special Format Tests will include scripts necessary for accommodated testing (e.g., Braille, one-itemper-page, and others) and will be provided as separate addenda to the manuals. Computer-based
manuals will also include scripts for the practice tests necessary to familiarize students with the
computer-based testing system. All manuals are published in print and on the contractor’s website.
Minimally, manuals will contain instructions and scripts for administering each portion of the
assessments for the appropriate grade levels for that administration. They also include information
about security of materials, packing and returning materials, receipt and distribution of materials, and
the Department’s test administration and security policies and procedures, which are subject to change
with each administration. Revisions will be made as necessary to reflect program changes.
Florida Standards Assessments, ITN 2014-47
45
The contractor must plan to schedule production of test administration manual covers as separate
mini-deliverables that will eventually be married to the final products. Each cover requires multiple
review rounds by graphics designers, editors, and content specialists, and careful planning is required
to meet production deadlines.
Test Administration Manuals are to be packaged (no shrink wrap) and delivered to districts for each
administration as a part of TM1 according to the specifications in Appendix E, Part 3. All manuals
must also be available in a Section-508 compliant (see Appendix E, Section 2) electronic format on
both the Department’s and contractor’s websites within seven (7) days of Departmental approval of
the web-tagged files. The suggested workflow for the Department and the contractor is provided in
Appendix G to help the respondent anticipate the level of work required on the review rounds.
7.4.7 Test Administration Ancillary Materials
For the purpose of this ITN, test administration “ancillary materials” refers to printed or online products other
than manuals produced by the contractor. Ancillary materials are developed with review rounds and email
approval. Some ancillary materials will be printed and shipped with test materials (e.g., document count
forms, return labels, paper bands). Some will be included as perforated pages in the manuals (e.g., security
agreements, chain of custody form, blank administration record/security checklist, security log). Some will be
developed and provided in an online format only (e.g., comment forms).
This section contains details regarding the ancillary materials required for administration. These materials are
designed to ensure overall security, efficient and secure handling of materials, and standardized
administration activities. In addition to the materials listed here, the contractor will provide, based on
Department request and approval, a number of memoranda to assist in communications with districts
regarding test administration processes and requirements.
The contractor will provide the required publication production and software development resources to
produce the ancillary materials using current staff that have the necessary expertise and serve in such
capacities on a full-time basis, through temporary staffing, or by subcontracting. Respondents should identify
in their replies the staff who will be assigned to develop the ancillary materials and/or subcontractors proposed
to complete these project tasks, as well as their qualifications (education and experience) to do so.
Most of the products identified in this section will be provided in formats that permit them to be accessed
electronically via the internet on both the contractor’s and Department’s websites. Posted products must be
available for viewing and downloading and must be provided in an ADA-compliant format, that is, a format
compliant with Section 508 [29 U.S.C. § 794(d) & 36 CFR Part 1194] of the Rehabilitation Act, as amended,
and s. 282.601, et seq., F.S. The Department will run standard Section 508-compliance checks on all
documents to be posted on the Department website to ensure their accessibility. The contractor will be
required to modify any noncompliant elements and provide compliant documents. The contractor is
responsible for securing all copyright requirements, if applicable, for materials submitted for nonsecure web
posting. (See Appendix E, Part 2 for additional information about Department guidelines for Section 508 pdf
publications.)
The contractor will work with the Department to identify the specific requirements for files to be made
accessible on the online management system and will provide the required files in a timely manner.
The contractor is responsible for developing the following materials for distribution to districts for each
administration. These materials will be produced following the quantities and specifications identified in
Appendix E. The contractor is also required to provide various other materials required to implement
administration activities, including forms, labels, and documents.
7.4.7.1
Training Resources for School and District Technology Coordinators (CBT)
The contractor must provide training materials to support CBT test administrations. These
materials may include screen-shots and will be delivered via narrated PowerPoint web
presentations, WebEx, training module, or other similar mode through the CBT test platform.
Florida Standards Assessments, ITN 2014-47
46
These training materials will consist of an outline of key issues to be covered during school and
district technology coordinator training with explanations accompanied by full-color graphics,
including screen-shots, representing critical elements or live demonstrations of processes in the
online management system.
All training materials are to be made available to districts in a Section-508 compliant electronic
format on both the Department’s and contractor’s website within seven (7) days of Departmental
final approval of the web-tagged files.
7.4.7.2
Instructions and Materials for Braille Tests
The contractor must develop Braille scripts and Braille notes for the administration of Braille tests
for each subject and grade. The braille notes are secure materials and must be handled
accordingly. The Braille notes and Braille scripts are distributed with the secure test materials, and
they must be approved by the Department prior to production. In addition, some items may require
manipulatives to allow access for visually impaired students. These manipulatives are provided,
prepared, and shipped to the districts by the contractor.
7.4.7.3
Student Preidentification Labels
The contractor will produce student preidentification labels for each student to be tested on a
paper-based assessment for whom preidentification data were provided by the districts. (See
Section 7.5.2 for more information about the preidentification process.) Labels will be provided for
each student answer document as needed. Local school staff will be responsible for placing labels
on student answer documents.
•
One label for each answer document not already preprinted
•
Sorted as specified by each district (may be different for each grade/subject and different for
schools within a district)
•
Sorted with page breaks as specified by the district/Department
•
3” x 3 ½” adhesive labels or approved equivalent
•
A supplementary supply of labels will be produced for spring administrations (Wave 2) based
on PreID information gathered in January
7.4.7.4
Student PreID Rosters
The contractor will provide a preidentification list containing all information included on the student
PreID labels for each school in a format approved by the Department. The list is produced from
the same file used to print student PreID labels and contains the same information. (See Section
7.5.2 for more information about the preidentification process.) One row of information per student
will be used to verify the accuracy of student demographic information. Rosters will be provided in
hard copies to schools and electronically via the contractor’s secure website. 8 ½” x 11” paper is
preferred, but the Department may approve 8 ½” x 14” if this is required for readability.
7.4.7.5
Document Count Form (DCF)
The contractor will provide Document Count Forms (DCF). The DCF is a pregridded, scannable, 8
½” X 11” document used to inventory answer documents returned by each school. The form
specifies the subject and grade level test placed underneath it and ensures that the correct number
of documents is scanned. Forms must be preidentified with district/school names and numbers,
and blank forms will also be sent as this information may also be hand-gridded. The district/school
number should match the numbers on the answer documents underneath them, and the number of
documents gridded on the DCF should match the number of documents beneath it. If
discrepancies are discovered during scanning, the contractor will work to resolve them, as
described in the Scoring and Reporting Specifications.
Florida Standards Assessments, ITN 2014-47
47
If the respondent employs a different acceptable method/form for the process described above, the
Department and the contractor will agree on the final process during the negotiation phase.
7.4.7.6
Administration Record/Security Checklist
Security barcode numbers of all secure materials listed in ascending order by product, will be
provided to a school/district with space to indicate assignment and return of materials. The
Department expects that the checklist will also include box numbers and section breaks between
packages and will be provided electronically in an editable format to districts.
If the respondent employs a different acceptable method/form for the process described above, the
Department and the contractor will agree on the final process during the negotiation phase.
7.4.7.7
Comment Forms
The contractor will work with the Department to create the following ADA-compliant online forms
that will be posted to the contractor’s website for each administration:
•
Test Administrator Comment Form (One for paper-based and one for computer-based)
•
School Coordinator Comment Form (One for paper-based and one for computer-based)
•
District Coordinator Comment Form (One for paper-based and one for computer-based)
•
Technology Coordinator Form (CBT only)
•
CBT Student Comment Form
The final output of the comment form responses is described in #26 under Section 7.4.1.
7.4.7.8
Return Shipping Labels
The contractor will provide preprinted, adhesive, color-coded (as required by the Department)
labels for district use in returning materials.
7.4.7.9
Materials Return Kit – District
The contractor will provide envelopes containing return shipping labels, bills of lading, return
instructions memorandum, and Braille notes and scripts, if applicable.
7.4.7.10
Materials Return Kit – School
The contractor will provide envelopes containing document count forms, paper bands, and colorcoded return shipping labels.
7.4.7.11
Miscellaneous Return Materials
The contractor will provide envelopes and boxes necessary for packaging large-print, Braille, and
other special program and special format materials for district use in returning materials.
7.4.7.12
Paper Bands
Printed paper bands, preferably 4” x 24” adhesive, will be provided by the contractor for bundling
scorable materials and provided to schools and districts. The adhesive must not cover the entire
width of the band to avoid adhering to the documents contained in the band. The bands must be
printed on one side to identify the type of answer document in bundle. Bands must be 8pt.
Carolina paper or approved equivalent. In addition, paper band quality must be suitable for filling in
information lines on band with multiple pen/pencil types (e.g., markers, ball point pens, pencils)
without smearing/smudging.
If the respondent employs a different acceptable method for the process described above, the
Department and the selected contractor will agree on the final process during the negotiation
phase.
Florida Standards Assessments, ITN 2014-47
48
7.4.7.13
Rulers / Manipulatives / Calculators
If any test items require the use of a ruler or manipulative, the contractor will produce/provide these
materials for testing in a reliable, cost-efficient manner. Respondents should describe proposed
functionality and features of calculators required for Mathematics tests and the Department will
provide an approved list of calculators for student use. (Contractors will not be required to provide
hand-held calculators to districts.) Any testing accessories required for computer-based tests must
be built into the platform and must also be available for paper-based tests. If materials not listed in
this ITN are needed for testing, they should be clearly identified in the respondent’s reply and the
Department will specify its requirements during the negotiation phase.
7.4.7.14
Miscellaneous Memos, Forms, Labels, and Other Products
The contractor will provide the following products:
•
Miscellaneous memoranda to the district coordinators may be necessary for each
administration. Letters and memoranda will be of a length to be determined by the
Department, up to five (5) pages each, and length of documents may change at each
production round.
•
Cover memoranda for all shipments of materials indicating what is being shipped and how it is
packed.
•
Packing lists must be provided with all shipments (district coordinator receives a hard copy of
the district and all school packing lists and receives all lists electronically as soon as they are
available).
•
Pallet lists (maps) must be provided to any district that has materials on pallets.
•
Other miscellaneous memoranda, labels, forms, etc., will be required to implement the
program.
7.4.8 Online Management System
The contractor’s online management system described in this section will be the primary resource for all
student data management (paper-based and computer-based), as well as test administration and reporting
tasks. The expectations of this ITN are written with the full understanding that capabilities for an online
management system are expanding quickly. The Department is interested in reviewing innovative procedures
and techniques for managing student assessment online, which take advantage of the maturation of
technology in educational assessment through the life of the contract arising from this ITN. Therefore, the
respondent may submit a reply that varies from the expectations herein as long as the solution meets the
needs of the Department and the intent of the ITN. Respondents should agree to work with the Department to
amend the specifications, expectations, and requirements to ensure that the Florida testing program keeps
pace with the expanding capabilities of test delivery and management via computer. Any such variations will
be resolved and agreed to during the negotiation phase.
The online management system should be a secure site that requires usernames and passwords and
accounts should be customizable based on the access a user will have (state-level, district-level, school-level,
etc.). Depending on the access level, a user should be able to view other organizations in the system (i.e., a
state-level user may select and view any district or school in the state) and have the ability to make changes
or alter data in those organizations.
The Department will provide the contractor with a list of the current names, addresses, email addresses, and
phone and fax numbers of the district assessment coordinators. The contractor will populate the passwordprotected online system with this information to create initial district-level user accounts. New accounts should
be able to be created individually or via file upload to create multiple new accounts or make updates to
existing accounts.
Florida Standards Assessments, ITN 2014-47
49
The system should allow districts to upload enrollment counts, preidentification files, and to place additional
orders for materials. Using the Department’s Master School Identification (MSID) file, the contractor will
populate the system with all active districts and schools. The system must have the ability to add additional
schools that are not on the MSID file (e.g., private schools), and these schools may require special school
types as determined by the department.
Components of the contractor’s online systems (e.g., websites, databases) must be submitted to the
Department for review of content, layout, aesthetic quality, and functionality. The contractor will make any
requested changes to such systems. The online systems may not be used for contractor advertisements or
other promotional purposes. The “branding” of these systems should be primarily for Florida’s assessment
system, but may include contractor logos and other information as approved by the Department.
The system should also accommodate CBT delivery. All systems that identify, prepare for, and monitor
student test participation must have the capacity to distinguish between paper-based and computer-based
tests, including all applications of preidentification and enrollment files. These systems must also be able to
accurately track student participation by delivery mode (paper-based, computer-based, or both) and reflect the
test delivery mode(s) in each student file.
The system should be designed to provide user access to various non-secure and secure links including:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
general system introduction and descriptive information;
updates to current operational status and special situations;
self-running software to test the suitability of computers for test delivery;
self-running software to download a “secure browser” to ensure test and system security;
all user guides, manuals, training, and ancillary materials in downloadable format;
procedures to add, modify, or remove students or student information in preparation for testing or
reporting;
record test group codes and accommodations for computer-based tests;
ability to invalidate computer-based tests;
procedures for school and district staff to update contact information;
access to an online help-desk chat system;
system status website to display operability status of testing components;
access to instances of all computer-delivered tests serving Florida K-12 education programs;
access to order additional test materials and track shipments;
access to packing lists, pallet maps, security checklists (pre-populated with security numbers of secure
materials), and material summaries;
results files and reports;
web-based student practice tests;
self-paced training resources for test setup and administration tasks;
Training center to enable practice for test setup tasks and to run an infrastructure test to test capacity at
testing sites; and an
infrastructure test.
The system must collect and maintain records for students as they move through Florida’s public school
systems. An individual student's test results must be maintained in electronic archive for the life of the
contract and must be returned to the Department at contract termination.
Florida Standards Assessments, ITN 2014-47
50
7.4.8.1
Test Staff Contact Information
It is necessary for the contractor to maintain accurate lists of district and school information and
staff associated with Florida public education testing programs as specified by the Department.
This information must be easy to update by school staff as information changes. This information
must include each user’s security authorization level and provide an accessible log, in viewable
and printable format, of each user’s participation in the system, including type of participation, date
and time of participation, etc. (see also Section 7.4.1).
7.4.8.2
Test Staff Passwords
The system must securely deliver and maintain system access passwords to users as necessary.
The passwords must access a hierarchical permissions structure available for school, district, and
state users. The contractor will work with the Department to determine expiration timelines for
passwords and user accounts.
7.4.8.3
Test Session Requirements
The system must have available during all steps in initiating and delivering tests:
• a school look-up sub-system;
• a searchable, sortable student look-up sub-system;
• rosters of students in each test session.
During the test session, automated backup, recovery, and restart procedures for the system must
be in place. The system will recover data from any unforeseen test interruption and return the testtaker to the point of interruption. The system must also show student status and progress through
the items in the test. The Department should be able to view uploaded student responses realtime while students are completing the assessments in the event of an unforeseen test interruption.
Contractor and department users must have the ability to restart a submitted test. The tests should
only be available during specified administration windows. The features and interface of the
system may be customizable, and the contractor will consider requests for customization from the
Department.
7.4.8.4
Security Requirements
Security of tests and individual test results must be maintained. Access to student test results
must be restricted to authorized Department staff and school district personnel based on a secure,
encrypted, password protected system.
The system must assure proper identification of each student and the accurate matching of the
student to his/her test results. Preferably, the system must ensure that a student can take each
computer-based test just once per login unless authorized by the department. If an error is found
(e.g., student tests with the wrong login information) the system must have the ability to link the test
to the correct student profile.
The ability to make changes to data or processes in any part of the system must be based on a
password-protected, hierarchical permissions structure.
Respondents should respond to each of the following items:
• Describe the security controls over all system aspects.
• Describe the “levels” of security provided in the system.
• Describe how the ability to limit access to specific system functions or modules is provided.
• Describe the log-in security method and assurances to maintain individual confidentiality of
test results.
• Describe the authentication process to ensure that an individual is the person logging into the
system (e.g., the student taking the test is who s/he says s/he is).
Florida Standards Assessments, ITN 2014-47
51
The contractor and Department staff will create a report that includes a checklist of all operable
aspects of the system to be used each invoice period. The contractor will prepare the report listing
any malfunctions or unavailability of the system for Department signoff each period.
7.4.9 Computer-Based Test Administration Overview
The expectations in this ITN are written with the full understanding that computer-based testing (CBT)
capabilities are expanding quickly. The Department is interested in reviewing innovative procedures and
techniques for item development, test delivery, scoring (see Section 7.5.4), and reporting (see Section 7.5.5)
which take advantage of the maturation of technology in educational assessment through the life of the
contract arising from this ITN. Therefore, the respondent may submit a reply that varies from the expectations
herein as long as the solution meets the needs of the Department and the intent of the ITN. Respondents
should agree to work with the Department to amend the specifications, expectations, and requirements to
ensure that the Florida testing program keeps pace with the expanding capabilities of test delivery and
management via computer. Any such variations will be resolved and agreed to during the negotiation phase.
The computer-based test system should be capable of being customized to meet the needs of Florida
students, school and district personnel, and the Department. The reply should indicate procedures used to
ensure that the computer-based tests accurately assess students possessing a diverse range of physical,
sensory, and cognitive abilities; the system should ensure adherence to universal design concepts. The
system should be easy to use for all participants, intuitive, and deliver a consistent “look and feel” for all
computer-delivered test programs provided by the Department. This includes item layouts, navigation tools,
and administrative controls such as managing participant demographic and performance information, as well
as CBT setup activities. The contractor will modify any proposed system to achieve these objectives.
The contractor and Department staff will create a report that includes a checklist of all operable aspects of the
system to be used each invoice period. The contractor will prepare the report listing any malfunctions or
unavailability of the system for Department signoff each period.
If the Department determines that the computer-based system is not performing adequately, exhibits
malfunctions that may significantly affect the student testing experience, or is unable to meet the requirements
listed in this ITN, the contractor will propose a solution to the Department, including providing paper-based
tests at no additional cost, that allows for students to participate in the assessments and for the Department to
gather the necessary data.
7.4.9.1
Hardware and Software Requirements
The system should:
• Support both wired and wireless connections to the system and all eligible devices that meet
the specifications in the table in Section 7.4.9.2. The respondent should also commit to
continued support for future updates similar to those made available on the latest version of
Partnership for Assessment of Readiness for College and Careers' (PARCC) Technology
Guidelines that can be found at (http://www.parcconline.org/technology). Other assistive
technologies may be needed for students requiring accommodations.
•
Support eligible devices with capabilities to “lock down” the device to temporarily disable
features, functionalities, and applications that could present a security risk during test
administration. Features that will need to be controlled during test administration include, but
are not limited to, unrestricted Internet access, cameras (still and video), screen capture (live
and recorded), email, instant messaging, Bluetooth connections, application switching, and
printing. The contractor will develop, deliver, and continuously improve capabilities to “lock
down” the device for the applicable operating systems and browsers.
Florida Standards Assessments, ITN 2014-47
52
•
Support the current version within six months of release and two prior versions of the web
browsers listed below (as of June 2013).
Web Browser Support List:
Apple Safari
Google Chrome
Microsoft Internet Explorer
Mozilla Firefox
7.4.9.2
Eligible Device Specifications
At the time of the development of this ITN, the device specifications that are recommended for
Florida’s public schools are provided below. The respondent’s platform for computer-based testing
should be able to perform, at a minimum, on devices that meet the specifications in Tables
7.4.9.2.A and 7.4.9.2.B.
Table 7.4.9.2.A. Device Specifications for Desktop, Laptop, Netbook, and Thin Client/VDI Computers
Table 7.4.9.2.A Device Specifications for Desktop, Laptop, Netbook, and Thin Client1/VDI Computers
Operating System
Windows
Specifications
Windows 7 or newer
Mac OS
Mac OS X 10.7 or newer
Linux
Linux: Ubuntu 11.10, Fedora 16 or newer
Chrome OS
Chrome OS 19 or newer
Memory
1 GB RAM or greater
Connectivity
Screen Size
Screen Resolution
Input Device Requirements
Headphone/Earphone and
Microphone Requirements
(Some student
accommodations may require
headphones/ earphones (e.g.,
text to speech).
Additional Guidance
Computers must be able to connect to the Internet via wired or wireless networks.
9.5 inch screen size or larger
1024 x 768 resolution2 or higher
Keyboard
Mouse or Touchpad or Touchscreen
The input device must allow students to select/deselect, drag, and highlight text, objects,
and areas. The input device must allow students to enter letters, numbers, and symbols
and shift, tab, return, delete, and backspace. To meet security guidelines, each
Bluetooth/wireless keyboard must be configured to pair with only a single computer during
assessment administration.
Other assistive technologies may be needed for students requiring accommodations.
Headphones/Earphones
Microphone
1 Each
computer operating in a thin client environment must meet or exceed minimum
hardware specifications, as well as bandwidth and security requirements.
2Computers
must accommodate the 1024 x 768 screen resolution minimum without
panning. Some netbook computers may have screen resolutions slightly less than the
1024 x 768 minimum, yet may meet all other minimum requirements. Depending on
netbook model specifics, school technology administrators may be able to reset screen
resolution to meet guidelines.
Florida Standards Assessments, ITN 2014-47
53
Table 7.4.9.2.B. Device Specifications for Tablets
Table 7.4.9.2.B Device Specifications for Tablets
Operating System
Android
Specifications
Android 4.0 or newer
(with 1 GB RAM or greater)
Apple iOS
iPad 2 or newer running iOS 6 or newer
(with 512 MB RAM or greater)
Windows
Memory
Connectivity
Windows 8 or newer
(with 1 GB RAM or greater)
By operating system
Tablets must be able to connect to the Internet via wired or wireless networks.
Screen Size
9.5 inch screen size or larger
Screen Resolution
1024 x 768 resolution2 or higher
Input Device
Requirements
Keyboard
Touchscreen or Mouse
Due to the onscreen space occupied by a tablet’s virtual keyboard, assessments
will require external keyboards for test takers using tablets so as not to limit or
obscure the view of test item content and related functionalities when text input is
required.
External keyboards must allow students to enter letters, numbers, and symbols and
shift, tab, return, delete, and backspace. Tablet touchscreen interfaces can be used
for student interactions with the assessments other than text input, including to
select/deselect, drag, and highlight text, objects, and areas. To meet security
guidelines, each Bluetooth/wireless keyboard must be configured to pair with only a
single computer during assessment administration.
Headphone/Earphone
and Microphone
Requirements (Some
student accommodations
may require headphones/
earphones (e.g., text to
speech).
Other assistive technologies may be needed for students requiring
accommodations.
Headphones/Earphones
Microphone
Florida Standards Assessments, ITN 2014-47
54
7.4.9.3
System Performance Requirements
The system must demonstrate and report that it can meet the following performance requirements
in a controlled Internet access environment, requiring a bandwidth connection of no more than five
(5) kilobits per second (kbps) per simultaneous test taker using a caching solution or fifty (50) kbps
per simultaneous test taker without caching:
• deliver less than one second mean screen refresh rate for 90% of all users; and,
•
ensure that no users have a refresh rate of greater than five (5) seconds.
The respondent should describe the ability to support a substantial number of simultaneous online
users of the system (up to fifty percent (50%) of the current Membership in Florida Public Schools
as determined through the most recent Survey 2 data under the Education Information &
Accountability Services (EIAS) found at http://www.fldoe.org/eias/eiaspubs/pubstudent.asp). The
respondent should consider any services provided to other states when presenting this
information. The contractor must continually monitor and report to the Department the number of
simultaneous online users of the system during student testing periods.
7.4.9.4
Connectivity Requirements
The respondent should consider existing and planned communications infrastructures when
proposing its system. Preferably, school, district, and Department technology architecture and
computing hardware should not have to be replaced.
7.4.9.5
Test Site Set-Up/Certification Requirements
The contractor will develop specifications for, and deliver a system to operate/deliver online tests
and that serves as:
• a secure browser, which both [a] locks access to all other computer and web programs,
applications, and files, and [b] masks and directly accesses the testing host web site; and,
•
a system check, monitoring required computer software and connectivity readiness for testing
and which can be run from individual machines.
The contractor will be responsible for assisting schools in certifying that computers are properly
prepared and can make the necessary Internet connections to successfully deliver tests. The
contractor will develop and make available an infrastructure test, consisting of mock items, that
replicates the size of the largest CBT and is run on the actual testing platform (i.e., users log in to
the testing platform as if they were logging into a live test). The contractor will assist the
Department in developing a guide to walk users through running the trial on machines that will be
used for testing to help them identify technical issues and make adjustments prior to live testing.
The respondent should describe the proposed test site set-up, certification, and system check
measures in the reply.
THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK
Florida Standards Assessments, ITN 2014-47
55
7.4.9.6
Computer-Based Test Presentation Requirements
Table 7.4.9.6 CBT Tools and Features
The following table lists required and preferred tools and features in the CBT platform.
Accommodated features are required. More information about the test accommodations that must
be provided for students are provided in Appendix H.
Table 7.4.9.6 CBT Tools and Features
Tool
Description
Required Features
Calculator
Computer-based calculators will be available to students for the mathematics tests.
Numbers and functions on the on-screen calculator will be activated by mouse-click on
the displayed button, or by keystroke of the appropriate keyboard key. User-acceptance
testing will be required to assure that calculations are performed in the same sequence
and with the same rounding rules applied as the hand-held calculators. Respondents
should describe proposed functionality and features of calculators to be included in
computer-based test forms.
Reference Sheet
The system must be capable of displaying a pop-up reference sheet for mathematics
tests. This will be capable of being enlarged and the Department prefers that, when
open, the reference sheet not obscure portions of the test item.
Ruler
If it is determined that some mathematics test items require the use of a ruler, a pop-up
ruler will be available to students. The ruler will be easily moved and rotated on-screen
by the student.
Preferred Features
Straight Edge
A pop-up straight edge, without measurement increments. The straight edge will be
easily moved and rotated on-screen by the student.
Option Eliminator
A feature for eliminating options. However, the feature cannot eliminate the option by
placing any mark on or in close proximity to the bubble that would be used for marking a
response. It must allow the student to easily read the option after it has been
eliminated. Students must be able to undo the elimination.
Bookmark
Mark test items for review and navigate easily to return to previous items, including
those bookmarked for later review. Bookmarked items should be visible on an item
summary page at the end of the assessment.
Highlighter
Highlight text on all tests and to erase the highlighting. If a student highlights in a
reading passage, the highlighting should remain through all items related to that
passage until the student erases it.
Notes
Allow students to create electronic notes for reading passages if desired. These notes
must not hinder reading of the passage. Notes should travel for each passage and item
associated with that passage, then refresh with each new passage.
Footnote Pop-up
Reading items require a pop-up footnote. That is, if the student clicks on a footnote
number occurring in the passage, the text for the footnote appears. Footnotes must
also appear at the end of the passage.
Florida Standards Assessments, ITN 2014-47
56
Table 7.4.9.6 CBT Tools and Features
Tool
Description
Student
Tutorials/Practice
Sessions
Each student that takes the computer-based test must have the opportunity to learn how
to use the tools, select/enter responses, and navigate the system in a short practice
session with instructions that may be either teacher-directed or independently
conducted, as described in Section 7.3.7.5. The contractor will produce and make
available a web-based practice test for each assessment. The contractor may also
provide access to a selectable practice test at the beginning of any test session
within the platform.
Accommodated Features (Required)
Variable Font
The system must be able to display items and text using variable font sizes. For
example, footnotes for words in passages must appear smaller at the end of the
passage than the text used in the passage.
AdministratorSelectable Zoom
The system must be able to Zoom in increments from approximately 12 point to
approximately 72 point available for the student to zoom or for the administrator to
preset. Graphics must be able to be enlarged also. Any buttons, tools, the
pointer/cursor, and the scrollbar must also enlarge based on the zoom level selected.
AdministratorSelectable
Large-Print Font
The system must allow for selection of a large print font of at least 14 point font and up
to 72 point font. The student must then be able to zoom as indicated above.
AdministratorSelectable Variable
Font and Background
Colors
The system must allow for selection from a pallet of colors for font and background
colors. The Department prefers fonts of black, dark blue, light blue, pink, and yellow
with backgrounds of white, black, brown, or dark blue. Other colors may be
recommended. Any buttons, tools, the pointer/cursor, and the scrollbar must also
change based on the colors/fonts selected.
AdministratorSelectable Screen
Reader
Provide an integrated (preferably password-enabled) selectable screen reader capable
of reading via headset to the student, all text, mathematical symbols, and text within
graphics. The contractor must ensure that the screen reader has one consistent voice
and is not operating-system specific. The contractor must run quality control checks on
all products, test forms, graphics, and items to ensure the screen reader pronounces
words correctly.
AdministratorSelectable Assistive
Devices Integration
Support assistive/adaptive devices commonly available to support computer input and
interaction to persons with disabilities.
7.4.9.7
Computer-Based Test Screen Layout Requirements
The contractor is required to recommend a set of screen layouts for the presentation of computerdelivered test items. (See also Sections 7.3.11.2 and 7.4.8) These layouts must include:
• prominently displayed student identifying information [name, student ID, etc., as specified by
the Department];
•
icons to access the item tools/references noted above; and
•
icons presented with consistent location and look and feel across all screen layout options.
Florida Standards Assessments, ITN 2014-47
57
Additionally, the system must code and capture in the item bank the basic screen layout used
to deliver the item on computer to students.
7.4.9.8
Training Delivery Requirements
The contractor will develop specifications for and produce a training plan to provide initial training
and supply materials to support on-going training for audiences including, but not limited to:
• the Department K-12 Student Assessment program team;
•
other Department staff;
•
district assessment coordinators;
•
school assessment coordinators;
•
school and district hardware/network technical staff; and,
•
school and district test administrators.
The Department prefers the contractor to deliver in-person training to approximately 500 key
Department and district personnel in as many as ten (10) sessions of not more than one-half (½)
day each, held throughout Florida prior to the first implementation of the new computer-delivery
platform, which will occur in the spring of 2015. Additional training should be provided via WebEx
or similar web-delivered, interactive sessions scheduled at least six (6) weeks in advance of the
opening of each online assessment window. These training requirements are in addition to the
training materials described in Section 7.4.7.1.
7.4.9.9
Technology-Supported Item Types
The Department is interested in leveraging empirically-validated item formats which take
advantage of the capabilities of computer technology. Examples of these item types may include:
• Drag-and-Drop – in which students select from multiple possible responses to select one, via
mouse-click or key-stroke combination, and then “drag” it to insert (drop) it in an indicated
blank to correctly complete a mathematics formula, text-editing task, etc. This process may be
repeated for the same test item so that multiple answers are placed into multiple blanks.
•
Rotate and other image transformation items.
•
Other formats as the contractor may recommend or the Department may request.
•
The respondent may include the capacity to support these types of items in the reply.
7.4.10 Customer Service and Technical Support
The contractor will provide customer service via toll-free phone and email between 7:00 a.m. and 8:30 p.m.,
Eastern Time. The service will be staffed appropriately, as needed, with personnel designated to respond to
assessment program calls and perform other Florida assessment program tasks specific to the assessments
detailed in this ITN. Customer service staff must document all communications in a log and furnish it daily to
the Department during each administration or upon request as needed. The contractor will develop a call topic
matrix, with the Department’s guidance, to ensure that calls are responded to in a consistent and appropriate
manner.
Beginning not less than four weeks before and extending one (1) full week after each test administration, the
contractor must use additional customer service personnel to handle computer-based test administration calls,
and personnel who are able to provide advanced technical support must be immediately available for any
issues that cannot be resolved by call center staff. The Department is interested in systems which provide
help desk functionality via telephone and Internet “chat” systems.
The system should employ a Department auditable job-ticket method of tracking issue types, status, and
individual client trouble calls/emails including current aggregate status information for resolution.
Florida Standards Assessments, ITN 2014-47
58
7.5.0 Scanning, Scoring, and Reporting
This section of the ITN provides a description of the scanning, scoring, and reporting requirements for this
project. Section 7.5.2 provides details for preparing school and student information, and Sections 7.5.3, 7.5.4,
and 7.5.5 provide the requirements for the scanning, scoring, and reporting, respectively. The steps
necessary to verify the accuracy of these procedures are described in Section 7.5.6.
The contractor must assign experienced staff to complete the scanning, scoring, and reporting tasks and to
coordinate with the test development, the test administration, and the psychometric staff during various
phases of the scanning, scoring, and reporting. As described in Section 7.7.2, it is anticipated that one of the
four project managers will focus on scoring and reporting and one will focus on computer-based testing. The
one-page vitae for each of the professional personnel must indicate their scanning, scoring, and reporting
experience within the organization to ensure the critical and extra-critical dates may be met.
The contractor’s scoring and reporting staff must coordinate with the development team to ensure the file
layouts accurately reflect the final tests; with the test administration team to obtain information about testing
procedures, schools, and students; and with the psychometric staff to ensure timely file verification and
transfer during the scoring and reporting process.
The contractor must verify the accuracy of all files and reports prior to submission to the Department for
review. The contractor will develop staffing plans to complete its review of files and reports and then provide
the Department time to complete its review.
Because of the secure nature of the personally identifiable student information that is used for this assessment
program, stringent security measures must be in place for the Department and the selected contractor. See
Appendix I for more information on these measures.
7.5.1 Scoring and Reporting Specifications
In April of each year, the contractor will submit the first draft of the Scoring and Reporting Specifications
for all tests within a school year. These specifications provide details to ensure accurate and timely scanning,
scoring, and reporting for the following school year. The specifications are initially drafted by the contractor
using the specifications from the prior school year, updating procedures or products as instructed by the
Department. This initial draft includes anticipated schedules for planning the next school year, details on file
naming conventions, as well as the scanning, editing, scoring, and reporting procedures. These specifications
do not contain confidential information or exact replicas of products; they provide the general requirements
and processing steps using information from the prior year to assist in planning. If the initial draft needs
extensive editing, a second draft must be submitted by the contractor.
These specifications are used to communicate between the contractor and the Department throughout the
school year with details added as necessary during the year to ensure that procedures and products are
sufficiently described and any changes in procedures are documented. The contractor will anticipate having
additional documentation for its internal communications.
At the initial planning meeting, the Department will provide the contractor with the 2013-2014 Scoring and
Reporting Specifications with a list of updates for the contractor to make for the initial draft of the 2014-15
specifications. The reply should clearly indicate how the scoring and reporting staff will coordinate with the
other staff members to enable the contractor to develop the Scoring and Reporting Specifications each April
for the following school year while at the same time conducting the current year’s spring reporting activities.
During the school year, the contractor will maintain an addendum to the specifications noting updates, issues,
or changes to the specifications. This specifications addendum will be used to document changes made in the
specifications during the school year and to update the specifications for the next school year. If necessary,
the contractor may be requested to provide the Department with updated versions of the specifications
throughout the year. The contractor will submit the final specifications addendum at the same time as
the last closeout file of student results for that school year.
Florida Standards Assessments, ITN 2014-47
59
After the initial draft of the specifications has been reviewed by the Department and any edits or comments
have been incorporated by the contractor, development of file layouts and other procedures may begin for the
first administration in that school year.
See Appendix B for the critical and extra critical dates associated with the reports described in this section.
7.5.2 Preparing School and Student Information (MSID and PreID)
1. To aid in the accuracy of reporting results, before the initial wave of reporting the contractor must perform
checks to be sure that school names and numbers are accurate and that each student’s scores are
reported to the correct school. Due to the numerous types of special schools (e.g., home schools and
virtual schools) and numerous students testing in schools that are different than the schools where their
scores are reported, school numbers must often be updated after scanning. The contractor must provide
staff necessary to make updates to student information and school information between scanning and
reporting when conflicting information has been provided. The contractor will assist the Department and
districts to ensure that each student’s scores are reported to the correct school during the initial wave of
reporting.
2. The Department maintains a Master School Identification (MSID) website with active school information.
The Department will determine dates that the contractor will download information from the MSID website
to be used for each administration. Due to multiple waves of reporting for each administration, as well as
reporting annual EOC results, it is important to use consistent school information and to report scores
using the information from the MSID website. The Department will provide the contractor with any
changes or updates to any information on this site, such as changing a school name for the annual
reporting. The contractor will not make changes to school information or add/delete schools unless
directed by the Department.
3. To reduce the amount of information a student must provide during testing, for each administration the
contractor will obtain student information from each district prior to testing. This preidentification (PreID)
information includes student demographic information, requests for special print materials, and school
numbers of the school where the student will test and the school where their results will be reported.
In the past, the Department has obtained PreID information in a text file from each district during one time
period each fall and summer, and two time periods each spring. The PreID information was then loaded
into the CBT system or used to create adhesive PreID labels for answer documents.
The reply should include a description of how the contractor would obtain and update the PreID
information to provide the most accurate results. Respondents may propose other options to adhesive
labels, such as pregridding (preslugging) answer documents. The Department wants to obtain the most
cost efficient methods that will ensure accurate results.
4. The Department will determine the dates the districts will be permitted to submit PreID information, the
number of PreID windows, how and which information may be updated, the rules for accepting or
rejecting records or files, and the final method of preidentifying the answer documents.
5. The contractor will draft the PreID file layouts and the PreID instructions for each administration and
incorporate changes as requested by the Department. The Department will send the final PreID
instructions and file layout to the districts. The contractor will also post the layout on the online
management system as instructed by the Department.
6. The contractor must complete numerous steps to ensure the accuracy of the PreID information submitted
by each district, and the contractor must work with district staff when necessary to ensure that the PreID
files are properly uploaded in a timely manner with just one unique record for each student in the district.
a. First, the contractor must be sure that only schools that are active in the MSID file or approved by the
Department (e.g., a participating private school or community college) are used in each district’s
PreID file. Then, the contractor must ensure that each field in the PreID file submitted by the district
Florida Standards Assessments, ITN 2014-47
60
is completed with valid characters as described in the layout and that the values are reasonable. For
example, if all records submitted for Algebra 1 are coded as first-time takers in a district or all are
coded as female, the contractor must alert the district and permit updates as necessary. Also, the
contractor must compare the numbers of students in each district to the number of students tested in
the previous comparable administration to identify any issues with the number of records in the
district’s file and take corrective action as necessary. The reply should include a description of the
system it will use to ensure the accuracy and reasonableness of these files.
b. Two days prior to the close of each PreID window, the contractor will contact the districts that have
not successfully submitted a PreID file and aid the district to ensure each district is able to submit its
data.
c. During the PreID window, to assist district staff in reviewing the files as read by the contractor, the
contractor creates PreID Frequency Reports with n-counts for various demographic groups for each
district to review in the online management system. The contractor will also generate a PreID
Frequency Report with state totals for the Department to review.
d. The district may submit additional files to make corrections during the PreID window. The contractor
will draft the format of these reports and make updates as provided by the Department. The reply
should include examples of the types of reports that might be used to aid districts in reviewing the
PreID data and methods to make updates and corrections while ensuring the final files include one
record per student with accurate information.
e. After the PreID window closes, the contractor provides the Department with a state PreID file
containing all the accepted records that were submitted by the districts. This file will be in the same
format as the files provided by the districts, one record per student, except with a unique PreID
Number, generated by the contractor, for each student. The PreID Number is used for each test the
student is assigned within that administration and the PreID Number is unique across administrations
for each school year. For example, if a student in grade 8 is taking ELA/L and Algebra 1 in the
spring, that student will have one PreID number for both spring assessments, and that number will
not be used again during that school year.
f.
If a student‘s school is incorrect in the district’s PreID file in (e.g., student moved), school or district
staff will grid the student information on the cover of the answer document or add student information
using the online management system prior to testing.
g. If a student will be tested in a school other than the school used for reporting, PreID labels are sent to
the school where the student will test or the CBT record is set up where the student will test. This is
called the Alternate District/School Number.
h. The contractor will print PreID Student Rosters and include them with the shipments of PreID labels
(or preidentified answer documents) to districts. These rosters list the information on the PreID labels
and an explanation of the codes used on the PreID labels. The contractor will provide hard-copy
PreID Student Rosters with the PreID labels and electronic copies on the contractor's secure website
as specified in the Scoring and Reporting Specifications. Each spring, via a second PreID window to
provide updates closer to testing, the contractor will provide a supplemental shipment of PreID labels
and rosters to districts that update their PreID files.
i.
As permitted by the Department, districts may subsequently edit some of the information using the
online management system or add new students for the CBT system. Students who were not
included in the PreID files that are taking paper tests will not be loaded into the PreID system, and
PreID labels are not sent for new students taking paper tests.
Florida Standards Assessments, ITN 2014-47
61
7.5.3 Scanning/Editing PBTs and Extracting CBT Records
Scanning and editing requirements in this section refer to the PBTs. The contractor’s CBT system must be
capable of creating a file with each student’s information and the exact responses entered by each student,
including the fill-in response and performance tasks, character by character before editing, similar to the scan
files described in this section for the PBTs. The reply should include a description of how the scanning and
editing procedures in this section will be completed, as well as a description of the types of CBT extract files
that will be available for the Department prior to the creation of the scored files described in the next section of
this ITN.
1. For each administration, the test administration manuals describe in detail the test materials provided to
districts and schools and the procedures for returning the materials and completing information in the CBT
system. The contractor will establish procedures to verify that the district and school staff completed
these procedures correctly. The contractor will work with the Department and districts if any anomalies
are found and make corrections as necessary. This includes mistakes made by districts such as returning
to-be-scored answer documents to the wrong site or miscoding a school number on a document.
2. The reply should include a description of where the answer documents will be scanned, the types of
scanners to be used and their capacity during the time this project will be scored, general editing and
proofing procedures of documents scanned, the types of documents and procedures the contractor
proposes to ensure that the correct number of answer documents have been received and scanned to be
scored. In the past districts packaged to-be-scored answer documents with a Document Count Form
(DCF header) on top of documents of the same type. Alternate methods to ensure that the correct number
of answer documents is scanned from each school may be proposed. See Section 7.4.1 for more
information on the return of materials.
3. The contractor will provide the Department with scan files for the PBTs and the extract files from the CBT
system prior to creating the scored files. The contractor will draft layouts for these scan and extract files
and incorporate changes as requested by the Department. The Department will specify the fields in the
file, valid values for the fields, and the contents of each file (e.g., by grade or by subject).
4. The contractor will provide two versions of each scan file. The original scan file contains the exact
information as scanned and the second, the edited scan file, contains the final information from the
scanning system incorporating any edits made as described in the Scoring and Reporting Specifications.
For example, if the student wrote her name on the top of the answer document, but did not grid the
bubbles on the document, the original scan file would have a blank in the name field and the edited scan
file would have the name as entered by the contractor based on the hand-written name. The edit rules
are designed to ensure a student will receive the correct report during the initial wave of reporting. The
reply should include a description of how the edits would be made to original scan file data to ensure
accurate and timely reporting of results. If necessary, the contractor will update the edited scan files to
ensure the correct information is transferred to the first version of the State Student Results file (SSR).
5. If any pages with responses cannot be scanned, all responses key entered into the system must be
checked by a qualified person other than the person who initially key entered the responses.
6. The contractor will maintain edit logs that indicate why records on scan files were edited and by whom.
The logs will indicate the names of anyone who key entered or verified responses.
7. The contractor will provide the original scan files and the edited scan files before scored files are created.
See Section 7.5.6 for more information on the steps for verifying data and file delivery. If any edits were
not made correctly, the contractor will update the edit files.
See Section 7.6.2.2 for information on the records that must be scanned or extracted prior to the
beginning of the psychometric tasks. The contractor’s psychometric and scoring/reporting staff must
coordinate schedules and activities to ensure that all files are transferred as expected (times, names,
layouts, etc.).
Florida Standards Assessments, ITN 2014-47
62
7.5.4 Scoring
The basic types of scored files created by the contractor and history files used and maintained by the
contractor are described in this section. The contractor drafts detailed layouts for each file and then
incorporates changes requested by the Department.
7.5.4.1
Types of Scored Files
From the PBT edited scan files and the CBT extract files, scored files are created. Scored files that
contain data for students throughout the state are called State Student Results (SSR) files. These
files contain information from the scan and extract files, information from the students’ preidentification labels when used, students’ scores, and additional information regarding item (test
question) data and scoring. The districts are also provided a truncated version of the SSR with just
their students’ data, but no item data. This file is called the District Student Results (DSR) file.
1. Before the full-state SSRs are prepared, the psychometric staff (Department and contractor),
begin their calibration and equating activities with a subset of the state records. These files
are called calibration SSRs. Initially they only contain the raw score data, then after the
calibration and equating, they are completed with the other derived scores as specified by the
layout. See Section 7.6.0 for more information on the calibration, equating, and scaling
procedures. Coordination of the details for the transfer of these files from scoring/reporting
staff and psychometric staff is crucial to the timely release of reports and files.
2. Aggregated files contain school, district, and state data based on the scores reported on the
SSRs. These files contain information that is reported to the schools and districts as well as
additional information such as standard deviations of the scores. The files with all the data are
called State Aggregated Results (SAR) files, and the districts receive a District Aggregated
Results (DAR) with the data for the state and their schools and district.
3. After the initial release of files and reports, there are up to three waves of late reporting for
students whose scores were not included in the original release of data, whose records had to
be updated, or whose scores are released after suppression. The contractor will generate late
reporting SSRs, DSRs, and provide student reports in a manner similar to the initial wave of
reporting. SARs and DARs are not created for late reporting.
4. After the last wave of late reporting, the contractor creates and provides close-out SSRs and
DSRs that contain the final information and scores for that administration.
5. After each summer EOC administration, the contractor will create annual SSRs, DSRs,
SARs, and DARs by concatenating the SSRs and aggregating data combined for the fall,
winter, spring, and summer administrations for the school year for subject tests that require
multiple test administrations within a school year. The Department will specify how to handle
any anomalies in reporting across administrations (e.g., a school name changed).
6. For each spring close-out SSR and each annual SSR, the Department inserts demographic
information from its Survey 3 membership file of student data and aggregates the data for
approximately 50 demographic groups. The Department will provide these Demographic
SSRs and SARs to the contractor. From these files, the contractor will create the
Demographic DSRs and DARs and post these files to the online management system for
district-level access.
7. The contractor posts most files to the online management system, but may also be required to
provide some files to the districts on CD/DVDs. See Table 7.5.5 for additional information.
7.5.4.2
Types of History Files
Three types of history files may be used for this project. The Department will provide the
contractor with the layouts and necessary history files to begin the project, and then the contractor
will maintain the files and update the files and layouts as instructed by the Department.
Florida Standards Assessments, ITN 2014-47
63
1. One type of history file may be used for the ELA/L Retake. Depending on policy decisions
regarding scoring, the ELA/L Retake history file may be called the Alternate Passing Score
(APS) file and would provide information on students who were grandfathered into a lower
passing scale score requirement. Students who take the first administration of the Grade 10
ELA/L before standards are set may be grandfathered into a lower passing cut score than set
for other students. If this occurs, for each administration, the contractor will use this file to
determine if a student is eligible for the APS and will update the records for students who test.
The Scoring and Reporting Specifications will detail the matching criteria.
2. The second type of history file will be used for the Algebra 1, Geometry, and Algebra 2 EOC
Assessments for FDOE verification of the testing status (Retaker vs. First-time Tester) of each
participating student. This kind of history file provides information for students beginning with
the first time the subject test is administered statewide. Each administration, the contractor
will use these files to determine if a student is a first-time tester or a retaker. The Scoring and
Reporting Specifications will detail the matching criteria and process for using these files.
Some aggregated data are based on first-time testers only, so the accuracy of the first-time
status for each student is very important.
3. A third type of history file will be used to track scores across grade levels for the Grades 3-11
ELA/L and Grades 3-8 Mathematics students. Starting with the first ELA/L and Mathematics
administration, the contractor will begin maintaining a file with student data for Grades 3 – 10
ELA/L and Grades 3 – 8 Mathematics, called the ELA/L and Mathematics History File.
Scores for up to 7 administrations will be added to this file using the closeout files for each
administration. The contractor will update the file after each administration and will provide it
to FDOE at the end of the contract. The Scoring and Reporting Specifications will detail the
matching criteria and process for using these files.
The contractor will update these history files each administration and provide a copy to the
Department.
7.5.4.3
Types of Scores and Codes on the SSR/DSRs
1. The scoring process begins with the Test Maps that are created by the contractor when a test
form is created. The Test Maps are used to identify item placement in the test book and other
information about the item. The Test Maps include information regarding which items are core
items and field test items, which responses are correct, and the category used for subscores
within each content area.
2. If the Test Map or a core item is determined to have a flaw, the Department may choose not to
include an item in the students’ scores. The contractor must have procedures to provide the
flexibility to edit the Test Map information during the scoring process, if necessary.
3. Excluding field test item data, subject raw scores and raw subscores are then generated by
the contractor.
4. See Section 7.6.1 for information regarding the calculation of scale scores (SS) by the
contractor.
5. Once the SS are determined, the score is classified in one of the Achievement Levels (AL)
by the contractor.
6. Florida statute requires that passing scores be set for all grades and subjects. The contractor
will report the scores as specified by the Department.
7. The ELA/L Retake may have two (2) cut scores for passing, depending on policy decisions
that will be made prior to spring 2014. If so, the contractor will first determine if the student is
Florida Standards Assessments, ITN 2014-47
64
on the APS file described in this section and then determine the passing status for each
student.
8. For each special school, there is a defined School Type number that is used to determine
which reports are prepared for a school and whether the results are included in district and
state aggregated data. For example, home schools are school type 10, and they are not
included in aggregated results within a district.
9. Scores are not reported for some answer documents (e.g., students who did not answer
enough questions to have a score reported). The student Score Status Flag determines
whether the student scores are reported and aggregated. The Scoring and Reporting
Specifications provide details on these codes.
10. Many codes are related to each other and all values must be internally consistent. For
example, the School Type Codes and the ESE Primary Exceptionality codes are used to
determine the Curriculum Group and the Standard Curriculum Codes. The Ethnicity
Code and the Race Codes are used to generate the Report Race Code. Students who use
large print and Braille documents must have specific Primary Exceptionality Codes and
accommodation codes that must be checked and updated by the contractor if necessary.
11. Numerous other codes (approximately 15-20 for each subject) are provided on the SSRs.
These types of codes may change occasionally based on changes in the assessment program
and needs of the schools and districts. The contractor must be flexible in modifying the codes
and including additional codes as needed to aid in the accurate and timely reporting of useful
results.
12. After the scores and codes are verified by the contractor, the first version of the full-state State
Student Results (SSR) file is provided to the Department. The contractor will verify that the
data were properly transferred from the corresponding edited scan file, that all scores, flags,
and codes were set correctly, and that all fields contain valid values as defined by the layout.
See Section 7.5.6 for additional information on the steps for verifying mock data, early district
return data, and state data.
7.5.4.4
Handling Special Print Documents
1. This project includes Large Print, Braille, and One-Item-Per-Page (LP, BRL, and OIPP)
special print test materials. The contractor is responsible for numerous tasks to prepare the
special print documents for scoring and reporting. The scores for students using these
materials are included in the initial release of reports.
2. The Scoring and Reporting Specifications will include the steps the contractor follows from the
time these materials are unpacked until the scores are reported. If any to-be-scored answer
documents are not packaged correctly, the contractor is responsible for properly preparing the
materials or contacting the district if necessary.
3. The contractor is responsible for the transcription of the responses from the LP book to the
regular print book according to guidelines provided by the Department. The contractor may
suggest ways to complete this process using the CBT system to expedite the scoring process.
4. The contractor is responsible for transcribing Braille answer documents to regular print or into
the CBT system. If a student uses an OIPP document, the contractor is responsible for the
transcription of the responses from the OIPP book to the regular print book according to
guidelines provided by the Department. Transcription services may be subcontracted, but all
transcriptions must be completed in time to report the students’ results in the initial reporting.
This process must be carefully monitored and transcription must be double checked by a
person other than the person who initially transcribed the responses.
Florida Standards Assessments, ITN 2014-47
65
5. The contractor will create a Special Document Tracking Log to inventory and track the
processing of all special print documents. The contractor and the Department will determine
the format of the log to track the answer document information and times for the log to be
provided to the Department.
7.5.4.5
Handscoring
Handscoring refers to the processes necessary for determining the rating of a student’s response
on the writing prompts or on the performance-based tasks in ELA/L and Mathematics that
respondents may propose. Both paper-based and computer-based tests will contain performance
tasks, and so the Contractor must have the capability to score both hand-written and key-entered
student responses. The Department requires handscoring processes that are reliable and valid, as
well as efficient in terms of time and expenditures. This section describes procedural, design, and
implementation requirements for scoring student writing and performance tasks. Respondents
may propose other processes and procedures that accomplish the tasks in an equally reliable and
valid manner, and are at least as efficient as those outlined below. The final processes will be
agreed upon during the negotiation phase. See Appendix J for more details regarding
handscoring.
1. Students’ constructed responses should be scored by trained readers using imaging
technology. Respondents should provide a detailed explanation of the system that will be
used.
2. Respondents should also provide a detailed description of how the security of the test items
and student responses will be maintained throughout the handscoring process.
3. Trained professional readers will score written answers using criteria established by the
Department in coordination with the contractor. Readers must have a college degree and
must qualify to score Florida student responses by accurately scoring a series of training and
qualifying papers. The readers must be monitored carefully and continuously throughout
scoring for their accuracy (validity) and reliability (inter-rater agreement). In addition, team
leaders must “read behind” each reader to determine the accuracy of their scoring. Readers
must also be periodically given papers that have been previously scored to determine the
accuracy of their scoring (validity responses). Quality control metrics will be established by the
Department after contract execution. Student responses for constructed-response items in
ELA/L tests in grades 3 and 10 and for Algebra 1 will be scored by at least two scorers (100%
second read). For all other ELA/L and Mathematics performance tasks, responses will be
scored by one reader, with twenty percent (20%) of these papers having a second reader. For
these papers that are read a second time, if the two (2) readers’ scores do not match, a formal
resolution process will be established in order to obtain exact or adjacent scores for quality
control purposes only.
4. Handscoring may be done at the contractor’s established scoring site(s). Respondents may
also propose solutions that include distributed scoring. The scoring sites must be secure
facilities that employ procedures and technology to control access and maintain the security of
materials. Access to the sites must be limited to full and part-time employees and authorized
visitors. Secure materials must be accounted for, and computers containing secure materials
must be secured from unauthorized access or copying of files.
5. The contractor should incorporate the following requirements into written handscoring
specifications prior to scoring. The handscoring specifications will be a detailed guide to
conducting handscoring that will be used by the contractor’s handscoring managers and the
Department. Draft handscoring specifications must be delivered by the contractor for review
and approval at least three (3) months before the first test administration. The handscoring
specifications should include the following:
Florida Standards Assessments, ITN 2014-47
66
a. Handscoring schedules.
b. Site requirements (includes security and access during scoring).
c. Scoring design (includes expected number of reads; groupings of items assigned to
readers; assignment of first and second reads).
d. Personnel (includes minimum requirements, such as education and experience, for
scoring directors, team leaders, readers; number of personnel; security agreements).
e. Rangefinding meetings (includes staffing, meeting procedures).
f.
Rangefinder review meetings (includes staffing; meeting procedures).
g. Training and qualifying materials (includes listing/description of required materials;
annotations for training and qualifying rounds; review and approval process).
h. Validity sets (includes number and type of validity responses; transcription).
i.
Scorer degree verification procedures.
j.
Training procedures.
k. Qualifying standards required of scorers before live scoring of student work.
l.
Handscoring reports (includes listing/description of handscoring reports; distribution).
m. Scoring process (includes administration of validity sets; rules for determining and
assigning resolution reads).
n. Monitoring for scorer drift and other sources of scorer inconsistency and retraining
(includes read-behind guidelines; use of daily and cumulative reader reports).
o. Processing requirements (includes programming and design requirements for
implementing online handscoring of images).
p. Providing electronic copies of annotations used for training materials, calibration, and
recalibration of readers,
6. Department staff must be provided on-line access to all handscoring systems and reports and
will communicate frequently with the contractor throughout the scoring process. During live
scoring daily handscoring status calls must be held between the Department and the
contractor.
7. At the end of handscoring, the contractor must produce a report that captures at least the
following results for all prompts and performance-based items and tasks that required
handscoring:
a. Item or task ID, the form number, and sequence number
b. Total number of reads for each item
c. Inter-Rater Reliability statistics including number and percentage of perfect agreement,
number and percentage of adjacent agreement, and number and percentage of nonadjacent agreement
d. Validity Statistics including the number of validity reads, number and percentage of
perfect agreement, number and percentage of adjacent agreement, and number and
percentage of non-adjacent agreement
e. Distribution for scores and nonscorable codes from handscoring
Florida Standards Assessments, ITN 2014-47
67
f.
The mean, median, modal score, and score variance and standard deviations for scorable
responses
g. Final quality-assurance monitoring reports from the handscoring system that include
rescoring and retraining rates
h. Scoring notes handscoring.
The handscoring statistical data will be provided in an electronic data file in a format
agreed upon by the contractor and the Department.
Provide copies of all materials used in training and recalibration.
8. It is expected that Department staff and Florida educators will be involved in the process to
select student responses that will be used in training and qualifying scorers.
7.5.5 Reporting
This section includes requirements for reporting results to students, schools, and districts. This includes the
posting of the scored files (DSRs, SARs, and DARs) described in Section 7.5.4, the release of hard copy
student reports and PDFs of other reports described in this section, and the creation of files produced for the
Department’s website.
Although the reports used in the past may be used as a model for these reports, the Department would
consider changes to the report design to be more attractive or to present the data more clearly. The reply
should include a description of how the respondent will design the reports, sample reports, and any
suggestions for improvements to the current models. The report design is reviewed by the Department after
each administration. If requested, the selected contractor will make modifications to the design after an
administration and prior to the next administration. In addition, if high-level changes (change in Commissioner
of Education, significant policy change) occur after designs have been approved; the contractor will make the
appropriate modifications to the report design.
If the respondent does not have qualified, professional staff devoted primarily to visual graphic arts and report
design, a subcontractor(s) should be identified to provide these design services. The reply should include the
names and documentation for the qualified staff or the subcontractors who will provide these design services.
The final report design will take numerous iterations to ensure that all types of scores have been clearly
presented. The Department will determine the final colors, design, and content of all reports.
A general description of each type of report is provided below. Most of the reports are designed by test, that
is, one (1) report form for English Language Arts/Literacy, one (1) for Mathematics, one (1) for each EOC
subject test, and one (1) for Retake.
Student Reports
• For each student that tested by test.
• Includes the student’s scores (e.g., scale scores, achievement levels, raw subscores, passing status),
information about the scores (e.g., passing scores, points possible), and comparison data (e.g., state
means).
• Two (2) hard copies (one (1) for parent and one for school) shipped to districts, packaged by school.
• Printed front and back with up to eight (8) colors plus black.
• Some text translated to Spanish and Haitian Creole. The Spanish and Haitian Creole translations on the
student reports must be created by someone familiar with the terminology used in Florida’s schools and
proofed by an independent party, that is, someone not involved in the original translation.
• Delivered with cover memoranda and packing lists following specifications similar to those for distributing
test materials.
• Hard copies produced for initial reporting and late reporting waves.
Florida Standards Assessments, ITN 2014-47
68
For ELA/L and Mathematics test, the student report will include historical scores that are up to seven (7)
test administrations with a bar graph.
• At all grades/subjects containing performance tasks, each student will receive a report showing his/her
response to one performance task in each grade/subject. The report will include a copy of the answer
given, the number of points earned on the item, descriptive information about the question, and scores
(points earned) earned on all of the performance items.
School Report of Students
• For each school by subject test.
• Includes the scores for all students in the school that tested.
• One (1) PDF by school posted on the online management system for school access for initial reporting
only.
• One (1) PDF of all schools in a district posted in the online management system for district access for
initial reporting only.
• One (1) color plus black.
• PDFs produced for initial reporting and late reporting wave.
• Reports created for the late reporting wave include only the students with reports in that wave. The
reports will denote that they are part of late reporting and the reports will be delivered to districts on
CD/DVD.
District Report of Schools
• For each district by test.
• Includes aggregated data for the schools in the district (mean scores, percent passing, etc.).
• One (1) PDF by test by district posted in the online management system for school and district access.
• One (1) color plus black.
• One (1) Excel spreadsheet for all districts and schools by test with the same data as the PDFs for the
Department to use for its web postings. This may be a slightly modified version of the PDF without color.
• PDFs produced for initial reporting only.
State Report of Districts
• For each test.
• Includes aggregated data for the districts in the state.
• One (1) PDF by test, posted in the online management system for school and district access.
• One (1) color plus black.
• One (1) Excel spreadsheet by test with the same data as the PDF for the Department to use for its web
postings. This may be a slightly modified version of the PDF without color.
• PDFs produced for initial reporting only.
District Summary
• For each subject.
• Includes the aggregated data for each grade level tested in the district and the state.
• One (1) PDF by subject by district posted in the online management system for school and district access.
• One (1) color plus black.
• PDFs produced for initial reporting only.
State Summary
• For each subject.
• Includes the aggregated data for each grade level tested in the state.
•
Florida Standards Assessments, ITN 2014-47
69
• One (1) PDF by subject posted in the online management system for school and district access.
• One (1) color plus black.
• PDFs produced for initial reporting only.
CD/DVD/Jump Drive with Student Report Shipments
• The initial shipment of Student Reports includes a CD, DVD, or jump drive with the DSRs and DARs and
their layouts.
• The late reporting shipments include a CD, DVD, or jump drive with the DSRs and layout and the PDFs of
the School Reports of Students for the late reporting students.
The Scoring and Reporting Specifications provide specific details on handling special student score flags and
school types and the specific information and scores provided on each report.
The packaging and distribution requirements for test materials (see Section 7.4.1) also pertain to the packaging
and distribution of the reports.
Table 7.5.5 below shows these reports grouped as they are released to districts and schools.
Table 7.5.5 Results Released to Districts and Schools
Table 7.5.5 Results Released to Districts and Schools
Results
Release
Number
Released To
Method of Release
Contents of Release
To Districts
Online Management
System
District Student Results File (DSR .txt)
School Report of Students (PDF of all students in district)
2
To Districts
Online Management
System
State Summary (PDF)
District Summary (PDF)
State Report of Districts (PDF)
District Report of Schools (PDF)
State Aggregated Results File (SAR .txt)
District Aggregated Results File (DAR .txt)
3
To Schools
Online Management
System
State Summary (PDF)
District Summary (PDF)
State Report of Districts (PDF)
District Report of Schools (PDF)
School Reports of Students (PDF of own school only)
4
To Districts and
Schools
Shipments to
Districts
5
To Districts and
Schools
Waves of Shipments
to Districts
Late Reporting Student Reports (hard copies)
Late Reporting School Reports of Students (PDFs on
CD/DVD)
District Student Results Files (DSR .txt)
6
To Districts and
Schools
Shipment to Districts
Writing Images (PDFs on CD/DVD) (spring only)
Online Management
System
Demographic SARs, DARs, and DSRs
1
7
To Districts
Florida Standards Assessments, ITN 2014-47
Initial Release Student Reports (hard copies)
CD/DVD with DSR, SAR, DAR and Layouts
70
7.5.5.1
Initial Reporting Guidelines
The Department coordinates the schedule for the release of data to the public. The contractor must be
flexible with the reporting release schedule. The general guidelines are provided in this section.
Detailed guidelines are specified in the Scoring and Reporting Specifications.
The basic steps for the initial reporting of full-state data for Department review are listed below. The
Department may request slight modifications to these steps.
a. The contractor posts the full-state scan and edit files version A for the Department on the
SFTP site.
b. If necessary, the Department provides an issue log for the state scan and edit files.
c. The contractor posts the full-state SSR version A for the Department and any independent
checkers specified by the Department (see Section 7.5.6).
d. Contractor provides data and recommendations to the Department based on the review for
anomalous scores; the Department will review and identify any data to hold for investigation.
e. If necessary, the Department will provide an issue log for the SSR files.
f.
The contractor posts the full-state SSR version B and the SAR version B (no version A)
g. If necessary, the Department will provide an issue log.
h. If necessary, the contractor corrects and reposts SSRs and SARs until approved.
i.
The contractor’s psychometric and scoring/reporting staff verify the accuracy of the final files.
j.
As requested for initial checking, the contractor creates DSRs and DARs and posts on the
SFTP site for the Department review and updates as necessary until approved.
k. During this process, the Department will identify a version of the files to be used to check the
shipment of student reports and CD/DVDs. The contractor ships a sample of this shipment to
the Department to check.
l.
If requested, the contractor posts additional samples of district and school PDFs of the reports
on the SFTP site.
m. After the PDFs are approved by the Department, the contractor will create all DSRs and DARs
and reports on the SFTP site and, for state access only, in the online management system.
n. After Department approval and guidance, the files and reports will be available for districts and
schools in the online management system.
o. The student reports and the CD/DVDs will be shipped for delivery by the critical date in the
contract.
7.5.5.2
Late Reporting Guidelines
For each administration, various reasons make late reporting necessary. For example, to-bescored answer documents were sent to the wrong location, updates were made based on the
review of documents for graduating students (see Section 7.5.6.5), or the original student report
was sent to the wrong school or student.
Immediately after the initial wave of reporting, the contractor begins checking scores of graduating
students as specified in Section 7.5.6.5. Any updates will be provided in the first wave of late
reporting.
The contractor will prepare for up to three waves of late reporting. The contractor and the
Department check the late reporting files in a manner similar to the initial reporting wave.
Florida Standards Assessments, ITN 2014-47
71
Depending on the administration and number of inquiries from districts, the Department will
determine the late reporting schedule. The contractor must be prepared to adhere to a rigid late
reporting schedule to resolve any issues with reporting for one administration before the next
administration begins.
The late reporting shipments include the following reports and files.
•
Student Reports – two (2) hard copies as in initial reporting with a note or footnote denoting it
is a late report.
•
CD/DVD with DSR and layout (no DAR)
•
CD/DVD of School Report of Students (not in the online management system) with a note or
footnote denoting it is a late report.
After the initial release of data, the Department opens a website called the Score Inquiry System
for district staff to inquire about missing reports or to ask for a student’s score to be reviewed. This
system is designed so the contractor and the Department can communicate requests to search for
a student’s answer document or report, or to review a student’s score. Maintenance of this system
is the responsibility of the Department.
The contractor must search for missing answer documents in the not-to-be scored materials if
requested by the district to find any documents that were mislabeled or returned incorrectly.
7.5.5.3
Close-Out and Annual Reporting Guidelines
After the last wave of late reporting, the contactor creates close-out SSRs for each administration.
These SSRs contain the final records for all student with erroneous records removed. For
example, if the district/school number was incorrect in the initial reporting and corrected in late
reporting, the close-out SSR contains the corrected record. The incorrect record is removed.
The close-out files are provided to the Department for review and approval, but they are not posted
for districts.
After the summer EOC close-out files have been approved, the contractor creates annual SSRs
following guidelines provided by the Department for tests that require multiple administrations
within a school year. These annual SSRs contain the records for students from each
administration that school year. A few fields may be slightly altered from the close-out files for
each administration, for example, if a school name changes during the school year, the name will
be updated so the SAR shows annual aggregated data for that school using the correct name.
7.5.5.4
Demographic Reporting Guidelines
After the spring close-out SSRs are approved by the Department, the Department inserts
demographic information from its Survey 3 membership file of student data and aggregates these
data for approximately 50 demographic groups. The Department then returns the Demographic
SSR and SAR to the contractor to create Demographic DSRs and DARs.
After the Demographic SARs are created and approved, the contractor also creates Excel
spreadsheets with the data that can be used for the Department’s interactive demographic
websites, which can be viewed here for reference:
https://app1.fldoe.org/fcatdemographics/ and
http://app1.fldoe.org/FEocDemographics/MainMenu.aspx.
The Department will specify the format for the spreadsheets.
After the annual SSRs and SARs are created for the subjects that require the multiple test
administrations within a school year, this same process is completed for those files.
Florida Standards Assessments, ITN 2014-47
72
7.5.6 Verifying Data
Due to the tight timelines and quantity of data, all initial scoring and reporting processes are checked with
mock data prior to the administration. Then, once actual data are available, a small sample of answer
documents are used to begin checking, called early district return data. After all answer documents have
been scanned, the files and reports are checked for completeness, accuracy, and possible irregularities
(cheating) as described in this section. The verification process may require numerous iterations of the files to
ensure that each student’s scores are accurately reported.
The reply should include a description of the procedures to be used to verify the accuracy of data as outlined
in this section. This includes the staffing necessary to complete the mock data process before the
administration begins and the verification of each version of the files given the tight time constraints.
The Department may have independent contractors assist in a review of the procedures used in this project
and verification of the files and reports. Currently the Department has contracts with two independent parties
for this purpose. One (1) contractor is primarily concerned with reviewing the procedures and the
specifications for the scanning, scoring, and reporting with an emphasis on the psychometric aspects and
primarily coordinating with the psychometric staff. The other contractor is primarily concerned with ensuring
the procedures are followed, verifying codes and scores on the scan files, the SSRs/DSRs, the SAR/DARs,
the Student Reports, and the PDFs of the other reports coordinating with the scoring and reporting staff.
The contractor will be responsible for coordinating with the Department’s independent contractors for the
review. This may include touring the scanning site and discussions with the contractor’s scanning, scoring,
psychometric, and reporting staff. During the psychometric activities, the independent contractors may
participate in the meetings and discussions, and during the scoring activities, the contractor may be asked to
provide a portion of the reports and files to the independent contractors when reports and files are provided to
the Department.
7.5.6.1
Checking Mock Data
1. All mock data files, reports, and procedures are generated and delivered using methods as
close as possible to the same way final files will be generated and delivered. This process
begins with mock PreID files, mock PreID labels, mock PreID Frequency Reports, mock
history files, and mock answer documents, and then all headers, files, and reports for the initial
wave of reporting are generated using the mock data. The mock student reports are packed
and shipped as specified by the Department with the media containing the DSR and DAR, and
the mock files and PDFs of reports are also posted to the SFTP site for the Department
review.
2. Actual school information from the MSID file will be used in the mock data process to check
the processing of the special school types and special districts.
3. Mock data cases may be generated using different methods. To check scanning of PBTs,
some mock cases must be manually gridded, but others may be auto-gridded (preslugged)
documents. Some CBT data must be generated by the contractor staff simulating the
processes that a CBT student would use for each form of each test. To have enough cases
to check all scoring and aggregating rules without using too many paper answer documents
and time entering CBT responses, some additional cases may be computer generated,
meaning for PBTs physical documents are never gridded or scanned for those cases and, for
CBTs no one sat at the computer and key entered the responses in the same manner as a
student enters the responses.
4. For PBTs, only the cases to check erasures must be manually gridded and other references in
this section to cases with gridding may be auto-gridded.
Florida Standards Assessments, ITN 2014-47
73
5. For each CBT, the PBT accommodated form must also be checked. Large Print and Braille
paper documents are not used for mock processing; however the regular print answer
documents and the headers are created with the appropriate Large Print and Braille codes.
6. The Scoring and Reporting Specifications indicate the various cases that must be included in
the mock data. To easily check that all values scanned on PBTs and key entered on CBTs
are properly recorded on the files, the mock data will include cases to check the values using
specific patterns (e.g., alternating A,B,C,D,A,B,C,D responses). Cases must be included to
be sure that each value on the SSR, SARs, and reports are accurate. This includes
processing erasures, multigrids, demographic values, score flag codes, school type values,
scores, etc.
7. The contractor must design the mock data files to check all procedures through the release of
results. To reduce the amount of time to review these data and iterations required, the
contractor will complete their review of all mock cases, possible values of codes and scores,
and all reports before providing the data to the Department. The reply should include a
description of how the respondent will complete these mock data, mock files, and mock
reports prior to the first day of testing for each administration.
7.5.6.2
Checking Early District Return Data
1. To begin the verification of actual (live) data, the Department will identify some districts to be
processed first, called Early Districts. The contractor will arrange for the pickup of answer
documents from these districts so they will be scanned first for Department review. The CBT
records for students in these districts will be extracted and provided to the Department for
review.
2. The selected contractor must anticipate that the spring scan check will be completed at the
contractor’s scanning site. The scan check for other administrations may be conducted at the
contractor’s scanning site or at the Department. If the scan checking is performed at the
Department, the contractor will post PDFs on the SFTP site of the headers and answer
documents or ship the required documents to the Department to complete the checking.
3. For the on-site review of answer documents, the contractor will provide the Department the
original scan and the edited scan files at least three (3) business days prior to the on-site
review.
4. The contractor will pay travel expenses for up to five (5) Department staff from Tallahassee to
the contractor’s scanning site to complete this checking. These expenses must be included in
the reimbursable funds described in Section 7.7.9.
5. During the on-site review, the contractor will provide computers to Department staff with the
following:
• Access to the CBTs, access to email, access to a high-speed printer, and, if requested,
SAS.
•
Easy access to the answer documents. Batches and stacks will be logically sequenced
with the headers used.
•
LP, Braille, and OIPP documents and updated Special Document Tracking logs.
•
Key Entry verification logs (e.g., double entry of responses if document is unscannable)
6. The Department’s independent contractors may participate in this review or they may visit the
site prior to this review.
7. Depending on the Department staff availability, the on-site review may be completed during
one week or in two consecutive weeks.
Florida Standards Assessments, ITN 2014-47
74
8. The reply may include additional and alternate methods to aid in meeting the data verification
and reporting requirements in the time allowed.
7.5.6.3.
Checking Data for Anomalous Scores
Before data are released to the public, the contractor must review the student and school scores
for anomalous data, that is, data that may not be a valid measure of student achievement. The
contractor must review the data for testing irregularities, such as a student copying another
student’s answers or an administrator changing answers.
If the respondent does not have staff to perform these analyses, a subcontractor should be
proposed. The Department will approve the subcontractor and final procedures to be followed for
this checking.
Anomalous scores would include the following.
• Students in the same testing group have similar response patterns
•
A student has an excessive number of erasures from wrong answers to right answers
•
Students in a testing group have an excessive number of erasures from wrong to right
answers
•
There was an unusual increase in school performance from a comparable administration
The guidelines and procedures used to perform the anomalous score checking will be refined by
the Department with the contractor’s assistance.
If an irregularity is found in the data, the Department may request for the contractor to hold (i.e., not
release) some data. This is handled using the Score Status Flags between the first and second
versions of the full-state SSRs. Schedules must be set to allow time for this review and
updating before the initial reporting for each administration.
7.5.6.4.
Checking Full-State Data – Initial Reporting
1. The contractor will verify that all files and all PDFs of reports have been produced according to
the Scoring and Reporting Specifications before providing them to the Department. In addition
to the checks described in this section, the contractor’s psychometric staff must also approve
the final SSRs and SARs before final reports are generated and data are released to the
public.
2. The verification begins by checking that the DCFs have been completed properly so student
reports will be sent to the correct school. If the school identified on the header is different from
the school on the student’s answer document, and both are active schools, the contractor
must identify the correct school for the student by contacting the district, if necessary.
3. The contractor will compare the edited scan files to the SSR and then the SSR to the SAR,
and compare reports to the SSR and SAR to be sure all information (scores, codes, school
numbers, etc.) is accurate and flowing through the systems correctly before submitting to the
Department for review. The contractor will verify that all scores, flags, and codes were set
correctly, and all fields contain valid values as defined by the layout.
4. The contractor will compare the final number of students tested in each school to the number
of students on the PreID files to identify any anomalies, such as a missing box. To ensure all
answer documents have been received for scoring prior to the delivery of the first scan file to
the Department, the contractor will review the numbers of records and contact the Department
and districts, using guidelines defined in the Scoring and Reporting Specifications, of any
concerns. The contractor will assist districts in returning any materials that were not handled
properly.
Florida Standards Assessments, ITN 2014-47
75
5. As specified by the Department, the contractor will also provide sample shipments including
the student reports and the CD/DVDs with files simulating the shipments to districts. The
Department will determine which districts shipments will be included in this verification to
ensure that a variety of reports are checked. They may be different from the districts selected
for early scan if those records are not diverse. As described earlier in this section, the
contractor may also be requested to send some reports to the Department’s independent
contractor.
6. Before the files and reports are available in the online management system, the contractor and
the Department will verify that the correct reports and files are available for district and school
users prior to the release of the data. The contractor will provide documentation to show that
the correct files and reports are being released as specified in the reporting schedule.
See Section 7.5.5 for more information on the reports.
7.5.6.5
Checking Graduating Student Scores (Manual Rescore)
Due to the high stakes of the tests for graduating students, the contractor must perform additional
checks for graduating students who test on paper for Grade 10 ELA/L and for Algebra 1. The
checks must be completed so any updates may be included in the first wave of late reporting.
1. For the tests required for graduation, the contractor must identify student in each grade who
failed by 5 or fewer Scale Score points or 15 Developmental Scale Score points for
Manual Rescoring. If there is an APS cut score established for the assessment, the APS cut
score must be used for selecting the students.
2. For each of these students, the contractor will review each answer for the subject failed and
verify that the items have been scored as the student intended. For example, the contractor
will look for unintentional stray marks in the answer bubbles that do not appear to be an
intentional grid, will confirm the hand written name on the document is the same as on the
PreID label if one (1) is used, and will also confirm that no incomplete erasures in the griddedresponse columns were recorded as a mark (e.g., double gridding because of incomplete
erasure). The item responses will be compared to the values on the SSR for any irregularities
in editing or processing.
3. If any editing, scanning, or processing irregularities are detected, or if an incomplete erasure
would change an item response, the student’s paper must be rescored with the updated
response(s), and a new SS must be calculated to determine the passing status of the student.
If the student passes, the record must be updated for the first wave of late reporting. If not,
no change is made in the record.
4. A new student report will be generated and shipped to the district and the student will be
included on the School Report of Students sent with the other late reporting wave 1 student
data, if any.
5. The contractor will provide the Department with the number of answer documents reviewed by
subject and the number that will have a Passing Status changed to Y when the new files are
submitted.
6. This process will be completed within sixty (60) days of the original release of scores for the
administration.
7.5.6.6
Checking Late Reports
The contractor and the Department check the late files, reports, and CD/DVDs in a manner similar
to the initial files and reports, although the files and reports produced are slightly different. The
Department checks the files and reports after the contractor.
See Section 7.5.5.2 for information on the procedures for the late reporting waves.
Florida Standards Assessments, ITN 2014-47
76
7.5.6.7
Checking Close-Out and Annual EOC Files
The contractor checks the close-out files by comparing each to the files previously generated for
the administration to ensure that records reported to the incorrect schools have been removed and
any scores or information for students have been updated or added as necessary.
The contractor will ensure that there are not erroneous duplicate records caused by the multiple
waves of reporting on the close-out files. The contractor must review the Late Reporting issue logs
to ensure erroneous records have been removed from these files. The Department checks the
files and reports after the contractor.
The contractor checks the annual EOC files similar to the initial files, ensuring that the firsttime/retaker status codes are properly set. The Department checks the files and reports after the
contractor.
7.5.6.8
Checking Demographic Files
Each spring the contractor will check the Demographic SSRs created by the Department to verify
that no records or scores were changed; only Survey 3 data will have been inserted. The
contractor will verify that only valid values are in the Survey 3 fields before creating the
Demographic SARs.
The contractor will check the Demographic DSRs, SARs, and DARs prior to release in a manner
similar to the initial release of the files. The Demographic Excel spreadsheets must also be
checked by the contractor to be sure they contain the same information as on the SAR. The
Department checks the files and reports after the contractor.
See Section 7.5.5.4 for information on the procedures for the demographic files.
7.6.0 Scaling, Equating, Scoring, and Special Psychometric Studies
During the timeframe of this ITN, the contractor will be responsible for drawing samples from the populations
of students, for calibrating field-test items, for constructing the new test forms for the future administrations,
and for Scaling, Equating, and Scoring (SES) of the assessment system. The system is complex and requires
the contractor to provide knowledgeable and experienced staff, including psychometricians and programmers.
Other than the annual regular psychometric operations, such as sampling, test construction, and SES for the
assessment system, the contractor will conduct a set of special psychometric studies for these assessments,
described in Section 7.6.3.6. The contractor will also cooperate to conduct another set of psychometric
studies as Florida transitions from FCAT 2.0 and EOC Assessments to the Florida Standards Assessments
(FSA) systems. As the Department transitions to the FSA, the contractor will assist the Department in the
establishment of FCAT 2.0 and EOC equivalent score scales, appropriately calibrate items in the new
assessments, properly combine performance indicators, construct meaningful score scales, possibly establish
vertical scales for ELA/L and Mathematics assessments, assist in defining achievement level descriptors, build
cut scores with criterion-referenced meaning on the new scales, link the performance of Florida’s students to
the scales of other assessments to make comparisons, and conduct other special analyses that may be
required by the Department or required for a peer review under the federal Elementary Secondary Education
Act (ESEA) program. This section of the ITN provides descriptions of related services and products and the
expected quality assurance steps that must be employed to ensure that all tested students receive
assessment results that are valid, reliable, and accurate.
Florida uses a Technical Advisory Committee (TAC) composed of selected Florida District Coordinators of
Assessment and selected experts from around the nation representing the areas of measurement and
statistics, students with disabilities, English language learners, and related alternate assessments. The TAC
members listen to issues and offer guidance on implementation of Florida’s assessment program. Currently
Florida’s TAC meets twice a year – in March and in November. The contractor is responsible for convening
and conducting these meetings.
Florida Standards Assessments, ITN 2014-47
77
The contractor may utilize proprietary software for any of the annual regular psychometric activities or the
special psychometric studies for the assessment system during the transition to the FSA, but is required to
provide this software for use by all relevant staff, including the Department and any subcontractors. Whatever
software is selected must be able to handle the large number of test cases, including large calibration
samples, and also accommodate full-state calibration with numbers of records in excess of 200,000 at a time.
The contractor will be responsible for assembling a psychometric team to support work performed under this
contract consisting of the contractor’s psychometricians and programmers, and at least one (1) outside
entity subcontracted to participate in all SES activities to independently perform these particular
operations and verify all results. The Department’s psychometric team consists of two psychometricians plus
three graduate interns pursuing their doctorate in educational measurement to be part of the team replicating
all SES activities and overseeing the entire process.
A separate procurement is being issued for the FCAT 2.0 and Florida EOC Assessments for subjects not
covered as part of this ITN (Science, Social Studies, etc.). The contractor selected as part of this ITN process
should anticipate coordinating with the contractor from the FCAT 2.0/Florida EOC Assessments procurement
for some of the tasks described in this scope of work.
7.6.1 Assessments
7.6.1.1
Florida Standards Summative Assessments (Scaling, Equating, and Scoring)
1. The contractor will develop and maintain test construction specifications describing the details
of test construction, including content and psychometric guidelines for building equivalent
forms. Item selection will be based on matching target test characteristics, test information
functions, and standard error curves in addition to meeting other content and psychometric
requirements, plus further constraints.
2. For the scaling of items, the Department prefers to use a three-parameter logistic Item
Response Theory (IRT) model for multiple choice items, two-parameter logistic IRT model for
gridded-response items, and two-parameter generalized partial credit model for constructedresponse items. For the equating of the tests to their base year scales, the Department
prefers to employ test characteristic curve equating using the Stocking & Lord procedure and
triangulating the equating with IRT based other equating procedures, such as Haebera,
Mean/Mean, and Mean/Sigma. For scaling student performance, the Department prefers to
use Maximum Likelihood Estimation to score examinee response patterns.
For
dimensionality, the Department supports using a variety of item types measuring various
student skills on the Florida Standards, but maintains validity for a single dimension. However,
these are just the Department’s preferences for scaling, equating, scoring, and dimensionality.
The contractor must conduct thorough research studies and collect empirical evidence to
determine whether these options are psychometrically defensible, whether items measuring
various constructs can be combined into a single assessment instrument and scored, and
whether the preferred options can be implemented operationally to obtain assessment results
that are valid, reliable, and accurate. If research and empirical evidence(s) suggest the use of
a different set of techniques in scaling, equating, and scoring of the assessments, the
respondent should provide a detailed description of the measurement models proposed for
use in calibration, equating method for creating interchangeable scale scores, and scoring
algorithm for placing examinees onto a score scale. The contractor will be responsible for
designing sampling plans, drawing the samples from populations of examinees, and supplying
the data files to the scaling and equating team described in Section 7.6.2.1 so that the
required analyses can be performed in a timely manner. The contractor will be responsible for
calibrating the tests, placing item parameter estimates onto the base year’s scale, and
providing the final scaled item parameters for the purpose of scoring. After the Department
approves the final scaled item parameters, the contractor will score student responses, the
Florida Standards Assessments, ITN 2014-47
78
entity subcontracted to participate in all SES activities will independently verify the scale
scores, and the Department’s psychometric team will replicate and verify the scale scores.
3. For the Assessments in Algebra 1, Algebra 2, and Geometry, the contractor must construct
multiple forms of these assessments by following content and psychometric requirements as
indicated in the test construction specifications. As for the scaling, equating, and scoring
process for these assessments, the Department desires to follow the same process as
indicated in the previous paragraph, but since there are multiple forms of the assessments at
this time, the Department prefers to concurrently calibrate these forms so that all item
parameter estimates can first be placed onto a common scale, and then, equating these item
parameter estimates to the base year scale via anchor items. In other words, equivalency of
the multiple forms of new tests is verified through a concurrent calibration, plus post-equating
via common items between the item bank and the new test forms. However, as indicated
earlier, the contractor must show evidence that the Department’s preferences are
psychometrically defensible and operationally feasible. The respondent may include in the
reply a different proposal for scaling, equating, and scoring of these assessments to obtain
assessment results that are valid, reliable, and accurate. Also, when multiple forms of these
assessments are being created, the contractor must construct the new test forms that are
sufficiently different in an effort to discourage cheating and to ensure a valid measure of
student learning. In addition, the department will consider the use of pre-equated forms for
administrations other than the spring administration.
4. Because every effort is made to develop items and construct tests according to universal
design principles, we anticipate all Braille versions of the assessments will contain the same
items as the regular version of the tests. If they do not contain the same items, the contractor
will develop a separate scoring process to be approved by the Department and used by the
contractor and subcontractor.
7.6.1.2
Retake Assessments (Scaling, Equating, and Scoring)
The contractor’s content and psychometric teams will work together with the Department’s content
and psychometric teams to select retake test forms, which may consist of a previously-used intact
form, or another proposed design that yields a test of similar difficulty to the operational form. The
inclusion of performance tasks in the retake form may be proposed, however, consideration will
also be given to forms that do not include performance tasks.
The contractor will administer the selected test form and will score student responses. The entity
subcontracted to participate in all SES activities must independently verify the scale scores
generated by the contractor. The same procedure will be repeated to select different forms for
subsequent years.
7.6.2 Scaling, Equating, and Scoring (Psychometrics)
1. The contractor will construct Scaling, Equating, and Scoring (SES) specifications for the assessments.
The specifications will describe the specific steps for implementing the scaling, equating, and scoring
requirements and will explain how, by when, and by whom all of the necessary steps will be completed
and verified. The specifications will also address the steps, procedures, and software required for
replication of the scaling, equating, and scoring procedures by the Department. The specifications will be
reviewed by the Department and TAC members, if necessary, and updated by the contractor for each
subsequent administration. The Department will provide the contractor with copies of existing
specification documents.
2. Before operational SES activities start, the contractor and other members of scaling and equating team
must conduct dry-run SES activities on mock data files to make sure all operations laid out in the SES
specifications can be successfully executed during the operational SES process. For this purpose, the
contractor will prepare “mock” data for the assessments using the State Student Results file (SSR), as
Florida Standards Assessments, ITN 2014-47
79
indicated in Section 7.5.6.1. The purpose of this mock data will be to provide a vehicle for the SES dry
run; i.e., a practice check of the operation of computer programs used in scaling, equating, creating
adjusted item parameters, and scoring. The psychometric staff for the identified calibration team will be
available for daily discussions and consultation throughout the parallel dry-run calibration activities.
3. It is the responsibility of the contractor to keep all the documents/electronic files from the scaling,
equating, and scoring activities, such as the classical and IRT item statistics, alternative equating
solutions summarizing the reasons why anchor items are removed or kept in the anchor set, the impact of
decisions on the transformation coefficients and scale scores, and the other pertinent decisions made by
the Department. The format of these documents must be proposed by the contractor and mutually
agreed upon by the contractor and the Department.
4. The SES specifications must include how the contractor’s subcontractor will independently verify the
scaling, equating, and scoring operations. For the entire SES process, the contractor will provide all of the
resources, including software, staff support, and data files to permit parallel calibration, equating, and scoring
of data during the same time period that contractor are producing the scaled item parameter estimates.
7.6.2.1
Scaling and Equating Team
The contractor will assist the Department in coordinating the work of the scaling and equating team
during dry-run activities and regular administrations each spring. The contractor is required to
subcontract with at least one outside entity to work on the scaling and equating team during all
SES activities. The contractor’s, the subcontractor’s, and the Department’s psychometric staff will
each independently perform SES tasks to verify results. The specifications will clearly describe the
scaling and equating team consisting of the Department’s, the contractor’s, and third party’s
psychometricians and programmers, as well as the roles assigned to each entity. The
Department’s psychometric team will oversee the entire SES process. Respondents should
indicate in their replies the qualified personnel/subcontractors who will provide these professional
services and include their vitae or other documentation of expertise and experience.
7.6.2.2
Selection of Calibration Samples
Anchor, core, and field-test items are calibrated on representative samples. Thus, the contractor is
responsible for designing sampling plans and drawing the representative samples from the
respective populations, which will include at least 65% of the respective population, including
approximately 90% of Miami-Dade. The contractor will also plan in a way so that a representative
set of approximately 5,000 students take anchor and field-tests forms. The contractor’s sampling
specifications must include analyzing the most recent SSR files and establishing a road map for
gathering the required data proven to represent the demographic characteristics of the state and
student performance. The specifications will also address the steps, procedures, and software
required for replication of the sampling procedures by the Department. The specifications will be
reviewed by the Department and TAC members, if necessary, and updated by the contractor for
each subsequent administration.
Before SES activities start, the contractor must also check and verify the representativeness of the
most recent data, and then provide the files to scaling and equating team members so that
necessary analyses can be performed in a timely fashion.
The contractor must repeat this process for each assessment administered each spring.
THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK
Florida Standards Assessments, ITN 2014-47
80
7.6.3 Technical Oversight and Special Studies
7.6.3.1
Contractor’s Role
The contractor must provide psychometric direction and oversight for all aspects of the assessment
programs described in this ITN. The psychometricians assigned to this project must have
extensive experience in state assessment programs. The psychometricians will plan, conduct,
analyze, and report on test construction, scaling, equating, scoring, and the special studies
described in this section.
7.6.3.2
Technical Report
Each year, the contractor will produce a technical report based on an overall analysis of that year’s
administrations. The technical report will also include a comparison of the characteristics of the
current test administration to previous administrations. The technical report will be reviewed by the
Department and by the Technical Advisory Committee (TAC) prior to completion of the final copy of
the report. The technical report will include a section on field-test administrations. The report will
include tabular and graphic displays of data to illustrate the characteristics and quality of test
scores. At a minimum, the technical report will include the topics listed below in the draft table of
contents.
A. DRAFT TECHNICAL REPORT TABLE OF CONTENTS
B. BACKGROUND OF ASSESSMENTS
C. ITEM DEVELOPMENT & TEST CONSTRUCTION
• Item Development
• Item Review
• Field Testing
• Item Statistics
o Classical Statistics
o Item Response Theory Statistics: Difficulty, Discrimination, and Guessing
o Differential Item Functioning (DIF) Statistics
o Model Fit
o Local Independence
o Dimensionality (factor analyses, etc.)
• Test Construction
D. ADMINISTRATION
• Eligibility
• Administration Procedures
• Accommodations
• Test Security
• Data Forensics Program
E. REPORTS
• Appropriate Uses for Scores and Reports
• Reports Provided
• Description of Scores
• Appropriate Score Uses
• Cautions for Score Use
F. PERFORMANCE STANDARDS
• Setting Performance Standards
• Interim Performance Standards
Florida Standards Assessments, ITN 2014-47
81
G. SAMPLING
• Rationale
• Procedure
• Representativeness
H. SCALING
• Rationale
• Measurement Models
• Scale Scores
• Scale Drift
• Vertical Scaling
• Checking IRT Assumptions (Model Fit, Local Dependence, Dimensionality, etc.)
I.
EQUATING
• Rationale
• Pre-Equating
• Post-Equating
• Development Procedure for Future Forms
• Sources of Error Variances to Score Instability
• Linking Assessments During Transitioning Years
• Linking Assessments to Other Scales for Performance Comparision
J. RELIABILITY
• Estimating Reliability
• Standard Error of Measurement
• Student Classification Accuracy and Consistency
• Reliability for sub groups in the population
• Reliability of year-to-year changes in school means
K. VALIDITY
• Perspectives on Test Validity
• Criterion Validity
• Content and Curricular Validity
• Construct Validity
• Consequential Validity
L. CONSTRUCTED-RESPONSE SCORING
• Scoring Process
• Quality Control
• Handscoring Reliability and Validity
• Rater Effects
M. QUALITY CONTROL PROCEDURES
• Quality Control for Test Construction
• Quality Control for Non-Scannable Documents
• Quality Control in Data Preparation
• Quality Control in Production Control
• Quality Control in Scanning
• Quality Control in Editing and Data Input
• Quality Control for Test Form Equating
• Quality Control in Scoring and Reporting
Florida Standards Assessments, ITN 2014-47
82
N. SUMMARY STATISTICS
• Item level statistics
• Test level statistics
• Review and comparision of raw scores, sub-scores, scale scores, distributions for total
and sub-populations across years to monitor the trends across years and sub-populations
• Review and comparision of percent proficiency rates across years on different populations
• Frequency distributions of student achievement (all students and by sub-group) across
years by grades and subject
• Scale drift indicators
• Correlations of student performance across the various subject areas tested
Following each field test or operational administration, differential item functioning (DIF) analyses,
to detect possible item bias, will be conducted, preferably using Mantzel-Haenszel method. DIF
analyses are conducted for Caucasian, African-American, and Hispanic racial/ethnic groups and by
gender. Values for items resulting from these analyses will be included in the item-banking
system. Changes in DIF values across administrations will be analyzed and presented in the
Technical Report.
The contractor will provide a detailed item analysis for each test administration and provide the
results in the annual Technical Report. There are several item level statistics, such as pointbiserial, corrected point-biserial, fit, etc. available in the current item bank. Thus, the contractor is
responsible for calculating these statistics following each administration and populating the findings
into their respective positions in the bank.
The Department will work with the contractor to determine the contents of the written report. The
contractor will provide several printed copies of the report and also an electronic version in the
format determined by the Department.
7.6.3.3
Transitioning from FCAT 2.0 and EOC Assessments to Florida Standards Assessments
(FSA) – Linking the Scales
As the assessment system transitions to the FSA in spring 2015, the scales of the new FSA will not
yet have any meaning attached. Therefore, the Department requires technical assistance from the
contractor so that the scales of these assessment systems can be linked through a special and
defensible psychometric operation. The contractor will provide a detailed scale linking plan for
transitioning from FCAT 2.0 Reading & Mathematics and EOC Algebra 1 & Geometry to FSA.
Linking through a special psychometric operation may go beyond employing a simple linking
procedure, since each score unit on the two separate scales may have to be linked to each other
so that achievement levels can be determined for student and school accountability purposes. The
Department may directly engage additional contractors, or the contractor’s subcontractor (if
proposed) to triangulate the results.
The plan will include alternative options for linking, how the alternatives will be operationalized,
scoring plans, and procedures. The plan should include a comprehensive and detailed timeline for
all tasks and events. The plan will be reviewed and vetted by Florida TAC members after
submission, and may require revision as a result of this review. This review will address the
appropriateness and validity of the psychometric procedure selected for linking, as well as the
validity, reliability and defensibility of the linked scores.
The respondent’s reply should describe available psychometric methods for establishing interim
scores for high-stakes student-level decisions, including an explanation of the defensibility of the
proposed methods. During the transition years, as student performance on other assessments will
be used to determine the achievement of testing standards, the linking/concordant score studies
will be conducted by the contractor and concordant scores will be recommended to the
Florida Standards Assessments, ITN 2014-47
83
Department. The contractor selected through this ITN process will be expected to secure external
legal opinion regarding the defensibility of the linking method to be employed.
7.6.3.4
Implement Achievement Level Standard Setting Meetings
The contractor is responsible for facilitating the Department’s process to establish achievement
levels and associated cut scores for the Florida Standards ELA/L, Mathematics, Algebra 1, Algebra
2, and Geometry Assessments; achievement levels, passing scores, retention cut for grade 3
ELA/L, graduation standards for ELA/L, and achievement levels and passing scores for the Algebra
1, Algebra 2, and Geometry Assessments in consultation with Florida educators and citizens.
The contractor will be responsible for developing a Standard Setting Plan for each standard-setting
process utilizing IRT item values and appropriate, reliable, valid, and defensible procedures. The
Standard Setting Plan will address the nature of the proficiency level standards, methods for
determining the standards, and procedures for validating and analyzing the quality of information
reported using the achievement levels. In addition to describing the general standard-setting
strategy, the Standard Setting Plan will describe in detail how participants will be identified and how
the standard-setting procedure will be implemented, such as the materials to be prepared by the
contractor, how test forms will be organized, and how the contractor will analyze the data and
present impact results.
The contractor will be responsible for organizing and implementing the standard-setting processes,
based on the Standard Setting Plans, and for assisting the Department in conducting the standardsetting meetings. Each standard-setting process will involve meetings of standard-setting
committees, which will involve Florida teachers and instructional leaders, followed by a reactor
meeting involving other Florida education stakeholders. The educator meetings in summer 2015
will result in content-based achievement level descriptions for each grade and recommended cut
scores for each test. The Department desires to present proposed cut scores to the State Board of
Education in January following each meeting; therefore the final report of activities of each meeting
should be available for the Department no later than November following each meeting. The final
report of standard setting activities should include in detail process and activities followed during
the standard setting, including a description of materials used, statistical background, participants,
changing of recommendations through rounds, and participant evaluation of meeting processes.
7.6.3.5
Vertical Scaling
The contractor will be responsible for conducting the vertical scaling for the Florida Standards
ELA/L and Mathematics Assessments. This scale provides for reporting growth continuously from
grades 3 to 10 in ELA/L and from grades 3 to 8 in Mathematics. In other words, each student’s
“growth” between adjacent grade levels will be calculated and reported on this scale as a type of
“value-added” score.
The contractor’s plan for vertical scaling, provided prior to the selection of items and construction of
forms for the appropriate administration, should recommend and describe the process of
determining hierarchical common content areas across grades, a recommended method for
vertical scaling that takes into account potential uses of the scale for NCLB and Florida’s
accountability system, the data collection procedures, and how and by whom and by when all of
the necessary steps will be completed. The vertical scaling plan must be approved by the
Department and presented to the Technical Advisory Committee. After vertical scaling studies are
conducted, the contractor will prepare a technical report including a description of the process,
statistical background, methodology, the recommended scale, and characteristics of the scale.
The final report will be presented to the Technical Advisory Committee.
The contractor should anticipate replicating the vertical scaling study three years after the
completion of the initial scaling study. This work may occur in the optional renewal period and is
contingent on contract renewal.
Florida Standards Assessments, ITN 2014-47
84
7.6.3.6
Scaling and Equating Studies
As explained in Sections 7.6.1.1 and 7.6.1.2, the Florida Standards Summative Assessments will
be pre-equated during test construction and post-equated after each spring test administration.
The retake assessments, such as Grade 10 ELA/L or Algebra 1 Retake will be pre-equated only,
consisting of a previously-used intact form, or another proposed design that yields a test of similar
difficulty to the operational form. In other words, during the time frame of this ITN, there will be
several equating activities to put the new test forms on the same scale through a series of
intermediate equating procedures (known as equating chains). In this case, equating error from
each of these intermediate equating procedures may accumulate to a point where the
comparability of scale scores across time is questioned. For this reason, it is essential for the
Department to employ practices that ensure comparable scores across time in the presence of
equating chains. The contractor is required to submit a research plan to be conducted in spring
2018 to investigate the extent of scale drift on the assessments. The research plan(s) must be
approved by the Technical Advisory Committee and the Department, and the resulting report(s)
must be submitted to the Department and presented to the Technical Advisory Committee.
7.6.3.6.1 Investigate the Comparability between Devices Delivering Assessments
EOC Assessments will be administered online and delivered by several different
devices throughout the state since school districts are using many different machines
when administering the assessments, such as laptops and desktops. The Department
prefers to make sure these devices used in delivering the assessments are not
providing any advantage or disadvantage to the performance of students. As a result,
there is a need to investigate whether there is an impact of certain devices on the
assessment results. The contractor must determine the groups of testing devices,
based on the types of devices that work with the contractor’s platform, whose impact
will be investigated, create a plan for the study, present the plan to the Department staff
and TAC members, and upon approval from the Department, start investigating any
impact on item-level, test form-level, and reported score-level. The results of the study
and contractor’s recommendations must also be presented to both the Department staff
and TAC members.
7.6.3.6.2 Linking Florida Standards Assessments to Other Assessments Measuring the
Same Standards
The Department intends to compare Florida student performance on FSA to other
students’ performance on the same standards from other states, also representing the
nation. Thus, contractor must link the scales of Florida Standards Assessments to the
scales from other assessments measuring the same standards.
By August of 2015, the contractor will complete a comprehensive curriculum mapping
study to determine the degree of overlap between Florida’s standards and those in
other states in order to determine the degree of alignment. The mapping study results
will be analyzed by the Department and contractor’s staff to determine if Florida’s
standards are sufficiently like those of other states to ensure a valid assessment for
purposes of reporting comparative information or linked data on Florida Standards
Assessments.
Assuming there is adequate alignment, the contractor will work with the Department to
design, plan, and conduct a linking study. The contractor’s plan for linking should
recommend and describe the process of determining the linking methodologies, the
data collection procedures, and how and by whom and by when all of the necessary
steps will be completed. The linking plan must be approved by the Department and
presented to the Technical Advisory Committee. After the linking studies are
conducted, the contractor will prepare a technical report including a description of the
Florida Standards Assessments, ITN 2014-47
85
process, statistical background and methodology, and characteristics of the scale. The
final report will be presented to the Technical Advisory Committee. The contractor will
also generate data files with linked scores and prepare reports.
If a linking study requires collecting data from either a common set of students taking
both assessments or from different sets of students taking common set of items, the
contractor selected through this ITN will be responsible for scoring the test forms
administered to students for linking, including scanning/imaging, processing,
handscoring, calibration (generation of item statistics), scoring, and production of
student results files including linked scaled scores. The scoring will be conducted after
the completion of all scoring and reporting for the operational tests given in Florida
during spring of 2016. It is anticipated that the results files from the linking study will be
provided to the Department by July 2016.
As an outcome of the linking study, the contractor will provide scoring tables displaying
how Florida’s scale scores are aligned to the ones of other assessments measuring the
same standards. Upon first use of these linking tables in reporting the results of the
spring 2017 Florida Standards Assessments, the contractor will provide psychometric
services to verify the reporting as well as the reliability and validity of the linked scores
for spring 2017.
The contractor should anticipate replicating the linking study three years after the
completion of the initial linking study. This work may occur in the optional renewal
period and is contingent on contract renewal.
7.6.3.6.3 Linking Florida Standards Assessments to International Assessments
The Department intends to compare Florida student performance on Florida Standards
Assessments to other students’ performance on International Assessments, such as
Program for International Student Assessment (PISA) and Trends in International
Mathematics and Science Study (TIMSS). Thus, contractor must link the scales of
Florida Standards Assessments to the scales from international assessments.
By August 2015, the contractor will develop a study plan for the linking Florida
Standards Assessments to International Assessments. The contractor’s plan will
include, review of the most promising and most feasible methods likely to be used in
international linking, discuss the technical feasibility of each method given the nature
and design of the Florida Standards Assessments and those of international
assessments, such as PISA and TIMSS, propose method(s) for linking the scales of
assessments, propose alternative ways for collecting data, explain the process of
establishing the linking relationship(s) in detail, and provide a comprehensive and
detailed timeline for all tasks and events. The contractor will also share the report with
the Department and its TAC members.
Upon approval from the Department, the contractor will finalize the plan for the linking
and implement all of the activities. The contractor selected through this ITN will be
responsible for scoring the test forms administered to students for linking, including
scanning/imaging, processing, handscoring, calibration (generation of item statistics),
scoring, and production of student results files including linked scaled scores. The
scoring will be conducted after the completion of all scoring and reporting for the
operational tests given in Florida during spring of 2016. It is anticipated that the results
files from the linking study will be provided to the Department by August 2016.
As an outcome of the linking study, the contractor will provide scoring tables displaying
how Florida’s scale scores are aligned to the ones of international assessments. Upon
first use of these linking tables in reporting the results of the spring 2017 Florida
Florida Standards Assessments, ITN 2014-47
86
Standards Assessments, the contractor will provide psychometric services to verify the
reporting as well as the reliability and validity of the linked scores for spring 2017.
7.7.0 Program Management
Respondents should provide their overall description of the approach to manage, implement, and support
Florida’s assessment program that specifically addresses the entire scope of work in this ITN. Respondents
should identify and describe all resources available to support all program activities. A management plan for
the contract should be incorporated into the respondent’s reply. In the event that multiple contracts are
awarded, additional program management capacity will be required to coordinate among the other contractors
and the Department.
Respondents should designate a Senior Program Manager who will have overall, daily responsibility for all
processes and deliverables under the full scope of this program.
The Chief of the Department’s Bureau of K-12 Assessment will be assigned as the program manager of the
contract resulting from this ITN under the overall supervision of the Director of Assessment, Division of
Accountability, Research and Measurement.
7.7.1 Provisions Governing the Work of the Contractor
The procedural requirements identified in this section govern the work of the contractor. This section
establishes specific requirements and general principles by which the program will be managed.
The work described in Sections 7.2.0 through 7.6.0 reflects design, development, validation, and
implementation requirements of Florida’s standards-based testing program. In preparing replies, respondents
should consider the following provisions governing the activities required by these sections.
a. All of the processes used by the contractor to complete the work identified in Sections 7.2.0 through 7.7.0
must be approved by the Department. All products must be approved by the Department at milestone
stages of development as described in the Production Schedule developed at the beginning of the contract.
The Department reserves the right to specify details of tasks and products.
b. Design control of all aspects of this program remains the responsibility of the Department. Changes in the
design made after the award of the contract will be at the discretion of the Department.
c.
The Department reserves the right to require changes in materials, produced in any medium, at any stage
of the program prior to final printing, production, or deployment.
d. Products and processes are modified to some extent for each assessment. For example, modifications
may be made to the design, format, or wording of printed and online products from one administration to the
next. All such modifications are to be expected as part of the scope of this program and will be at
contractor expense. Modifications may also be required for other products and procedures, such as data
analyses, file formats, and report designs. Consequently, all printed products developed under this contract
are to be only produced in quantities for the particular assessment, unless otherwise specified in writing by
the Department.
7.7.2 Contractor Staff and Responsibilities
The work described in the ITN constitutes a large, complex, challenging program that requires year-round
activity and close attention to overlapping tasks for different years and the impact on project resources. The
contractor is expected to provide a sufficient number of highly-qualified personnel to work closely with the
Department to manage the contract. Respondents should carefully consider whether they have sufficient and
knowledgeable human resources to complete the work activities of the ITN and should demonstrate in their
replies that they have sufficient and knowledgeable human resources to complete program work. Companies
should consider bidding only if they are committed to assigning the number and quality of staff and staff time
required to complete a program of this magnitude.
The contractor must assign professional staff members who are qualified, experienced, and capable of
providing technical assistance to the Department. Please note the following staff requirements:
Florida Standards Assessments, ITN 2014-47
87
a. The contractor will assign at least one (1) full-time senior program manager, four (4) full-time
project managers, and two (2) full-time administrative assistants. The full-time senior program
manager will concentrate on ensuring quality and timeliness of the full range of services and products
required by this ITN. The contractor’s management team is expected to lead project and support staff
and to manage implementation activities throughout the contract period. Although this section outlines
the minimum staffing requirements for this program, the contractor must provide adequate staff to
maintain the highest quality products and must deliver these products according to the Department’s
timelines.
b. It is anticipated that the four (4) full-time project managers will focus on (a) test development, (b) test
administration, (c) scoring and reporting, and (d) computer-based testing. Respondents may propose a
modification to this project assignment for the managers.
c. Implementing the program requires considerable administrative activities involving the maintenance of
schedules, the development of specifications, production of management reports and documents,
arrangement of meeting and travel for educators and Department staff, and communication with district
and other personnel. Administrative assistants are necessary to serve in a dedicated role as meeting
organizers and as direct contacts for the many Florida citizens that may attend assessment meetings
throughout the year. The Department requires a single point of contact to discuss each meeting's
requirements and venue options and to communicate with participants throughout the travel process, and
to address issues that may require late-notice changes in travel plans.
d. The managers should be members of the regular organizational staff with experience within the company
or institution in coordinating the types of activities identified by this ITN. In their replies, respondents are
expected to demonstrate that the individuals identified as managers have sufficient authority across
departments within the organization to ensure that the work of the contract has the necessary priority to
be completed with the highest quality and on time.
e. Respondents should include pertinent staffing charts in their replies that show the regular institutional
organization staffing and the proposed organization of program staff for this work. In these charts, the
full-time equivalency devoted to work under the contract for each staff member with significant
responsibilities will be identified. Persons proposed to fill all key positions will be identified by name.
f.
Respondents should indicate, by name, the professional personnel to be responsible for major contract
activities, with an estimation of the amount of time, as a percent of full-time, each person will devote to
each task. This information will be included in a separate chart in the management reply. An appendix to
the technical reply should contain one (1) page vitae for each of the professional personnel, including
subcontractors, to be assigned to the program, indicating relevant educational background and
professional experience.
g. In a separate chart(s), respondents should identify the full extent of staff resources to be allocated to the
program to conduct activities related to the item and test development; computer-based test system and
content support; ancillary materials development; scanning, imaging, data processing, and reporting
activities; and scoring, equating, and technical analysis/special studies activities for each year of the
program. The chart should identify the major activities by month and the number of staff by category
and the amount of time in full-time equivalencies (equivalent person-days) assigned to complete the
activities.
h. In addition, the contractor must provide for five (5) contractor staff positions funded through the
reimbursable category and described in Section 7.7.9.6.
Florida Standards Assessments, ITN 2014-47
88
7.7.3 Subcontractors
In addition to any subcontractors required in this ITN, the prime contractor may choose to employ one (1) or
more additional subcontractors for the completion of these tasks. If the respondent proposes to employ a
subcontractor(s), the qualifications and experience of the subcontractor(s) should be documented in the reply
at the same level of detail as those of the respondent.
The prime contractor will assume responsibility for all services offered in the reply whether or not they are
performed or produced by the contractor or by subcontractors. The Department will consider the prime
contractor to be the sole point of contact for contractual matters, including payment of any and all charges
resulting from the contract.
The respondent should describe how the work of all subcontractors will be monitored and guided to ensure
adherence to requirements in this ITN, including resulting specifications and schedules. The contractor will
provide for quality assurance activities in oversight of the subcontractor’s performance.
A separate chart in the reply should identify all of the subcontractors proposed to be involved in the program,
the assessment components they will be involved with, and the specific services they will provide.
Throughout the life of the contract, any proposed changes in subcontractors must be approved by the
Department. The contractor will provide the Department with a qualifications and experience summary for
review and consideration for acceptance in advance of a proposed subcontractor’s start of work on program
processes or deliverables. As part of the Annual Report, all subcontractors, their services, and Departmental
approval status will be listed.
A detailed description of the subcontracted services to be provided under the ensuing contract should be
included in the reply. All subcontracted services should be documented. Each service (e.g., printing, computer
operations, test development, psychometric services, legal counsel, quality control, reporting) should be fully
described, including resources, level of service, scheduling, and quality commitments. The respondent should
identify the proportion of materials, etc., to be generated by any and all involved subcontractors.
No contractor- or subcontractor-provided work related to customer service or to any secure document or data
will be provided by a persons or agencies physically located outside of the United States.
7.7.4 Program Management Communication
Weekly program management conference calls will be conducted at an agreed upon day and time in order to
ensure that all requirements and timelines are maintained and to address needs for problem solving and timesensitive adjustments. The Department may determine periodically that weekly, focused conference calls are
required as activities dictate. The contractor will provide a toll-free conference line for all calls between contractor
and Department staff required for the effective coordination of the assessment program.
To meet timelines for completing tasks, both the Department and the contractor will need to use secure
electronic communications, including, but not limited to, a secure FTP site, a website, a teleconferencing site,
etc., provided by the contractor in addition to overnight express delivery of materials as necessary. In
addition, the Department and the contractor will need access to a secure site that can be used with districts
and with subcontractors to transfer secure materials.
The contractor and the Department may determine that, for selected meetings, video teleconference meetings
will be more time and cost efficient than travel to a common meeting site. In this case, the contractor will
coordinate all logistics and provisions for the teleconference.
To assist Department staff with communication during on-site visits to the contractor’s offices or other work
sites, the contractor will provide access to lockable office space, including a telephone with a speaker, a fax
machine, and a computer with Internet access and a local printer.
Florida Standards Assessments, ITN 2014-47
89
7.7.5 Program Management Meetings
The successful operation of the program will require regular management meetings between the contractor and
Department staff. At the Department’s request, the management meetings will take place in Tallahassee, the
contractor’s headquarters or other worksite, or in a location convenient and agreed upon by the contractor and the
Department. Contractors should anticipate that the first comprehensive program management meeting will take
place at the contractor’s headquarters within thirty (30) days of contract execution.
The contractor must expect that up to fifteen (15) of their staff will travel to attend approximately two (2)
comprehensive program management meetings per year. The contractor will also send all relevant staff to up to
four (4) additional annual management meetings that are project specific, such as the scoring and reporting
debrief. All relevant subcontractors must also attend these meetings. The contractor will be responsible for the
logistics, facilities, and travel costs of their staff and required subcontractors’ staff for all management meetings.
These meetings will also involve relevant Department staff. Provisions will be made for electronic participation of
any staff member unable to travel to these meetings. In the event that multiple contracts are awarded, the
contractor should expect part of the meetings will include all other contractors and Department staff, and that part
of the meetings will be reserved for one-on-one meetings between the Department and each contractor to focus
on topics and issues specific to each contractor.
The cost reply should include funding specifically for Department staff travel for program management meetings
and other oversight activities. Program oversight activities include work related to overall program management,
management of key processes, including development, test administration, scanning, handscoring, scoring,
programming, printing, and reporting. The contractor will be responsible for paying the cost of travel, lodging, and
per diem for up to ten (10) Department staff members to attend up to fifteen (15) trips annually. The cost of the
trips has been based on travel between Tallahassee and the contractor’s headquarters or between Tallahassee
and the contractor’s or subcontractor’s headquarters, scanning/processing sites, or scoring/handscoring sites.
Funds set aside for this purpose are described in Section 7.7.9.5.
7.7.6 Program Management Reports
Five types of reports are to be prepared by the contractor to facilitate and document program management
communications with the Department: (1) the Program Schedule, (2) the Weekly Management and Weekly Action
Item Reports, (3) a Technical Report, (4) Annual Program Review, including the Operational Addendum, and (5)
the Accounting Report. In addition to written reports, the Department may require the contractor’s participation
in presentations to Department senior management or other entities at places and times determined by the
Department.
1. The Program Schedule will be developed for each year of the program and will include both high-level
management summary information as well as specific dates associated with critical processes, milestones,
and deliverables for individual projects listed in the program management plan. The Program Schedule will
be derived from the program management plan at the beginning of each test administration year, and will be
updated quarterly. The Department expects submission of the Program Schedule in Microsoft Excel. If
significant changes occur between quarterly submissions of the program schedule, the Department may
request an updated schedule reflecting the changes at any time.
2. The contractor will provide a Weekly Management and Action Item Report to the Department. A template for
this report will be provided to the contractor, and the Department reserves the right to modify the design,
format, and content of this report at any time. The Weekly Management Report and Action Item Report will
report the status of recent and upcoming tasks, highlighting those leading up to deliverables and critical tasks,
report progress on program issues, identify unresolved issues, and maintain a list of requests requiring
change orders or amendments. The weekly report will be provided to the Department by no later than noon
one day prior to the weekly program management conference calls and will reflect the previous week’s work.
3. The contractor will produce a Technical Report and provide the initial draft document to the Department by
September 30 of each year of the contract arising from this ITN. The Department will work with the
Florida Standards Assessments, ITN 2014-47
90
contractor to identify the specific information to be provided in the Technical Report each year. A
proposed report structure is provided in Section 7.6.3.2.
4. The contractor will produce two copies of the Annual Program Review each year to be provided only on
CDs/DVDs/jump drives inside plastic sleeves in loose-leaf binders. Along with other materials described
below, the Program Review will include a record of the contractor’s services and deliverables in response to
contractual/ITN requirements. This review must identify major program accomplishments and successes, as
well as areas of the program that require changes and improvement in the upcoming year. The review is
intended to be a history of program events and to describe recommended improvements or changes for
subsequent years. The annual review will be based on information gathered by the contractor’s program and
project management staff throughout the year. The review will address all aspects of program service and
products including management, technology, interpretive products, communications, development, test
production, test administration, materials shipping/pickup, scoring, and reporting processes. It will also follow
the annual administrative cycle, beginning with fall and ending with the following summer.
The Annual Program Review will include:
– details of meetings held with educators (dates, locations, purpose)
– milestones attained/deliverables completed for each project area
– problems encountered, root cause analyses, and corrective action(s) taken
– performance metrics reports as required by the Department noting changes over time
The Annual Program Review will include electronic files of deliverables and documentation on services
required by this ITN, beginning with the fall test administration and ending with the summer test
administration. It will include:
• PDFs of the test and answer books,
• PDFs of all screenshots for all computer-based test approved forms,
• PDFs of all passage booklets, reference sheets, large print test books, one-item-per-page test books,
large print reference sheets, large print response grid instruction sheets, and CBT work folders,
• PDFs (untagged and tagged, if applicable) of all interpretive and ancillary products,
• Native application files of all interpretive and ancillary products,
• PDFs, or other appropriate file formats such as video, of all other products, including memoranda and
report templates,
• Missing materials and resolution reports,
• Final Program Schedule reflecting planned vs. actual dates of task completion,
• Weekly Management and Action Item reports,
• File Layouts, and
• Specifications documents.
5. The collected Accounting Reports will provide a summary of the Quarterly Accounting Reports, including
expenditures by program task and level of effort as well as expenditures and balances of Reimbursable
Funding Categories identified in Section 7.7.9, including balances resulting from executed change orders and
contract amendments.
7.7.7 Meetings Involving Florida Educators and Other Citizens
The contractor will be responsible for arranging, organizing, and paying the cost of bringing participants to
specific meetings as identified in this ITN. This section identifies points at which meetings with groups of
Florida educators and other citizens will be conducted to propose development and/or management
guidelines, to draft products, or to review work and products produced by the contractor. The Department will
designate Florida educators, and in some instances Florida citizens or consultants from outside the state, to
participate in these meetings.
Florida Standards Assessments, ITN 2014-47
91
All meetings are facilitated by Department staff unless determined otherwise by the Department. Contractor
staff will attend the meetings with Florida educators, as required. The contractor is responsible for on-site meeting
organization, support, and management. Generally, the contractor’s manager or team leaders for the program
area and an administrative assistant responsible for meeting planning will be present at meetings for overall
management and support purposes. The contractor will also be expected to send staff appropriate to assist and
participate in each meeting. Generally, for all meetings involving separate subject/grade groups, the contractor will
provide the appropriate professional staff to take part in each of the separate subject/grade groups. The contractor
will be responsible for all expenses associated with contractor staff travel and participation. All expenses must be
in accordance with the Department of Financial Services Guidelines for State Expenditures which prohibits the
purchase of food and beverages and other unallowable costs. The Guideline may be accessed through the
following link: http://www.myfloridacfo.com/aadir/reference_guide/.
The contractor will be responsible for arranging, organizing, attending, and paying travel, lodging and per diem
costs required to bring these participants together for the meetings. For meetings to be held in Florida, they
will typically be in a central Florida location, a hub city (e.g., Jacksonville), or in Tallahassee.
Table 7.7.1 provides information about the number, expected length, approximate number of participants, and
estimated cost of such meetings.
Table 7.7.7 Meetings of Educators and Other Citizens
Table 7.7.7 Meetings of Educators and Other Citizens
Meeting
Area
1
Item/ Passage
Bias
English
Language
Arts/ Literacy
(ELA/L),
Mathematics
7.3.4.1
1
Oct 2015,
2016, 2017
Oct
2018,
2019,
2020
2
Approx #
of
attendees/
ind mtg
30
2
Item/ Passage
Community
Sensitivity
ELA/L,
Mathematics
7.3.4.1
1
Apr 2015,
2016; May
2017
2
30
Jacksonville
or Central
Florida
3
Item Content
Review
ELA/L,
Mathematics
Test Items
7.3.4.2
1/
subject &
grade
Oct 2015,
2016, 2017
5
8
Jacksonville
or Central
Florida
4
Passage
Content
Review
ELA/L
7.3.4.2
1
Aug 2015,
2016, 2017
5
15
Jacksonville
or Central
Florida
5
Data Review
Mathematics
7.3.4.3
1
May or Jun
2015,
2016, 2017
2
9
Tallahassee
6
Standard
Setting
ELA/L and
Mathematics
7.6.3.4
1/
subject
Aug 2015
May
2018,
2019,
2020
Oct
2018,
2019,
2020
Aug
2018,
2019,
2020
May or
Jun
2018,
2019
N/A
5
300
Central FL
7
Standards
Reactor
ELA/L and
Mathematics
7.6.3.4
1
Aug 2015
N/A
3
30
Tallahassee
ITN
Ref.
Mtgs/
Year
Base
Contract
Florida Standards Assessments, ITN 2014-47
Renewal
Days/
Mtg
Likely Site
Jacksonville
or Central
Florida
92
Table 7.7.7 Meetings of Educators and Other Citizens
Approx #
of
attendees/
ind mtg
25/subject/
meeting
ITN
Ref.
Mtgs/
Year
Base
Contract
Renewal
Days/
Mtg
3 per
subject/fi
rst year;
2 per
subject
after first
year
2
TBD
TBD
3
Spring and
Nov 2015,
2016, 2017
Spring
and Nov
2018,
2019
2
20
Tallahassee,
Jacksonville,
Central FL, or
site of FERA
Sep
2017;
May and
Sep
2018,
2019,
May
2020
Sep
2017,
2018,
2020
5
10/subject
per grade
(3
grades/cou
rses per
subject)
Tallahassee
5
8
Tallahassee
Meeting
Area
8
Content
Advisory
ELA/L,
Mathematics
7.3.4.4
9
Technical
Advisory
Committee
Technical
7.6.0
10
Field Test
Rangefinding
ELA/L,
Mathematics
7.5.4.5
2/
subject
May and
Sep 2015,
2016, May
2017
11
Operational
Rangefinder
Selection
ELA/L,
Mathematics
7.5.4.5
1/
subject
Sep 2015,
2016,
Likely Site
Tallahassee,
Jacksonville,
or Central FL
7.7.8 Communication between the Contractor and Districts
Florida has 67 school districts conforming to the boundaries of Florida’s 67 counties. Within each school
district is a district coordinator of assessment assigned to work directly with the Department to implement
statewide assessment programs. All aspects of test administration and reporting are administered within the
districts through the district coordinator of assessment. Other district personnel involved in statewide test
administration include one testing coordinator per school and the teachers or guidance counselors who serve
as test administrators. While there are some testing sites that are not part of the public school system (e.g.,
private schools), most communication will still be directed through the district coordinator of assessment.
All communication between the contractor and school districts must be directed through or receive prior approval
by the Department. Memos required to implement various aspects of the assessment will be prepared by the
contractor and approved by the Department prior to release. The contractor will be required to email, mail, or fax
memoranda or sets of instructions to district coordinators of assessment as instructed by the Department.
Districts will be instructed to communicate directly with the contractor only for the purpose of clarifying meeting
travel logistics, ordering additional testing materials, correcting computer-based system problems, or to report
problems with the delivery/pickup of materials. For this purpose, the contractor will dedicate for the use of Florida’s
district and school personnel a toll-free help desk telephone, operated Monday through Friday 6:00 AM through
8:00 PM, Eastern Time, a toll-free fax number, and a web-based help site available during the same hours cited for
the help-desk telephone. This customer service is distinct from that described in Section 7.4.10.
Florida Standards Assessments, ITN 2014-47
93
7.7.9 Reimbursable Funding Categories
Six funding categories are designated for specified program functions and may be used only for those
functions. Use of these funds requires authorization by the Department contract or program manager or
program area leads. Authorization may be provided by specific email or other written communication, or by
Department approval of a specific plan for a meeting(s) or activities requiring use of these funds. Amounts are
designated for administration tasks. Respondents should include these amounts in their cost replies.
Monies not expended from these funds will be deducted from final contract payment at the conclusion of the
contract during contract resolution. The reimbursable fund balances will be reported by the contractor
quarterly with each invoice submission in the Accounting Report. The amounts specified in this section may
be increased or decreased by an appropriately negotiated amendment at any time during the lifetime of the
contract. The Accounting Report will indicate expenditures from, and the balances remaining, in these funds.
The final contract invoice will be accompanied by the final balance sheet for these funds. The amounts
allocated for these funds should be identified separately in the respondents’ cost reply.
7.7.9.1
Consultant Services
The contractor will provide to the Department technical advice and consultant service, as required,
from outside the contractor’s staff. The contractor will recommend and the Department will select
the consultant(s) to be used, who may or may not have been recommended by the contractor. A
contract between the consultant and the contractor will be required and will include the amount
allocated for honorarium, travel, lodging, and per diem as authorized by the Department.
Reimbursement for travel will be at rates authorized by Section 112.061, F. S., for travel and per
diem. The respondent should include in the reply the amount of $84,375.00 as a fixed amount for
the base contract period and $28,125.00 as a fixed amount for each year of the optional renewal
period.
7.7.9.2
Contingency Services
It is possible that unforeseen circumstances will make it either necessary or desirable to perform
tasks not specifically anticipated at this time, for example, data analyses beyond those described
herein, unexpected data processing, acquisition of additional materials, or programming for test
functionality not yet envisioned. These circumstances cannot be defined in advance, and the
Department must be able to request such extra services in a timely fashion so as not to impede
implementation of the assessments. In the reply, the respondent should include $84,375.00 as a
fixed amount for the full term of the base contract period and will include $28,125.00 as a fixed
amount for each year of the optional renewal period.
7.7.9.3
Author’s Alterations and Design Control
The Department reserves the right to make changes in materials at any stage of the program prior
to final production or delivery. Any changes to the Department approved, production-ready product
(e.g., digital proofs or final computer-based test files) that are required due to contractor error will
be paid for by the contractor. Otherwise, changes made to the Department approved, productionready product will be paid for as author’s alterations. If any alterations are necessary, the
Department will identify in writing to the contractor the alterations to be made. In the reply, the
respondent should include $84,375.00 as a fixed amount for the full term of the base contract
period and will include $28,125.00 as a fixed amount for each year of the optional renewal period.
7.7.9.4
Provision for Meetings with Florida Educators
The respondent will include a fixed amount to cover the costs of travel and per diem for Florida
educators, Florida citizens, and others participating in the meetings identified in Section 7.7.7. The
contractor will be responsible for the travel reimbursement for participants at rates authorized by
Section 112.061, F. S., for travel and per diem. In the reply, the respondent should include
$5,491,496.00 as a fixed amount for these meetings with Florida educators during the base contract
period and $1,830,498.00 for each year of the optional renewal period.
Florida Standards Assessments, ITN 2014-47
94
7.7.9.5
Provision for Meetings with Department Staff
A fixed amount will be included in the reply to cover the costs of travel and per diem for the
Department of Education staff or others participating in meetings with the contractor on behalf of
the Department. Reimbursement for staff will be at rates authorized by Section 112.061, F.S.
These funds will not be used to fund travel and per diem of the contractor’s staff. These funds will
be expended only with the approval of the Department. In the reply, the respondent should include
$438,750.00 as a fixed amount for ten (10) staff members to attend as many as fifteen (15)
meetings per year requiring four (4) travel days each during the contract during the base contract
period and $146,250.00 for each year of the optional renewal period.
7.7.9.6
Contractor Staff Positions
To improve work flow, work quality, and communications, the contractor will fund salary and
benefits for five (5) professional positions to work under the direction of the Bureau of K-12
Assessment within the Florida Department of Education. These positions will be three editors and
two Systems Project Consultants. The Department will work closely with the contractor to interview
and select individuals who are acceptable to both parties. The contractor will employ its standard
recruiting, screening, and hiring procedures to secure individuals to fill these positions. The
employees will be full-time employees of the contractor. The employees will receive benefits
typically provided to the contractor’s employee classification system. The employees will be paid
at a rate comparable to similarly skilled and experienced Department employees using the
contractor’s employee classification system.
The basic duties of the Computer-Based Test Editor (1) and the Materials Editors (2) include: (a)
write and edit materials for content, such as items, tests, reports, training materials, brochures,
scripts, or on-screen presentations; (b) develop, analyze, and select appropriate information and
present it in a form and at a level suitable for the intended audience; (c) review written products
and materials for conformity with standards of objectivity, style, and presentation; and, (d) perform
a variety of editorial duties related to editorial production. Candidates must possess one (1) year of
specialized experience directly related to these duties. Candidates may substitute two (2) full
academic years of related graduate level education in lieu of the work experience requirement.
The basic duties of the Systems Project Consultants (2) include: (a) participating in weekly
meetings and communicating issues between the FDOE and the contractor’s program managers,
(b) making suggestions for improvement to the contractor’s procedures to aid FDOE staff in data
verification activities, (c) assisting FDOE staff debugging SAS computer programs and analyzing
data files, (d) traveling, at the contractor’s expense, with FDOE staff to check scanning at the
contractor’s site, (e) assisting in the verification of reports of results using SAS computer programs,
(f) assisting in the verification of posting of files and reports on the portal, and (g) communicating
issues with District Coordinators of Assessment related to scoring and reporting activities and
issues. Candidates must have a thorough understanding of large scale standardized testing
programs and data verification techniques. The candidate must also have experience managing
data activities such as those required for this project, verifying data files, and using SAS, Excel,
and other Microsoft Office products.
The respondent should include in the response $1,800,000.00 as a total amount during the base
contract period and $600,000.00 for each year of the optional renewal period.
7.7.10
Final Delivery of Materials
The contractor agrees to deliver to the Department, upon request, all materials and products in all
forms that are developed for and used in conjunction with this program, including test books,
answer documents (paper and PDF), final electronic source files of interpretive products, software
source code, and ancillary materials, PDFs, and electronic files within 30 days following
acceptance by the Department of the final report for the program. Payment of the final program
Florida Standards Assessments, ITN 2014-47
95
invoice will not be made until all materials and certification of destruction, as appropriate, are
received and approved by the Department and final payment resolution is agreed to by both
parties.
At the completion of the contract, all data files, including those containing items or graphics,
remaining in the possession of the contractor will be destroyed once all handoffs are made to the
next contractor. Written verification of destruction will be provided to the Department as part of the
final contract report. Destruction will not take place until authorized by the Department.
7.8
DELIVERABLES (See Attached Section 7.8 Deliverables provided in an Excel Spreadsheet.)
SECTION 8 – OPENING, EVALUATION AND AWARD
8.0
REPLY OPENING
TECHNICAL & PRICE REPLIES WILL BE OPENED BY THE DEPARTMENT'S BUREAU OF CONTRACTS,
GRANTS AND PROCUREMENT MANAGEMENT SERVICES PERSONNEL AT 325 WEST GAINES
STREET, 332 TURLINGTON BUILDING , TALLAHASSEE, FLORIDA, AS SPECIFIED IN SECTION 3.4
CRITICAL EVENT DATES
8.1
REPLY EVALUATION AND NEGOTIATION PROCESS
Using the evaluation criteria specified below, in accordance with Section 287.057, F.S., the Department shall
evaluate and rank responsive Replies and, at the Department’s sole discretion, proceed to negotiate with one
or more Respondent(s) selected, as follows:
A. Selected Respondent(s) will be invited to commence contract negotiations. If necessary, the Department
will request revisions to the approach submitted by the selected Respondent(s) until it is satisfied that the
contract will serve the Department’s needs. The process will continue until a contract is negotiated and
executed. The Department may in its sole discretion, award and enter into contracts with more than one
Contractor, if in the best interest of the state.
B. The Department reserves the right to negotiate with all responsive and responsible Respondents, serially
or concurrently, to determine the best-suited solution. The ranking of Replies indicates the perceived
overall benefits of the proposed solution, but the Department retains the discretion to negotiate with other
qualified Respondents as deemed appropriate.
C. Before award, the Department reserves the right to seek clarifications, to request Reply revisions, and to
request any information deemed necessary for proper evaluation of Replies. Respondents may be
requested to make a presentation, provide additional references, provide the opportunity for site visits,
etc. The Department reserves the right to require attendance by particular representatives of the
Respondent. Any written summary of presentations or demonstrations shall include a list of attendees, a
copy of the agenda, and copies of any visuals or handouts, and shall become part of the Respondent’s
Reply. Failure to provide requested information may result in rejection of the Reply.
D. The focus of the negotiations will be on achieving the solution that provides the best value to the state.
E. In submitting a Reply Respondent agrees to be bound to the terms of this ITN, however, the Department
reserves the right to negotiate different terms and related price adjustments if the Department determines
that it is in the state’s best interest to do so.
F. The Department reserves the right to reject any and all Replies, if the Department determines such action
is in the best interest of the state or the Department. The Department reserves the right to negotiate
concurrently or separately with competing Respondents. The Department reserves the right to accept
portions of a competing Respondent’s Reply and merge such portions into one project, including
Florida Standards Assessments, ITN 2014-47
96
contracting with the entities offering such portions. The Department reserves the right to waive minor
irregularities in Replies.
Criteria for Evaluation
The Department shall evaluate and rank responsive Technical Replies and score them on a scale of 1 to 100
using the following criteria. The Department anticipates awarding the Contract (if any award(s) is made) after
negotiations to the responsible and responsive vendor determined to provide the best value to the state.
Each Respondent will receive one score for all criteria, other than price, regardless of the number of
assessment components it proposes to deliver. This score will constitute the Total Technical Score for the
Respondent. Respondents will also receive a separate price evaluation for each assessment component it
submits a price for (A-E in Attachment 1). The Department will add the Total Technical Score for each
Respondent to the price evaluation score for an assessment component to determine the total score for the
Respondent for that assessment component. Respondents will be ranked for each assessment component in
this manner.
The following criteria will be used to evaluate and rank Replies.
A)
B)
C)
D)
Executive Summary
Qualifications and Experience
Technical Plan
Management Plan
Total
N/A
10 points
50 points
15 points
75 points
E)
Price
25 points
25 points
Total
Price analysis is conducted through the comparison of Price Replies submitted. The Department shall score
responsive Price Replies up to 25 points using the below.
Price analysis is conducted through the comparison of Price Replies submitted.
The method of awarding points for the Price Reply will be done in accordance with the provisions of Section
287.0572, F.S., which requires use of the present value methodology. The present value discount rate will be
used in the computations and evaluation. To determine that discount rate, use the rates identified in Release
H.15, Select Interest Rates (Weekly), available online at http://www.federalreserve.gov/releases/h15/. Use the
most recent copy of the Federal Reserve Bulletin published at the time of issuance of the ITN. See the
“Critical Event Dates” for the release date.
The maximum points will be awarded to the lowest acceptable Price Reply. Replies with higher costs will
receive the fraction of the maximum points proportional to the ratio of the lowest Price Reply to the higher
Price Reply. The fractional value of points to be assigned will be rounded to one decimal place. The criteria
for price evaluation shall be based upon the following formula:
(Low Price/Respondent's Price) x Price Points = Respondent’s Awarded Points
The total maximum number of points that can be earned in the evaluation process is 100 points.
8.2
EVALUATIONS
The ITN is designed to assess the most points to the Respondent presenting the best solution for the required
services. The Evaluation Committee will consider only those Replies provided after the preliminary
administrative review (See SECTION 6.2.1) first completed by the Department’s Bureau of Contracts, Grants
and Procurement Management Services.
Each member of the Evaluation Committee will be provided a copy of each Technical Reply. Replies will be
evaluated on the criteria established in the section above entitled "Criteria for Evaluation" in order to assure
Florida Standards Assessments, ITN 2014-47
97
that Replies are uniformly rated. The Evaluation Committee will assign points, utilizing the technical
evaluation criteria identified herein and the Procurement Office will complete a technical summary.
The Department reserves the right to request oral presentations/seek clarification on any Reply submitted.
Information requested and received will be evaluated by the Committee based on the criteria established in
SECTION 6.2.2 above. During this stage Respondents will be asked to provide any clarifications needed by
the Evaluation Committee to assist in evaluating their Reply. Information received in this stage will be added
to the Respondent’s Reply and evaluated as a part of the appropriate section above.
8.3
POSTING OF INTENDED AWARD
The intended award will be posted in the Bureau of Contracts, Grants and Procurement Management
Services, 325 West Gaines Street, 332 Turlington Building, Tallahassee, Florida and on the VBS and will
remain posted for a period of seventy-two (72) hours. (See SECTION 3.4 CRITICAL EVENT DATES.)
8.3.1 Protest of Intended Award
Any Respondent who is adversely affected by the Department's recommended award or intended decision
must file a written “Intent to Protest” with the Department at the address of posting. See SECTION 4.4
PROCUREMENT PROTESTS/NOTICE OF RIGHTS for protest information.
8.3.2 Inability to Post
If the Department is unable to post as defined above, the Department will post a public notice on the VBS.
The Department will provide written notification of any future posting in a timely manner.
8.4
AWARD OF THE CONTRACT
Services will be authorized to begin when the Contractor receives a fully executed contract from the
Department’s Contract Administrator.
****
Florida Standards Assessments, ITN 2014-47
98
APPENDIX A:
TRANSITION TO COMPUTER-BASED TESTING AND ADDITIONAL TEST ADMINISTRATION DATES
All high school and End-of-Course assessments are currently computer-based assessments and are anticipated to
remain as such. Grade 11 English Language Arts/Literacy and Algebra 2 End-of-Course Assessments will be
implemented as computer-based assessments.
Table A.1: Transition to FSA Computer-Based Tests in Florida
Grades 3-8: ELA/L and Mathematics
Grade
PBT
PBT
PBT
PBT
Transition to
CBT
Mathematics
PBT
PBT
PBT
Transition to
CBT
CBT
PBT
PBT
Transition to
CBT
CBT
CBT
Mathematics
PBT
PBT
PBT
Transition to
CBT
CBT
PBT
Transition to
CBT
CBT
CBT
CBT
Mathematics
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
Mathematics
Transition to
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
Mathematics
PBT
Transition to
CBT
CBT
CBT
CBT
ELA/L
8
2017-2018
ELA/L
7
2016-2017
ELA/L
6
2015-2016
ELA/L
5
2014-2015
ELA/L
4
2013-2014
ELA/L
3
Subject
Transition to
CBT
CBT
CBT
CBT
CBT
Mathematics
PBT
PBT
Transition to
CBT
CBT
CBT
Key
CBT = Was computer-based test in prior year
Transition to CBT = First year of computer-based test in this grade/ content
PBT = Administered as a paper-based test
Florida Standards Assessments, ITN 2014-47
99
The following dates are for reference only and refer to assessments that are not included as part of this ITN.
Dates for the assessments that are included in this ITN are provided in Table 7.2.2.
Table A.2: Additional Administration Dates for FCAT 2.0 and Florida EOC Assessments
(Assessments not included in this ITN)
Florida End-of-Course Assessments (U.S. History,
FCAT 2.0
Civics, Biology 1, Algebra 1 Retake, Geometry Retake)
Fall
Spring
Year
Reading
Reading
Spring Science
Fall
Winter
Spring
Summer
Retake
Retake
Nov. 30– April 25–May July 11–22,
Oct. 5–16,
April 11–
April 11–15,
Sept. 14–25,
2015-16
Dec. 18,
27, 2016
2016
2015
15, 2016
2016
2015
2015
Nov. 28– April 24–May July 10–21,
April 17–21,
Sept. 12–23,
2016-17
NA
NA
Dec. 16,
26, 2017
2017
2017
2016
2016
Nov. 27– April 30–May July 16–27,
April 16–20,
Sept. 11–22,
2017-18
NA
NA
Dec. 15,
25, 2018
2018
2018
2017
2017
Nov. 28– April 29–May July 15–26,
April 15–19,
Sept. 10–21,
2018-19
NA
NA
Dec. 19,
24, 2019
2019
2019
2018
2018
April 13–17,
Sept. 9–20,
Dec. 2–20, April 27–May July 13–24,
2019-201
NA
NA
2020
2019
2019
22, 2020
2020
Nov. 30– April 26–May July 12–23,
April 12–16,
Sept. 14–25,
2020-211
NA
NA
Dec. 18,
21, 2021
2021
2021
2020
2020
1 Optional Renewal Period
Florida Standards Assessments, ITN 2014-47
100
APPENDIX B
CRITICAL AND EXTREMELY CRITICAL ACTIVITIES
Final dates for these activities and a final activities list will be determined in the negotiation process of this ITN and included in the final contract. The table below
describes the critical and extra critical tasks, the frequency for distribution, and any notes about the approximate delivery dates. Failure to provide these materials
by the agreed-upon dates will be cause for the assessment of liquidated damages.
Task Activity
Critical (C) or
Extra Critical
(XC)
Delivery of practice tests and sample test
materials to districts.
C
Frequency
Varies by deliverable
Delivery Notes (Approximations)
By November/December for assessments that are
introduced in the spring, regular sample tests (PDF or
computer-based) are delivered the initial year of the
assessment or the first year of a format change (i.e.,
transition from paper-based to computer-based).
ITN Ref.
7.3.7.5,
7.4.9.6
Special format print materials (e.g., Braille, large print)
are delivered upon request each year.
Delivery of the printed End-of-Course
test administration manuals and ancillary
materials to districts for the paper-based
and computer-based administrations.
C
Prior to each fall
administration (see note in
7.4.1 regarding the
possibility of 2 manuals
for 2014-15 school year)
Usually 70 calendar days before each administration.
7.4.1
Delivery of the printed ELA/L and
Mathematics test administration manuals
and ancillary materials to districts for the
paper-based and computer-based
administrations.
C
Prior to each spring
administration
Usually 70 calendar days before each administration.
7.4.1
Delivery of the printed ELA/L Retake test
administration manuals and ancillary
C
Prior to each fall
administration (beginning
Usually 70 calendar days before each administration.
7.4.1
Florida Standards Assessments, ITN 2014-47
101
Task Activity
Critical (C) or
Extra Critical
(XC)
materials to districts for the paper-based
and computer-based administrations.
Frequency
Delivery Notes (Approximations)
ITN Ref.
in fall 2015)
Delivery of the printed test materials to
districts.
XC
Prior to each
administration.
Usually 3 weeks before each administration.
7.4.1
Delivery of the preidentification labels to
districts.
XC
Prior to each
administration
Usually 3 weeks before each administration.
7.4.1
Delivery of the computer-based tests to
districts.
XC
Prior to each
administration
The first day of the test administration window, unless
the system requires pre-loading the assessments. (At
that point, they should be delivered in sufficient time
for the schools to load the test prior to the
administration.)
7.4.9
After each administration
Varies by results release deliverable. Initial results
delivered as early as two weeks after the end of the
administration.
For EOC administrations:
• RD1-RD3 (state, district, and school reports)
must be delivered within two weeks of the
last day of the administration window, per
statute.
• RD4 (hard copies of student results) must be
delivered within 1 week of RD1-3.
• RD5 (late reporting) should be completed
within 2 months of RD4.
• RD7 (demographic reports) will be delivered
in August/September for the previous school
year.
For ELA/L and Mathematics (Spring):
7.5.5
Delivery of test results to districts.
RD1 – XC
Results Delivery (RD) details for 1-7 are
provided in Table 7.5.5. This includes
print and electronic deliveries.
RD2 – XC
RD3 – XC
RD4 – XC
RD5 – C
RD6 – C
RD7 – C
Florida Standards Assessments, ITN 2014-47
102
Task Activity
Critical (C) or
Extra Critical
(XC)
Frequency
Delivery Notes (Approximations)
•
•
•
•
•
Florida Standards Assessments, ITN 2014-47
RD1-RD3 (state, district, and school reports)
delivered between the end of April and the
middle of May.
RD4 (hard copies of student results) must be
delivered before the week of June 8, per
statute.
RD5 (late reporting) should be completed
within 3 months of RD4.
RD6 (writing image media) is usually
delivered mid-July.
RD7 (demographic reports) will be delivered
in August/September for the previous school
year.
103
ITN Ref.
APPENDIX C
FLORIDA PUBLIC SCHOOL STUDENT PARTICIPATION ESTIMATES, DISTRICT NUMBERS, AND
SPECIAL DISTRICT/SCHOOL NUMBERS
STUDENT PARTICIPATION ESTIMATES
Table C.1 indicates the anticipated number of students that would be tested in each grade-level test administration.
The number of students expected to participate in ELA/L Retake administrations (Table C.2) will be calculated as a
percentage of the grade 10 population.
Table C.3 indicates the expected test takers for each EOC Assessment. This calculation assumes 200,000 new
students for each spring cohort. (Cohort, for the purposes of this ITN, refers to a group of students that enters grade
9 and is enrolled at the high school for a total of four years.)
Table C.1 Anticipated Number of Students Participating in Assessment Administrations
Table C.1 Anticipated Number of Students Participating in Assessment Administrations
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade
10
Grade
11
2013-14
212,774
201,371
198,642
203,053
210,018
211,085
222,533
206,675
198,131
2014-15
2015-16
2016-17
217,735
225,524
223,753
202,712
207,553
215,105
203,183
204,609
209,564
202,436
207,154
208,854
206,962
206,010
210,865
213,634
210,031
209,364
228,413
229,570
226,066
216,413
218,707
219,248
202,282
205,215
206,493
2017-181
224,786
213,553
217,105
213,364
212,354
214,365
225,364
216,066
207,248
2018-191 225,132 214,786
2019-201 224,893 215,132
1 Optional Renewal Period
215,553
216,734
221,105
219,654
216,864
224,634
215,854
220,268
230,365
231,467
215,364
220,540
206,707
203,211
Table C.2 Anticipated Number of Students Participating in ELA/L Retake Administrations
Table C.2 Anticipated Number of Students Participating in ELA/L
Retake Administrations
Fall Retake
Spring Retake
1
ELA/L
65%
45%1
1 As a percent of anticipated Grade 10 enrollment for applicable years
from Table C.1.
Florida Standards Assessments, ITN 2014-47
104
Table C.3 Anticipated Number of Students Participating in EOC Administrations
Table C.3 Anticipated Number of Students Participating in EOC Administrations
School
Year
Spring
Summer
Fall
Winter
2015-16
Spring
Summer
Fall
Winter
2016-17
Spring
Summer
Fall
Winter
2017-181
Spring
Summer
Fall
Winter
2018-191
Spring
Summer
Fall
Winter
2019-201
Spring
Summer
1 Optional Renewal Period
2014-15
Algebra 1
200,000
100,000
75,000
56,250
242,188
131,641
98,730
74,048
255,536
141,652
106,239
79,679
259,759
144,820
106,239
79,679
259,759
144,820
106,239
79,679
259,759
144,820
Florida Standards Assessments, ITN 2014-47
Algebra 2
200,000
50,000
12,500
3,125
200,781
50,195
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
Geometry
200,000
50,000
12,500
3,125
200,781
50,195
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
TOTAL
600,000
200,000
100,000
62,500
643,750
232,031
123,828
80,322
657,104
242,044
131,337
85,954
661,328
245,212
131,337
85,954
661,328
245,212
131,337
85,954
661,328
245,212
105
DISTRICT NUMBERS AND SPECIAL DISTRICT/SCHOOL NUMBERS
NOTE: The districts/schools participating in each test administration may vary. For example, some administer only
the tests required for graduation.
01-Alachua through 67-Washington
68-
Florida School for the Deaf and the Blind (FSDB)
68-0011 Deaf Elementary School
68-0014 Blind Elementary School
68-0012 Deaf Middle School
68-0015 Blind Middle School
68-0013 Deaf High School
68-0016 Blind High School
68-0019 Career Education and Transition High School
69-0011 Okeechobee Youth Development Center
71- Florida Virtual Schools
71-0300 FLVS Full-Time K-8
71-0400 FLVS Full-Time 9-12
71-0500 FLVS Part-Time 6-8
71-0600 FLVS Part-Time 9-12
71-0700 FLVS Part-Time K-5
72-75 University Laboratory (Lab) Schools
72-0011 A.D. Henderson University School & FAU High School
72-0020 FAU/SLCSD Palm Pointe Educational Research School @ Tradition
73-0341 Florida State University School (FSUS)
73-0351 The Pembroke Pines Florida
74-0351 Florida A&M University (FAMU) Developmental Research School
75-0391 University of Florida (UF) Lab School
P.K. Yonge Developmental Research School
80- State Colleges with Adult High School Programs
80-0005 Daytona State College (Volusia)
80-0007 Florida State College at Jacksonville (Duval)
80-0011 Indian River State College (St. Lucie)
80-0020 Pensacola State College (Escambia)
80-0025 Seminole State College (Seminole)
80-0026 South Florida State College (Highlands)
97 – Office of Independent Education (Florida Tax Credit Scholarship students)
98 – Ahfachkee School
Special Programs (The following special programs are designations within each district. The “XX” is replaced by the
two-digit district number.)
XX-3518 – McKay Scholarship Students
XX-9998 – Home Education Program Students
Florida Standards Assessments, ITN 2014-47
106
APPENDIX D
SAMPLE TEST DESIGN
Table D.1 below is provided as an example of one possible test design that may meet the criteria described in
Section 7.3.0. Respondents may propose other designs, and a final design will be discussed in negotiation and
agreed upon between the Department and the selected contractor.
Table D.1 Example Testing Sessions – Grades 3-8 and High School
Table D.1 Example Testing Sessions – Grades 3-8 and High School
Content Area
Specification
Session 1
Session 2
Session 3
English Language
Arts/Literacy
Time
80 minutes
80 minutes
80 minutes
Item Types
Multiple Choice and
Short Constructed
Response
Extended Writing in
Response to Text
Mathematics
Time
90 minutes
Multiple Choice
and Short
Constructed
Response
90 minutes
Item Types
Multiple Choice,
Gridded/Fill-in
Response, Short
Constructed
Response, and
1-2 Extended
Performance Tasks
Florida Standards Assessments, ITN 2014-47
Multiple Choice,
Gridded/Fill-in
Response, Short
Constructed
Response, and
1-2 Extended
Performance
Tasks
107
APPENDIX E
PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS
Appendix E is organized as follows:
Part 1 Operational Test Materials Product Specifications & Distribution Requirements
Part 2 Florida Department of Education PDF Guidelines for Section 508/ADA Documents
Part 3 Ancillary Materials Shipment Schedule & Print Quantities
APPENDIX E, Part 1
OPERATIONAL TEST MATERIALS
PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS
Test Documents – General Considerations
The contractor is responsible for producing camera-ready forms of these tests, printing the test documents,
and distributing them to school districts. The following requirements apply to al l test documents:
• A method to collect demographic information and other special information (e.g., invalidation)
o Preferred: Scannable document covers include the student demographic grids and other special
coding sections.
• A method to track and log secure materials
o Preferred: Unique security barcode numbers will be printed on each book containing test questions.
• Large Print and Braille (contracted and uncontracted) version of the test. Copies of the regular book (the same
form) must accompany the shipment of the large print and Braille materials. Braille notes also accompany the
Braille versions.
o Preferred: One form of each test (usually form 1) is produced in large print (minimum of 18 point
type on 14” x 17” approved paper) and in Braille.
• Forms will be spiraled for random distribution.
• Documents will be shrink-wrapped in packs of 5 and 25. Quantities smaller than 25 may be desirable and will
be approved by the Department when the document size makes lifting and packing difficult or hazardous.
• Test documents must be high-quality and durable
o Preferred: Test documents are 8 ½” x 11,” saddle stitched or bound; 60# white opaque cougar or
approved equivalent paper will be used for answer documents and interior pages of test books.
• Test documents should have high-quality, attractive printing and design
o Preferred: Covers are printed in one color of ink plus black; Interior pages are printed in one color,
some with both scannable and non-scannable ink; Color coding or other identification marks are
included on the spine of the documents to identify them when stacked.
Test Documents – Specific Considerations:
1. Grades 3-11 ELA/L Test Documents
• Up to 40 forms of scannable, sealed test and answer and/or answer documents
• Number of document types may vary if different portions of a test are in separate documents (e.g.,
writing folders)
• 1 Reading Passage Booklet as a CBT test accommodation.
• 1 standalone CBT Worksheet for each CBT tester.
2. Grades 3-8 Mathematics Documents
• Up to 40 forms of scannable, sealed test and answer and/or answer documents
• Number of document types may vary if different portions of a test are in separate documents (e.g.,
performance tasks)
• 1 standalone CBT Work Folder for each CBT tester.
• One standalone reference sheet per student, as applicable
Florida Standards Assessments, ITN 2014-47
108
3. Grade 10 ELA/L Retake
• Preferably, one form of a scannable, sealed test and answer and/or answer document
• Number of document types may vary if different portions of a test are in separate documents (e.g.,
performance tasks)
• 1 Reading Passage Booklet as a CBT test accommodation.
• 1 standalone CBT Worksheet for each CBT tester.
4. End-of-Course Tests
• 1 paper accommodation form of the Algebra 1 End-of-Course test and answer book (scannable,
with an outside seal).
• 1 paper accommodation form of the Geometry End-of-Course test and answer book (scannable,
with an outside seal).
• 1 paper accommodation form of the Algebra 2 End-of-Course test and answer book (scannable,
with an outside seal).
• 1 standalone CBT Work Folder for each CBT tester.
• 1 standalone reference sheet for each Algebra 1, Geometry, and Algebra 2 tester.
5. Other Accommodations
• 1 large print paper accommodation form for each test
• 1 Braille form divided into two volumes for each test
• 1 one-item-per-page accommodation form for each test
6. Accommodation Standalones
• 1 large print reference standalone sheet to accompany each applicable assessment form
• 1 large print response grid instructions standalone sheet to accompany each applicable assessment
form
• 6 pages of blank Braille paper to accompany each Braille assessment form
• 1 large print form of the CBT work folder for each CBT Mathematics assessment
• 1 large print form of the CBT worksheet for each CBT ELA/L assessment
Shipment Schedule and Print Quantities for ELA/L and Mathematics Assessments
• All test materials are shipped in TM2
• All test documents may contain up to 64 pages
• All grades 3-11 tests will have up to 40 test forms
• Grade 10 ELA/L Retake will have one test form
• Schools will receive one test form (with standalones, as applicable) per preidentified student
• Districts will receive 15% overage of all test materials
• For paper-based administration subjects, 10% extra test material is produced
Shipment Schedule and Print Quantities for EOC Assessments
• All test materials are shipped in TM2
• All test documents may contain up to 64 pages
• Spring EOC tests will have up to 4 unique test forms
• Fall, Winter, and Summer EOC tests will have one test form each
• Schools will receive one test form (with standalones, as applicable) per preidentified student
• Districts will receive 15% overage of all test materials
• For paper-based administration subjects, 10% extra test material is produced
Florida Standards Assessments, ITN 2014-47
109
APPENDIX E, Part 2
FLORIDA DEPARTMENT OF EDUCATION
PDF GUIDELINES FOR SECTION 508/ADA DOCUMENTS
The following guidelines may be revised throughout the contract to reflect changes in technology or requirements.
The contractor will be notified of any changes, as appropriate.
In order for PDF documents to be posted on the DOE website or any site that is affiliated with the DOE, they must
pass the Full Accessibility check without error in Adobe Acrobat 9.0 or X.
DOE uses Adobe Acrobat 9.0 and X to check accessibility of PDF documents. Currently, Adobe Acrobat 8.0 has
known issues with the accessibility checker. Although a document may pass the full check, this does not necessarily
mean the document is compliant according to DOE Web standards. The following additional checks are required:
1. PDF documents must have the correct reading order. The file must be able to be read by assistive technology in
a logical manner. This can be checked in several ways, which include the following items.
DOE uses at least two of the following to verify reading order:
a. Saving the file as an accessible .txt file and reading it to be sure it is correct
b. Using the “reading order tool” in Adobe Acrobat
c. If proficient using a “screen reader” (not the Adobe Acrobat “read out loud”), listen to the document
d. Using the Reflow view in Adobe Acrobat
e. Using the “Order” navigation panel
2. The document must have correct Tab order.
Tab order refers to how a user can “tab” through the content. The user should be able to tab through the
document in the order it is intended to be read. If the tab order is incorrect, the assistive technology user may
jump from one page to another and back and not realize it. Another example of incorrect tab order is the
practice of structuring tables or columned lists using tabs.
3. The document must have appropriate “alt text” on all images that have meaning. (Alt text means alternative text
that appropriately describes the image for an assistive technology user.)
a. Graphs, charts, flowcharts, etc., cannot simply have the word “graph” in alt text. It must contain enough
descriptive information for an unsighted user.
b. Graphics that do not have meaning need to be tagged as “artifacts” or “backgrounds.” If the primary purpose
of an image is decoration or if describing an image will take the listener out of the testing experience and
serve more as a distraction to the task at hand, then it would fall under this rule. During regular production
rounds, TDC content leads will determine which elements will be tagged as background.
c. Putting a blank space for the “alt text” area of an image in an attempt to get the screen reader to skip it is
not acceptable in “most” cases, as a screen reader will still say the word “graphic.” Tagging graphics that
have no meaning as artifacts or background will cause the screen reader to ignore them.
d. Graphs and charts need to be described completely to convey the same information to a listener as the
sighted user would get. If the graph is explained in detail in the “content text” of the document, you can refer
to that content in the alternate text. For example, if right below a pie chart is a paragraph explaining the
chart, then the “alt text” on the chart could simply state “Pie Graph which is explained in detail below.” This
is very helpful if the person converting the file to PDF is not the one who created it.
e. Alternate text should not be provided on anything except for image tags.
f. Links are not “clickable” in alternative text; therefore, including links in alternative text should be avoided.
Permission must be obtained from the DOE for special circumstances, such as the approach for tagging the
Periodic Table of the Elements in Interpretive Products.
4. If the State Seal, or any other image, is in the header and footer of a document, the first occurrence of the
header and footer needs to be tagged and readable by a screen reader. All other occurrences should be tagged
Florida Standards Assessments, ITN 2014-47
110
as artifacts or backgrounds. This will let the listener know that there is at least one state seal (or other image)
present, but does not repeat on every page that it appears. This rule also applies to headers and footers that
are repeated on each page of a document. Only new information in the header and footer selections, such as
page numbers, should be tagged.
5. Page numbers need to be tagged in such a way that the screen reader reads them. This is helpful in navigating
the document.
6. Tables must be tagged as tables, so they are read correctly. A common problem with tables is that they are
being tagged as paragraphs.
Tables and table components will be tagged as such: Table headers will have the
tags, table data will
have | tags , and table rows will have | tags. The proper order for tagging a table is as follows:
, , and ; or , , . The header section of a complex table (i.e, a table with
more than one header row) should begin with , and the body section should begin with | . To
check a table, compare the layout in the normal view against the layout in the View menu and select Reflow. If
the layout stays the same and does not shift, the table is properly tagged.
7. Documents greater than 10 pages in length should contain a clickable table of contents, except when it is not
feasible. The table of contents should contain the correct tag structure using and tags. For
documents that contain a table of contents that is not clickable, the table of contents needs to be made into
Bookmarks in the PDF and each table of contents listing needs to be made clickable within the table of contents.
The Bookmarks Panel should appear on the left side of the screen for documents that do not contain a table of
contents but are greater than 10 pages in length. This Bookmarks Panel will act as the document’s table of
contents. During regular production rounds, TDC content leads will determine which elements in TDC
documents will be bookmarked.
This is needed not only for easy navigation for those using assistive technologies, but it is a convenience to
sighted visitors as well, since it allows them to click on an item in the table of contents and be directed
immediately to the content of the document to which it refers.
8. Content headings must be tagged as headers and not tagged as paragraphs.
Headings need to be tagged with tags (or Heading 1 , Heading 2 , etc.). This helps the listener
know what subject matter is contained beneath the header and allows for easier navigation.
9. Files must be saved (in a reduced file size) to Adobe Acrobat version 7.0 with "Fast Web View" enabled for
documents tagged in Adobe Acrobat 9.0 or X. (The DOE has discovered that converting files created in Adobe
Acrobat 9.0 to Adobe Acrobat 5.0 increases rather than decreases the file size, and converting files created in
Adobe Acrobat X to Adobe Acrobat 6.0 or earlier versions distorts the font in interpretive products, so this should
be avoided.) Reducing the file size ensures that users who have older versions of Acrobat are able to access
the document. This also decreases the file size for quicker download.
10. Links must be functional.
a. Hyperlinks must not be broken and must link to the correct page. This includes e-mail addresses that are
clickable.
b. If a link is split between two lines, both lines need to be linked correctly.
c. Hyperlinks need to be standard, which means in blue font and underlined. This allows for sighted users to
quickly locate links within the document.
11. Color should not inhibit the readability of text, and color alone is not to be used to convey meaning. Some
assistive technologies do not reproduce colored text, and users who are colorblind often cannot differentiate
every color.
a. If color is used to convey meaning for sighted users, add a symbol in front of the colored items as well. ag
this symbol to indicate the meaning of the colored item.
Florida Standards Assessments, ITN 2014-47
111
b. Under certain DOE Web Services approved circumstances, the color-coded information may be described
using white text that is not viewable to the sighted user. This “invisible” information must be tagged so that it
is read prior to the colored text.
12. The following information needs to be noted in the Document Properties:
Title–Users can set up a screen reader to read the titles instead of the long file names.
Subject–Specific to the document, and may or may not be the same as the document title.
Keywords–As with other Web documents, these words are used for search engines and need to be specific to
the document’s contents.
Language–(Advanced tab) Specified to indicate the primary language of the document. The individual tag
properties need to indicate text within a document that may be of a different language (e.g., paragraph in
Spanish needs to specify “Spanish” and paragraph in Haitian Creole needs to specify “French”).
Author–At this time, the DOE is not requiring the inclusion of “Author;” however, if the name of the author is
pertinent to the overall document, it is suggested that a name be included.
13. Lists should be tagged correctly and not tagged as text. The correct order options for list tags are as follows:
, , ; , , , ; or , , . If lists have lists within list
items, they need to be tagged correctly. Always tag multiple-choice answer choices/options as lists. Sample
multiple-choice items appear in Sample Test Materials and in Item Specification books.
14. Forms that need to be filled out should contain the appropriate form fields and check boxes so users may fill
them out on the Web. Examples of forms should be indicated as such and do not need to contain form fields
and check boxes, because they are samples. The answer grids for sample testing materials are not considered
forms at this time for feasibility reasons. Form field properties need to be filled out so that the information that
needs to be entered into the form fields is clear. (The "Tooltip" in the form's properties is the portion of the tag
that is read by the screen reader.) If two form fields have the same "Name" in a document, then both fields will
populate when one of them is filled out; therefore, each form field must have a unique name in the form field's
properties. When tagging form fields, it is important to check the accuracy of the Tab order (see list item 2).
15. Pages cannot be blank. They must contain some text, such as “Notes” in the header or the disclaimer, “This
page is intentionally blank.”
16. Pages with content in landscape format should be presented in landscape format, so users may read the content
without having to print the page. In other words, the Pearson contractor shall rotate landscaped pages inserted
sideways so the content appears right side up to the viewer.
17. Words set in bold, underline, or italics may not be used alone to convey meaning. For example, a word set in
italics cannot have a coded meaning such as “italics= story title.”
a. Add a symbol in front of the italicized, bolded, or underlined information, and tag the symbol to indicate the
meaning italics, bolding, or underlining should convey.
b. Under certain DOE Web Services approved circumstances, the italicized, bolded, or underlined information
may be described using white text that is not viewable to the sighted user. This “invisible” information must
be tagged so that it is read prior to the colored text.
18. No text should be entered in the “Actual Text” field in the Touchup Properties box. If that is done, some readers
will only read the actual text and then stop instead of reading the entire document.
Note: DOE PDF accessibility guidelines were initially effective September 11, 2007. They were revised by DOE Web
Services to include Adobe Acrobat 9.0 on January 5, 2010, and by DOE Editorial staff on February 17, 2010, for
added clarifications and for the purposes of this document. Some DOE staff are now using Adobe Acrobat X, and the
Department is currently in the process of officially updating PDF accessibility guidelines to include Adobe Acrobat X
(3/27/12).
Florida Standards Assessments, ITN 2014-47
112
APPENDIX E, PART 3 – Ancillary Materials Shipment Schedule & Print Quantities
Product
Shipment Pages
Number of Copies
State
District
Schools
Electronic
Delivery
(508/ADA
compliant)
Test Administration Manual
TM1
Up to 400
25
10% overage
1/20 students
Student PreID Labels (Wave 1)
TM2
N/A
N/A
N/A
1/student
Student PreID Rosters (Wave 1)
Document Count Forms (Preidentified)
TM2
N/A
N/A
N/A
TM2
N/A
N/A
N/A
Document Count Forms (NonPreidentified)
TM2
N/A
N/A
N/A
1 per subject
2 per
subject/grade
5 per
subject/grade
Online Comment Forms
N/A
N/A
N/A
N/A
N/A
Yes
Security Checklist
N/A
variable
0
1
1
Yes
District Packing List
TM1, TM2,
TM3
variable
0
1
0
Yes
School Packing List
TM1, TM2,
TM3
variable
0
1
1
Yes
District Materials Return Kit
TM2
N/A
1
1
0
School Materials Return Kit
Return Materials (e.g., envelopes,
boxes, special formats)
TM2
N/A
1
0
1
TM2
N/A
0
As necessary
As necessary
N/A
0
As necessary
As necessary
Bills of Lading
Paper Bands
TM2
N/A
10
15% over
1/20 student
documents
Student PreID Labels (Wave 2)
TM3
N/A
0
0
1/student (As
necessary)
TM3
N/A
0
0
1/subject (As
necessary)
As necessary
Yes
As necessary
(Spring ELA/L and Mathematics only)
Student PreID Rosters (Wave 2)
(Spring ELA/L and Mathematics only)
Misc. Memoranda
Florida Standards Assessments, ITN 2014-47
variable
113
APPENDIX F
TEST SECURITY REQUIREMENTS, STATUTE, AND RULE
Chapter 1008.24 of Florida Statutes and Florida State Board of Education Rule 6A-10.042 establish the requirement
that Florida Department of Education tests are to be maintained in a secure manner during development,
administration, and scoring in order to preserve the integrity of the tests. When not in use, all test materials are to be
kept in secure, locked storage. Individuals who have access to secure test materials are not to copy or otherwise
reproduce test questions or reveal test questions verbally or in writing. Persons who are involved in administering or
proctoring the tests or preparing examinees for the tests are not to participate in, direct, aid, counsel, assist in, or
encourage any activity which could result in the inaccurate measurement or reporting of the examinees'
achievement. Examinees' answers to questions are not to be interfered with in any way by persons administering or
scoring the tests. Persons violating test security requirements are guilty of a first degree misdemeanor, punishable
by a fine of not more than $1,000.00 or imprisonment for not more than 90 days, or both.
The security requirements and penalties established by the rule and statute must be provided by the contractor to
each person who has access to tests or test questions during the development, printing, administration, or
scoring of the tests.
A copy of the Statute and Rule is part of this appendix.
Restrictions on Printing for Security Purposes
The prospective contractors should indicate their ability to comply with the following conditions relative to printing of
the required tests. These conditions are necessary as a means of maintaining test security. Lack of compliance with
these conditions may subject the reply to rejection if the Department determines it is in its best interest to do so.
1. All test negatives and plates must be maintained under lock and key by the printing supervisor.
2. Unauthorized personnel must not be permitted access to the test negatives, plates, or copies.
3. All plates and negatives must be destroyed by the contractor upon completion of this contract.
4. The Department reserves the right to conduct on-site spot checks of the printing processes.
5. All press pull-ins, trim, and waste material must be shredded at the end of each day's press run by a person
authorized to do so by the contractor.
6. Each production run must be made under close supervision of the printing supervisor.
THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK
Florida Standards Assessments, ITN 2014-47
114
FLORIDA TEST SECURITY STATUTE
1008.24
Test administration and security.—
(1)
A person may not knowingly and willfully violate test security rules adopted by the State Board of
Education for mandatory tests administered by or through the State Board of Education or the Commissioner of
Education to students, educators, or applicants for certification or administered by school districts pursuant to s.
1008.22, or, with respect to any such test, knowingly and willfully to:
(a)
Give examinees access to test questions prior to testing;
(b)
Copy, reproduce, or use in any manner inconsistent with test security rules all or any portion of any
secure test booklet;
(c)
Coach examinees during testing or alter or interfere with examinees’ responses in any way;
(d)
Make answer keys available to examinees;
(e)
Fail to follow security rules for distribution and return of secure test as directed, or fail to account for all
secure test materials before, during, and after testing;
(f)
Fail to follow test administration directions specified in the test administration manuals; or
(g)
Participate in, direct, aid, counsel, assist in, or encourage any of the acts prohibited in this section.
(2)
A person who violates this section commits a misdemeanor of the first degree, punishable as provided
in s. 775.082 or s. 775.083.
(3)
A school district may contract with qualified contractors to administer and proctor statewide
standardized assessments required under s. 1008.22 or assessments associated with Florida approved courses
under s. 1003.499, as approved by the Department of Education in accordance with rules of the State Board of
Education. Assessments may be administered or proctored by qualified contractors at sites that meet criteria
established by rules of the State Board of Education and adopted pursuant to ss. 120.536(1) and 120.54 to
implement the contracting requirements of this subsection.
(4) (a) A district school superintendent, a president of a public postsecondary educational institution, or a
president of a nonpublic postsecondary educational institution shall cooperate with the Commissioner of Education in
any investigation concerning the administration of a test administered pursuant to state statute or rule.
(b) The identity of a school or postsecondary educational institution, the personally identifiable information
of any personnel of any school district or postsecondary educational institution, or any specific allegations of
misconduct obtained or reported pursuant to an investigation conducted by the Department of Education of a testing
impropriety are confidential and exempt from s. 119.07(1) and s. 24(a), Art. I of the State Constitution until the
conclusion of the investigation or until such time as the investigation ceases to be active. For the purpose of this
paragraph, an investigation shall be deemed concluded upon a finding that no impropriety has occurred, upon the
conclusion of any resulting preliminary investigation pursuant to s. 1012.796, upon the completion of any resulting
investigation by a law enforcement agency, or upon the referral of the matter to an employer who has the authority to
take disciplinary action against an individual who is suspected of a testing impropriety. For the purpose of this
paragraph, an investigation shall be considered active so long as it is ongoing and there is a reasonable, good faith
anticipation that an administrative finding will be made in the foreseeable future. This paragraph is subject to the
Open Government Sunset Review Act in accordance with s. 119.15 and shall stand repealed on October 2, 2014,
unless reviewed and saved from repeal through reenactment by the Legislature.
(5)
Exceptional students with disabilities, as defined in s. 1003.01(3), shall have access to testing sites.
The Department of Education and each school district shall adopt policies that are necessary to ensure such access.
History.—s. 370, ch. 2002-387; s. 1, ch. 2009-143; s. 7, ch. 2013-225.
Florida Standards Assessments, ITN 2014-47
115
Florida Test Security State Board of Education Rule
6A-10.042 Maintenance of Test Security.
(1) Tests implemented in accordance with the requirements of Sections 1004.93, 1008.22, 1008.29, 1008.30,
1012.55, and 1012.56, Florida Statutes, shall be maintained and administered in a secure manner such that the
integrity of the tests shall be preserved.
(a) Test questions shall be preserved in a secure manner by individuals who are developing and validating the
tests. Such individuals shall not reveal in any manner, verbally or in writing, the test questions under development.
(b) Tests or individual test questions shall not be revealed, copied, or otherwise reproduced by persons who are
involved in the administration, proctoring, or scoring of any test.
(c) Examinees shall not be assisted in answering test questions by any means by persons administering or
proctoring the administration of any test.
(d) Examinees’ answers to questions shall not be interfered with in any way by persons administering,
proctoring, or scoring the examinations.
(e) Examinees shall not be given answer keys by any person.
(f) Persons who are involved in administering or proctoring the tests or persons who teach or otherwise prepare
examinees for the tests shall not participate in, direct, aid, counsel, assist in, or encourage any activity which could
result in the inaccurate measurement or reporting of the examinees’ achievement.
(g) Each person who has access to tests or test questions during the development, printing, administration, or
scoring of the tests shall be informed of specifications for maintaining test security, the provisions in statute and rule
governing test security, and a description of the penalties for breaches of test security.
(h) During each test administration, school district and institutional test administration coordinators and
contractors employing test administrators and proctors shall ensure that required testing procedures are being
followed at all test administration sites. Officials from the Department are authorized to conduct unannounced
observations of test administration procedures at any test administration site to ensure that testing procedures are
being correctly followed.
(2) Test materials, including all test booklets and other materials containing secure test questions, answer keys,
and student responses, shall be kept secure and precisely accounted for in accordance with the procedures specified
in the examination program administration manuals and other communications provided by the Department. Such
procedures shall include but are not limited to the following:
(a) All test materials shall be kept in secure, locked storage prior to and after administration of any test.
(b) All test materials shall be precisely accounted for and written documentation kept by test administrators and
proctors for each point at which test materials are distributed and returned.
(c) Any discrepancies noted in the number or serial numbers of testing materials received from contractors shall
be reported to the Department by designated institutional or school district personnel prior to the administration of the
test.
(d) In the event that test materials are determined to be missing while in the possession of an institution or
school district, designated institutional or school district personnel shall investigate the cause of the discrepancy and
provide the Department with a report of the investigation within thirty (30) calendar days of the initiation of the
investigation. At a minimum, the report shall include the nature of the situation, the time and place of occurrence,
and the names of the persons involved in or witness to the occurrence. Officials from the Department are authorized
to conduct additional investigations.
(e) In those cases where the responsibility for secure destruction of certain test materials is assigned by the
Department to designated institutional or school district personnel, the responsible institutional or school district
Florida Standards Assessments, ITN 2014-47
116
representative shall certify in writing that such destruction was accomplished in a secure manner.
(f) In those cases where test materials are permitted by the Department to be maintained in an institution or
school district, the test materials shall be maintained in a secure manner as specified in the instructions provided by
the Department. Access to the materials shall be limited to the individuals and purposes specified by the
Department.
(3) In those situations where an employee of the educational institution, school district, or contractor, or an
employee of the Department suspects a student of cheating on a test or suspects other violations of the provisions of
this rule, a report shall be made to the department or test support contractor, as specified in the test administration
procedures, within ten (10) calendar days. The report shall include a description of the incident, the names of the
persons involved in or witness to the incident, and other information as appropriate. Officials from the Department
are authorized to conduct additional investigations.
(4) Violations of test security provisions shall be subject to penalties provided in statute and State Board Rules.
Specific Authority 1001.02(1), 1008(24)(1) FS. Law Implemented 1001.02, 1008.24 FS. History–New 7-5-87, Amended 10-26-94.
Florida Standards Assessments, ITN 2014-47
117
APPENDIX G
TEST ADMINISTRATION MANUAL WORKFLOW
The production stages for test administration manuals are defined as follows and are first defined for the Department, and
then defined for the contractor. This is provided to help the respondent anticipate the level of work required on the review
rounds:
Table G.1 Test Administration Manual Workflow for FDOE and Contractor for Developing Florida-specific
Standalone Products
Table G.1 Test Administration Manual Workflow for FDOE and Contractor
Round
Task
FDOE
Manuscript
(MS) Round
Identify which prior administrations’ TAMs to use as a starting point for each section of the
current manual.
Identify high-level changes, additions, and omissions to the scripts and other sections to be
included.
Introduce known or potential policy updates.
Verify accuracy of all FDOE policies outlined in TAM; apply content; policy updates, where
necessary.
Review all TAM text by comparing current TAM to most recent and most relevant prior
TAM(s); verify that relevant text from each section of the prior TAM is reflected in current
TAM.
Identify any new naming conventions that need to be incorporated (e.g., changing
“calculator” to “approved four-function calculator”); note any global edits.
FDOE
Round
1
Verify accuracy of links, email addresses, phone numbers, statutes, dates, and schedules.
Review instructions in Test Administrator (TA), School Coordinator (SC), and Technology
Coordinator (TC) sections to verify consistency (e.g., items packed in DAC box).
Confirm TAM accurately indicates which ancillaries will be posted; verify file format(s).
Add queries for contractor program team in green text boxes.
Include placeholders (text in purple text box) in sections to be addressed at Round 2
(e.g., outstanding policy decisions).
File clean-up before returning TAM to contractor:
Change internal queries from orange to purple.
Remove FDOE internal queries where possible.
Florida Standards Assessments, ITN 2014-47
118
Round
Task
Confirm that all Round 1 edits have been applied.
Confirm that queries to contractor from Round 1 have been addressed.
Replace all placeholders with content.
For assessments with CBT & PBT TAMs:
Review instructions in PBT TAM against instructions in CBT TAM for consistency.
FDOE
Round
2
Compare CBT and/or PBT test form to TAM to
o Verify accuracy of scripts (incl. electronic practice test scripts),
o Verify instructions,
o Verify seal configurations,
o Verify page references,
o Verify titles, and
o Verify tool descriptions.
Verify session schedule references in scripts (e.g., Session 2, Day 2); verify session length
references in script.
Ensure screen shots and instructions are up-to-date.
Respond to contractor queries (light blue text boxes); apply checkmarks to approve
suggested edits, as needed.
Ensure that contractor program team checks have been made; include reminders/queries
regarding checks that must be performed; schedule conference call/WebEx, if necessary, to
discuss questions/concerns regarding Program Team (PT) verification.
Conduct roundtable read-through, comparing to multiple prior TAMs for consistency.
Ensure FDOE specialists (e.g., individuals from other offices) verify information (e.g., any
updates to students to be tested, any updates to ESE accommodation information)
File clean-up before returning TAM to contractor:
Change internal queries from orange to purple.
Remove FDOE internal queries where possible.
Florida Standards Assessments, ITN 2014-47
119
Round
Task
Confirm that all Round 2 edits have been applied.
Confirm that queries to contractor from Round 2 have been addressed.
Verify contractor program team checks have been performed by appropriate program team
members.
FDOE
Round
3
Check TOC, headers, footers, pagination, and internal page references for accuracy.
Perform a cold read of the entire TAM to identify any final text edits to be made.
Verify all FPOs (placeholders) have been replaced with accurate, up-to-date graphics.
Verify checklists and ancillaries in appendices for accuracy and consistency (i.e., that forms
are approved versions).
Review third-party proofreader’s queries/suggested edits; apply checkmarks to approve
suggested edits or indicate revised text using verbatim text edits in yellow text boxes.
File clean-up before returning TAM to contractor.
Remove FDOE internal queries.
Round
FDOE
Final
Round
Task
Confirm that all Round 3 edits have been applied.
Confirm that queries to contractor from Round 3 have been addressed.
Perform a final visual check of the document.
Florida Standards Assessments, ITN 2014-47
120
Round
Task
Draft new TAM using TAM Information Sheet and MS round FDOE markup in previous
TAM.
Communicate with FDOE about edits to be applied based on MS round markup. Identify
version(s) of previous TAM from which language has been pulled if not the approved final
version (e.g., language in x, y, and z sections of CBT TAM pulled form EOC TAM round 3).
Include queries to FDOE for sections that require confirmation, revision, or additional
information.
Contractor
Round
1
Insert FPO (placeholder) graphics; insert forms/images from contractor P&D program team
when available and note the following:
o Graphics should depict information for the applicable admin, such as correct start
date and logical start time.
o For CBT: Screenshots should be taken using the correct user profile (e.g., CBT
Coordinator).
o Sample students should have age appropriate date-of-birth fields and grade levels
should match the administration.
Perform a clean read of TAM and insert queries where text may need to be updated based
on comments at debrief, comment form feedback analysis, call center data, etc.; suggest
changes; highlight any tentative revisions for FDOE to accept/reject/revise.
File clean-up before posting TAM to FDOE:
Remove internal queries where possible.
Florida Standards Assessments, ITN 2014-47
121
Round
Task
Apply FDOE edits. If an FDOE edit seems incorrect or inappropriate, contact FDOE to inquire.
All edits should be applied unless, during contractor’s review, FDOE is consulted and agrees
to leave out edit.
Consult appropriate contractor program staff (per Florida Program Team org chart) and
acquire approval/sign-off on relevant script, technical, and P&D sections of TAM, including a
review of naming conventions for P&D materials.
Insert FPO (placeholder) graphics; insert forms/images from contractor P&D program team
when available and note the following:
o Graphics should depict information for the applicable admin, such as correct start
date and logical start time.
o For CBT: Screenshots should be taken using the correct user profile (e.g., CBT
Coordinator).
o Sample students should have age appropriate date-of-birth fields and grade levels
should match the administration.
Contractor
Round
2
Perform a role play to check operational scripts against currently available test forms/books
and verify accuracy of instructions, page references, screen shots, tool descriptions, etc.
Verify session schedule references in scripts (e.g., Session 2, Day 2); verify session length
references in script.
Verify the online management system instructions by intended user role.
Verify the online management system and computer-based platform screen shots.
Verify emails, phone numbers, URLs, dates, and schedules.
Verify which ancillaries will be posted on the online management system, and in what file
format(s) (check against approved ancillaries matrix).
Perform a clean read and provide queries/suggested additional edits. Suggested edits should
be indicated using blue comment boxes for FDOE to approve or reject in the next review
round (i.e., should not be incorporated into the body of the TAM prior to FDOE approval and
should not be made using yellow text boxes, which could be mistaken for FDOE verbatim
edits).
Indicate content that has and has not yet been verified (e.g., add a text box indicating that
school box contents have not yet been verified).
File clean-up before posting TAM to FDOE:
Remove contractor internal queries.
Replace FDOE queries, as necessary, and ensure contractor responses are included.
Florida Standards Assessments, ITN 2014-47
122
Round
Task
Apply FDOE edits. If an FDOE edit seems incorrect or inappropriate, contact FDOE to
inquire. All edits should be applied unless, during contractor’s review, FDOE is consulted and
agrees to leave out edit.
Confirm all FPOs have been replaced with updated artwork.
o Graphics should depict information for the applicable admin, such as correct start
date and logical start time.
o For CBT: Screenshots should be taken using the correct user profile (e.g., CBT
Coordinator).
o Sample students should have age appropriate date-of-birth fields and grade levels
should match the administration.
Pearson
Round
3
Verify online management system instructions are accurate and up to date.
Ensure all program team reviews are complete and that verification is indicated in TAM; where
suggested edits arise during PT verification, these edits should be placed in blue comment
boxes for FDOE approval prior to incorporation into text.
Perform final full review of entire TAM for content, formatting, consistency of information and
instructions across sections (e.g., TA, SC, and TC have same instructions and understanding
of responsibilities).
Consult appropriate contractor program staff (per Florida Program Team org chart) and
acquire approval/sign-off on relevant script, technical, and P&D sections of TAM.
Verify TOC, headers/footers, pagination, internal page references.
File clean-up before posting TAM to FDOE:
Remove contractor internal queries.
Replace FDOE queries, as necessary, and ensure contractor responses are included.
Round
Task
Confirm all edits have been applied.
Confirm that approved versions of all checklists and ancillaries are being used.
Contractor
Final
Round
Perform a final, full clean read.
File clean-up before posting TAM to FDOE:
Remove contractor internal queries.
Replace FDOE queries, as necessary, and ensure contractor responses are included.
Confirm that when the final version is posted to FDOE, no marks or comments are included except
where the copyeditor has made an unrequested edit or where FDOE had a query and contractor is
providing a response.
Florida Standards Assessments, ITN 2014-47
123
APPENDIX H
TEST ACCOMMODATIONS
Test Accommodations for Students with Disabilities (General Information)
As specified in this appendix, accommodations may be provided to students with disabilities enrolled in public
schools with current IEPs or Section 504 plans. Care must be taken to provide a student with only the
accommodations permitted for that student. In addition, students with disabilities must be provided the opportunity to
participate in practice activities for the tests with appropriate allowable accommodations.
Determination of appropriate accommodations in assessment situations for students with disabilities shall be based
on the individual needs of each student. Decisions on accommodations shall be made by the IEP or Section 504
team and recorded on the IEP or Section 504 plan. Students with disabilities should be oriented to any test situation
through test-taking instruction designed to familiarize them with testing format and procedures. This orientation
should take place near the time of testing. Guidelines recommended for making accommodation decisions include:
1. Accommodations should facilitate an accurate demonstration of what the student knows or can do.
2. Accommodations should not provide the student with an unfair advantage or interfere with the validity of a test;
accommodations must not change the underlying skills that are being measured by the test.
3. Accommodations must be the same or nearly the same as those needed and used by the student in completing
classroom instruction and assessment activities.
4. Accommodations must be necessary for enabling the student to demonstrate knowledge, ability, skill, or
mastery.
Students with disabilities who are not currently enrolled in public schools or receiving services through public school
programs and require accommodations in order to participate in a test administration may have access to
accommodations if the following information is provided:
•
•
Evidence that the student has been found eligible as a student with a disability as defined by s. 1003.01(3) (a),
F.S., or Rule 6A-19.001(6), FAC, and
Documentation that the requested accommodations are regularly used for instruction.
For further information, contact the Bureau of Exceptional Education and Student Services at 850-245-0475 or visit
the following website: www.fldoe.org/ese/fcatasd.asp.
Computer-Based Accommodations for Students with Disabilities
Students with disabilities may participate in CBT administrations using computer-based accommodations as
indicated on their IEPs or Section 504 plans. CBT accommodations may be included in the category of flexible
presentation accommodations for CBT administrations. They may be provided to students along with other
allowable test administration accommodations.
CBT accommodations must be accessible through the computer-based testing platform that will be used for nonaccommodated students. Accommodations should be administrator-selectable.
A. Large Print Font
A fixed large print font size (minimum 18-point) that is an enlargement of the regular type is available for
students who require it. Computer interface (scrollbar, buttons, cursor/pointer, etc.) is also enlarged accordingly.
B. Color Contrast
Options for variable font color and background color display preselected font and background color combinations
on the computer screen. Specific highlighter colors are also provided for students who require them.
The color combinations may include, but are not limited to, the following:
• White Text/Black Background (Blue Tracking; Green Highlighter)
• Black Text/Beige Background (Green Tracking; Yellow Highlighter)
Florida Standards Assessments, ITN 2014-47
124
•
Black Text/Light Green Background (Yellow Tracking; Orange Highlighter)
Without the use of color contrast, the screen displays the following:
• Black Text/White Background (Blue Tracking; Blue Highlighter)
C. Zoom
Feature allows students to self-select print size and magnification. Zoom levels include: 1x, 1.25x, 1.5x, 1.75x,
2x, 2.25x, 2.5x, 2.75x, 3x, 3.5x, 4x, 5x, 6x, 7x, 8x, 10x. Computer interface (scrollbar, buttons, cursor/pointer,
etc.) also zooms accordingly.
D. Screen Reader
The screen reader provides text-to-speech audio presentation of directions, test items, and answer choices. The
Department will provide the contractor with screen reader specifications at the beginning of the contract detailing
the various requirements and procedures for read-aloud text and technology.
E. Assistive Devices
The assistive devices form allows the use of computer-based assistive devices with no other accommodated
features enabled for students who require assistive devices but do not require other accommodated CBT forms.
Paper-Based Accommodations for Students with Disabilities
Students with disabilities may participate in CBT administrations using paper-based accommodations as indicated on
their IEPs or Section 504 plans. Paper-based accommodations may be included in the category of flexible
presentation accommodations for CBT administrations and may be provided to students along with other allowable
test administration accommodations.
In order to receive paper-based test materials for any CBT administration, the requirement for a paper-based test OR
one or more of the following conditions should be documented on a student’s IEP or Section 504 plan:
• The student cannot access a computer.
• The student requires an accommodation that cannot be implemented in conjunction with a computer-based
administration (e.g., the student requires a specialized setting or uses adaptive furniture that cannot
accommodate a computer workstation).
• The student requires an accommodated CBT form that is not available.
• The student has a flexible scheduling/extended time accommodation that requires testing one session over more
than one day.
• The student is hospital/homebound.
Other conditions, in addition to those listed above, may be considered by the student’s IEP or Section 504 plan team.
A. Regular Print
A regular print paper version of the test may be requested for a student who requires it.
B. Braille
A contracted or uncontracted Braille version may be requested for a student who uses Braille materials. Some
test items may be altered in format for Braille versions of the test as authorized by the Department. Test items
that have no application for the Braille reader will be deleted as authorized by the Department. Student
performance standards that cannot be assessed in the Braille format will be deleted according to the
requirements of s. 1008.22, F.S.
C. Large Print
A paper-based large print version of the test may be requested as a unique accommodation for eligible students
who are unable to access the computer-based large print or zoom accommodation via the computer-based
testing platform.
Florida Standards Assessments, ITN 2014-47
125
D. One-Item-Per-Page
One-item-per-page materials may be requested as a unique accommodation for paper-based tests for use by
students who require one test item on a page, fewer test items on a page, increased space between test items,
or true black-and-white print.
E. Reading Passage Booklet
Eligible students participating in computer-based Reading tests may be provided regular print or large print
booklets that contain the reading passages but do not contain the test items. Although Reading Passage
Booklets are included in the category of paper-based accommodations, students who require them will test on
the computer.
Administration Accommodations for Students with Disabilities
Administration accommodations must be provided as indicated on student IEPs or Section 504 plans. Listed below
are allowable administration accommodations for Florida’s statewide assessments. Not all allowable
accommodations are listed.
A. Flexible Presentation
• Paper-based test materials (regular print, braille, large print) and accommodated computer-based test forms
(large print font, color contrast, zoom, screen reader) may be requested for students who require them.
• Oral presentation may be provided for all directions and for items/answer choices other than passages and
items/answer choices in Reading tests. Reading passages, items, and answer choices may not be read
aloud or presented orally, but must be read by the student through visual or tactile means.
Mathematics, science, and history items/answer choices may be read aloud by a test administrator
or presented orally via screen reader.
• Signed presentation may be provided for all directions and for items/answer choices other than passages
and items/answer choices in Reading tests. Reading passages, items, and answer choices may not be
signed, but must be read by the student through visual or tactile means. The test administrator may
translate using the same method of sign language that the student regularly uses in the classroom, but must
be careful not to use signs that might lead the student to the correct response. In such cases, fingerspelling
may be used as an alternative.
• For oral or signed presentation, the test administrator may sign or read aloud directions, items, and answer
choices, other than passages and items/answer choices in Reading tests, to the student in the manner that
is regularly used in the classroom. Examples of allowable oral/signed presentation include, but are not
limited to, reading/signing items to a group of students, reading/signing every item to a student individually,
reading/signing items only when a student requests.
• Test directions may be repeated, clarified, or summarized.
• A student may be provided with a copy of directions from the administration script that is read by the test
administrator.
• A student may be allowed to demonstrate that he or she understands the directions (e.g., repeating or
paraphrasing).
• A student may read aloud test items and answer choices to him/herself. This would require use of a device
such as a WhisperPhone®, or would require that the student be tested in a separate setting so that other
students are not disturbed.
• Verbal encouragement (e.g., “keep working,” “make sure to answer every question”) may be used; however,
it may not be used to cue a student regarding correct/incorrect responses.
• Students may use magnification devices (e.g., CCTV/video magnifiers, reading loupes, handheld
magnifiers). Devices must be used without accessing image-upload features (e.g., devices with a memory
card must have the memory card removed during testing).
• A student may use a straightedge to maintain or enhance visual attention to test items.
Florida Standards Assessments, ITN 2014-47
126
•
•
•
•
•
Portions of the test may be masked to direct attention to uncovered item(s).
Colored transparencies/overlays may be used.
Test books may be secured to a work area if no adhesives are used on scannable documents. If
necessary, scannable documents may also be secured with adhesives if responses are transcribed into
replacement documents.
Positioning tools, such as a reading stand, may be used.
A student may use a highlighter to highlight key words or phrases in directions, passages, test items, and
answer choices.
B. Flexible Responding
• The student may use varied methods to respond to the test, including written, signed, and verbal responses.
(Written responses may include the use of devices such as the Graphic Aid for Mathematics or the
geoboard for students using braille.) A test administrator or proctor may transcribe student responses to the
format required by the test. Recorded responses must accurately reflect the response of the student,
without addition or enhancement by the test administrator or proctor.
• A student may dictate responses to a proctor.
• A student may use speech-to-text technology to record answers.
• A student may indicate answers to test items using the mouse, keyboard, or assistive technology that uses
either device.
• A student may use a computer switch to indicate answers.
• A student may use a computer/alternative keyboard to indicate answers.
• A student may use a pointing device to indicate answers.
• A student may use other communication devices to indicate answers.
• A student may use mathematics grids/guides to organize mathematical computation.
• A student may use special paper such as raised line, shaded line, or color-coded (would require that
responses are then transcribed by school/district personnel).
• Test administrators may check periodically to be sure that students are entering answer choices correctly on
the computer or marking in the correct spaces in the test book.
C. Flexible Scheduling
• A student may be administered a test during several brief sessions, allowing frequent breaks during the test
sessions. Between sessions and during breaks (e.g., restroom breaks, lunch breaks), students must be
closely monitored to ensure that they do not share responses or change responses to items that were
already completed.
• A student may use a specific time of day for specific sessions.
• A student may be provided extended time to complete a test. Extended time must be provided in
accordance with the student’s IEP or Section 504 plan. Extended time is not unlimited time; it should
align with the accommodation used regularly in the student’s classroom instruction and assessments. The
student is not required to use all of the extended time that is allowed and may end the testing session prior
to the expiration of the extended time.
• Other considerations for Scheduling accommodations when administering Florida’s statewide assessments
to students with disabilities include:
• Each test or test session must be completed within one school day. Students who have a flexible
scheduling accommodation that allows them to test over more than one day will be required to participate in
testing using paper-based test materials.
Florida Standards Assessments, ITN 2014-47
127
•
For students using paper-based test materials to test over more than one day, the following conditions must
be maintained to ensure the validity of the test administration across days:
The student may not be permitted to change his or her responses to items that were completed on a
previous day.
The test administrator must closely supervise the administration of the test on an individual basis to
ensure that answers from the previous day are not changed and that the student does not preview parts
of the test to be answered the following day.
The test administrator must use a paper clip or binder clip to secure the answer documents and prevent
the student from reviewing his or her answers from the previous day. ALL CLIPS MUST BE
REMOVED AFTER TESTING. Staples or tape should NOT be used on books containing student
answers.
D. Flexible Setting
• A student may be administered a test in a familiar place with a test proctor present, and/or a familiar person
who has been appropriately trained may administer the test. Procedures for test security must also be
followed and precautions taken in order to ensure that the test remains secure.
• A student may be administered a test individually or in a small group setting. A small group should be of a
size comparable to the normal instruction group size indicated on the student’s IEP or Section 504 plan.
• If a student requires use of an accommodation that may disturb other students, the student should be tested
in a separate setting.
• A student may use special lighting.
• A student may use adaptive or special furniture.
• Special acoustics, such as FM systems, may be used to enhance sound or special rooms may be used to
decrease auditory distractions.
• The opportunity for movement may be increased or decreased.
• Stimuli may be reduced (e.g., by limiting the number of items around the student’s desk or computer
station).
• White noise/sound machines or approved music may be used to reduce auditory distractions.
E. Assistive Devices and Tools
• Any student who requires software programs to ensure access to the computer and to approved assistive
technology (e.g., switch scanning software, specialized on-screen keyboards, mouse emulators) must be
able to use these devices with the computer-based testing platform.
• A student may use varied assistive devices to access the computer (e.g., alternative keyboards, trackballs,
joysticks, switch scanning systems, touch screens, mouth sticks, head wands, eye-gaze or head control
systems).
• Visual magnification and auditory amplification devices may be used.
• Real coins may be used to represent coins depicted in test items. Coins may not be used as counting
devices.
• For students with visual impairments, an abacus and products such as the Graphic Aid for Mathematics,
Math Window®, and the geoboard may be used. These devices are approved substitutes for paper and
pencil computation.
• Other assistive technology typically used by the student in classroom instruction may be used provided the
purpose of the testing is not violated. Implementation of assistive devices must ensure that test responses
are the independent work of the student.
• Students who use sign language as their primary means of communication may use an English/sign or
sign/English translation dictionary. The dictionary must be similar to one used in the instructional setting
Florida Standards Assessments, ITN 2014-47
128
•
and may not contain definitions of words. The dictionary may contain the sign picture, the word, synonyms,
and an index.
Manipulative materials, including, but not limited to, counters, base-10 blocks, clock faces, or geometric
shapes, are not allowed to be used by any student during the test administration.
FOR BRAILLE MATERIALS ONLY: While the use of manipulatives is not an allowable accommodation,
occasionally an item in a Braille test will require the use of manipulatives to give the student a tactile
reference. If/when such items occur, test administrators will be provided with the instructions allowing
students to use the manipulatives for those specific test items only. This modification is approved by
the Department and applies only to students using Braille test materials.
Unique Accommodations
In accordance with Rule 6A-1.0943, FAC, school districts may request unique accommodations for individual
students with disabilities. Unique accommodations usually involve alterations of existing test materials. In addition,
they must be regularly used by the student for classroom instruction and must not alter the underlying content of the
test. Each unique accommodation must be approved by the Commissioner of Education or a designee prior to its
use. Written requests for unique accommodations must be submitted by the district assessment coordinator to the
Bureau of K-12 Student Assessment at the FDOE. Accommodation requests must be reviewed by district level staff
before being sent to the FDOE. In addition, the signatures of both the district exceptional student education (ESE)
coordinator and the district assessment coordinator are required.
Administration Accommodations for English Language Learners (ELLs)
Districts are required to offer accommodations to students identified as ELLs. Allowable accommodations for ELLs
are listed below. The test may be administered with any one of these modifications or a combination of
accommodations that are determined to be appropriate for the particular needs of ELLs. However, all testing, with or
without accommodations, must be completed during the prescribed testing dates.
A. Flexible Setting
• ELLs may be offered the opportunity to be tested in a separate room with the English for Speakers of
Other Languages (ESOL) or heritage language teacher acting as test administrator. Parents must be
informed of this option for students not of legal age and shall be given the opportunity to select the
preferred method of test administration.
B. Flexible Scheduling
• ELLs may take a test session during several brief periods within one school day; however, each test or
test session must be completed within one school day.
• ELLs may be provided additional time; however, each test or test session must be completed within one
school day.
C. Assistance in Heritage Language
• ELLs may be provided limited assistance by an ESOL or heritage language teacher using the student’s
heritage language for all directions and for items/answer choices other than passages and items/answer
choices in Reading tests. This should not be interpreted as permission to provide oral presentation of
items/answer choices in English or in the student’s heritage language.
The ESOL or heritage language teacher may answer student questions about the general test
directions in their heritage language. If the test is administered to a group of students, the teacher may
answer questions about directions for the benefit of the group. Questions of clarification from individual
students must be answered on an individual basis without disturbing other students.
Mathematics, Science, and History items: The teacher may answer specific questions about a word
or phrase that is confusing the student because of limited English proficiency, but is prohibited from
giving assistance that will help the student solve problems and answer test items. A student’s
Florida Standards Assessments, ITN 2014-47
129
questions must not be answered in a way that would lead the student to infer the correct answer to an
item.
D. Approved Dictionary
• ELLs must have access to an English-to-heritage language translation dictionary and/or heritage
language-to-English translation dictionary, such as those made available to ELLs in an instructional
setting. However, a dictionary providing definitions written exclusively in the heritage language or in
English may not be provided.
Florida Standards Assessments, ITN 2014-47
130
APPENDIX I
DATA PRIVACY & SECURITY POLICY
TABLE OF CONTENTS
I.
INTRODUCTION
II. DEFINITIONS
III. PRIVACY OF PERSONAL INFORMATION
A. Basic Privacy Protections
B. Access to the Department PII
IV. INFORMATION SECURITY PROGRAM
A. Information Security Risk Assessment
B. Security Controls Implementation
C. Security Monitoring
D. Security Process Improvement
E. Breach Remediation
F. Organization, Responsibilities and Administration
G. Personnel Security Policy Overview
V. ENFORCEMENT
I.
INTRODUCTION
Personally Identifiable Information ("PII") on students, including information on their performance on the
assessments and limited demographic information, will be collected by the Contractor. Disclosure of PII to the
Department and its contractors is authorized by the Family Educational Rights and Privacy Act ("FERPA") for the
purposes of—
(1) Conducting studies, for or on behalf of the Department and Florida’s school districts, to develop, validate,
and administer predictive tests or to improve instruction (34 CFR 99.31(a)(6)(i)(A) & (C)); and
(2) Assisting in the evaluation of federal- and state-supported education programs and ensuring compliance
with federal legal requirements related to such programs – in particular related to state-level assessments
and accountability systems (34 CFR 99.31(a)(3) & 99.35).
More specifically, the data will be used by the Department or by one or more of the Department’s Contractors—
• to validate, pilot and field test, and improve the assessments;
• to report assessment results back to the Department and Florida school districts in a form that is useful to
them;
• to prepare reports on student performance for the Department and Florida school districts, and the public
(PII may not be included in public reports or in reports to states or school districts that were not the source
of the PII);
• to analyze test results for purposes of accountability, including promotion and graduation decisions for
individual students; teacher and school leader evaluations; school accountability determinations;
determinations of principal and teacher professional development and support needs; and teaching,
learning, and program improvement; and
• to carry out studies designed to improve instruction on behalf of the Department and Florida school districts.
Access to student data remains the legal responsibility of the Department and Florida school districts, in
accordance with FERPA.
Florida Standards Assessments, ITN 2014-47
131
This Policy describes, in general, (i) what steps the Department takes to protect PII that is accessed by or
provided to the Department or the Contractor; (ii) how that information is used; (iii) with whom the Department
shares that information, and (iv) the steps the Department takes to protect the security of that information.
Monitoring of the implementation of this Policy is the shared responsibility of the Department and the Contractor.
Each of the privacy and security provisions in this Policy is effective upon contract execution.
II. DEFINITIONS
Capitalized terms referenced herein but not otherwise defined shall have the meanings as set forth below:
“Authorized User” means an individual Department or Contractor employee authorized by the Department to
access PII maintained by the Department or the Department Contractor.
"Destroy" or "Destruction" means the act of ensuring the PII cannot be reused or reconstituted in a format which
could be used as originally intended and that the PII is virtually impossible to recover or is prohibitively expensive
to reconstitute in its original format.
“FERPA” means the Family Educational Rights and Privacy Act of 1974 (codified at 20 U.S.C. § 1232g) and its
implementing regulations, as they may be amended from time to time. The regulations are issued by the U.S.
Department of Education, and are available at http://www2.ed.gov/policy/gen/reg/ferpa/index.html.
"Contractor" means each Department contractor, including any subcontractor(s), that may be required to
maintain or handle PII in the course of providing services in support of the Department.
"Personally Identifiable Information" (or "PII") means any information defined as personally identifiable
information under FERPA.
III. PRIVACY OF PERSONAL INFORMATION
A. Basic Privacy Protections
1. Compliance with Law and Policy. All PII uploaded to or made accessible to the Department or
Contractor will be handled, processed, stored, transmitted and protected in accordance with all
applicable federal data privacy and security laws (including FERPA) and with this Policy.
2. Training. Employees of the Department (including temporary and contract employees) and the
employees (including temporary and contract employees) of the Contractor are educated and trained
on the proper uses and disclosures of PII and the importance of information privacy and security. Such
training will include training for new employees and refresher training for current employees.
3. Personnel Guidelines. All the Department and Contractor employees are required to be aware of and
work to protect the confidentiality, privacy, and security of PII. The Department, the Contractor, and
their respective personnel shall not access PII except to comply with a legal obligation under federal or
state law, regulation, subpoena or action by the Department that requires such access, or if there is
legitimate need for the information to maintain data systems or to perform required services. The
following list provides a general description of the internal policies with which the Department, the
Contractor, and their respective personnel are required to comply:
–
Limit internal access to PII to the Department and Contractor personnel with proper authorization
and allow use and/or disclosure internally, when necessary, solely to personnel with a legitimate
need for the PII to carry out the educational purposes of the Department.
–
Allow access to PII by external parties only as authorized by the Department.
–
Require that materials containing PII in electronic form are stored solely within encrypted data
repositories and PII are not available on unencrypted shared drives that are used by other users or
on a local drive. PII must be encrypted in transit as well.
Florida Standards Assessments, ITN 2014-47
132
–
When PII is no longer needed, delete access to PII, in accordance with secure destruction
procedures that the Department will provide.
–
Permit the Department and Contractor employees to store or download information onto a local or
encrypted portable devices or storage only when necessary, and to create a written record for
retention verifying that the information is encrypted and stored in password-protected files, and that
devices containing the information have appropriate security settings in place (such as encryption,
firewall protection, anti-virus software and malware protection).
–
Require that any downloaded materials consisting of PII remain in the United States.
–
Prohibit the unencrypted transmission of information from the Department, or any other source of
PII, wirelessly or across a public network to any third party.
B. Access to PII
1. The Department makes the final determination regarding access to PII.
2. Parent Complaints. The Department shall be responsible for handling any complaints from parents (or
students 18 and over) regarding the disclosure or use or accuracy of PII.
3. Privacy Administrators. The Department will designate a privacy administrator who is responsible for
managing access to PII maintained by the Department or the Contractor by designating authorized
users, and for determining the scope of PII to which they have access. The Privacy Administrator shall
be responsible for making all administrative decisions regarding access to and use of PII.
IV. INFORMATION SECURITY PROGRAM
The security of the PII is of critical importance to the Department. The Department's IT Security Program
consists of technical, physical and administrative safeguards to protect the PII. The Department's IT Security
Program is designed to identify, manage and control the risks to system and data availability, integrity, and
confidentiality, and to ensure accountability for system actions. The Department's Security Program includes,
and the security program of each Contractor is required to include, the following key general processes which
may be more fully described in other materials as necessary:
A. Information Security Risk Assessment
The Department periodically conducts, and the Contractor is required to periodically conduct, an accurate
and thorough assessment of the potential risks and vulnerabilities to the confidentiality, integrity, and
availability of electronic, paper, or other records containing PII maintained by the Department and the
Contractor; to report such risks as promptly as possible to the Department or other official within the
Department designated to be responsible for data privacy and security compliance; and to implement
security measures sufficient to reduce identified risks and vulnerabilities. Such measures shall be
implemented based on the level of risks, capabilities, and operating requirements. These measures must
include as appropriate and reasonable the following safeguards:
1. Administrative Safeguards
a. Sanctions: Appropriate sanctions against the Department or contractor personnel who fail to
comply with the Department security policies and procedures, with the potential for criminal referral
if warranted.
b. System Monitoring: Procedures to regularly review records of information systems activity,
including maintaining access logs, access reports, security incident tracking reports, and periodic
access audits.
c. Security Oversight: Assignment of one or more appropriate senior officials within the Department
and the Contractor, as applicable, to be responsible for developing, implementing, and monitoring
of safeguards and security issues.
d. Appropriate Access: Procedures to determine that the access of the Department and the
Contractor personnel to PII is appropriate and meets a legitimate need to support their roles in
Florida Standards Assessments, ITN 2014-47
133
business or educational operations. Procedures for establishing appropriate authorization and
authentication mechanisms for the Department and the Contractor personnel who have access to
PII.
e. Employee and Contractor Supervision: Procedures for regularly monitoring and supervising the
Department and Contractor personnel who have access to PII.
f.
Access Termination: Procedures for terminating access to PII when employment ends, or when an
individual no long has a legitimate need for access.
g. Recording Requests and Disclosures: Disclosures of PII to, and requests for disclosures of PII
from, third parties -- other than employees of the Department or of the Contractor that provided the
PII to the Department -- are recorded.
2. Physical Safeguards
a. Access to PII: Procedures that grant access to PII by establishing, documenting, reviewing, and
modifying a user’s right of access to a workstation, software application/transaction, or process.
b. Awareness Training: On-going security awareness through training or other means that provide
the Department and the Contractor personnel (including management) with updates to security
procedures and policies (including guarding against, detecting, and reporting malicious software).
Awareness training should also address procedures for monitoring log-in attempts and reporting
discrepancies, as well as procedures for safeguarding passwords.
c. Incident Response Plan: Procedures for responding to, documenting, and mitigating where
practicable suspected or known incidents involving a possible breach of security and their
outcomes.
d. Physical Access: Procedures to limit physical access to PII and the facility or facilities in which
they are housed while ensuring that properly authorized access is allowed, including physical
barriers that require electronic control validation (e.g., card access systems) or validation by human
security personnel.
e. Physical Identification Validation: Access is physically safeguarded to prevent tampering and theft,
including procedures to address control and validation of a person’s access to facilities based on
his or her need for access to the PII.
f.
Operational Environment: Procedures that specify the proper functions to be performed, the
manner in which they are to be performed, and the physical attributes of the surroundings of
facilities where PII is stored.
g. Media Movement: Procedures that govern the receipt and removal of hardware and electronic
media that contain PII into and out of a facility.
h. Encryption and Final Disposition of Information: Procedures addressing encryption of all data at
rest and in transit and the final disposition of PII. Procedures must include processes for the
continued encryption of PII through the time when its secure deletion/destruction has been
requested by the Department and the PII is deleted/destroyed.
3. Technical Safeguards
a. Data Transmissions: Technical safeguards, including encryption, to ensure PII transmitted over an
electronic communications network is not accessed by unauthorized persons or groups.
b. Data Integrity: Procedures that protect PII maintained by the Department or the Contractor from
improper alteration or destruction. These procedures will include mechanisms to authenticate
records and corroborate that they have not been altered or destroyed in an unauthorized manner.
Florida Standards Assessments, ITN 2014-47
134
c. Logging off Inactive Users: Inactive electronic sessions shall be designed to terminate
automatically after a specified period of time.
B. Security Controls Implementation
The Department and the Contractor will develop procedures addressing the acquisition and operation of
technology, the specific assignment of duties and responsibilities to managers and staff, the deployment of
risk-appropriate controls, and the need for management and staff to understand their responsibilities and
have the knowledge, skills and motivation necessary to fulfill their duties.
C. Security Monitoring
In combination with periodic security risk assessments, the Department and the Contractor will use a variety
of approaches and technologies to make sure that risks and incidents are appropriately detected, assessed
and mitigated on an ongoing basis. The Department and the Contractor will also assess on an ongoing
basis whether controls are effective and performing as intended, including intrusion monitoring and data loss
prevention.
D. Security Process Improvement
Based on the Department's security risk assessments and ongoing security monitoring, the Department and
the Contractor will gather and analyze information regarding new threats and vulnerabilities, actual data
attacks on the Department and the Contractor, and new opportunities for managing security risks and
incidents. The Department and the Contractor will use this information to update and improve their risk
assessment strategy and control processes.
E. Breach Remediation
The Department and the Contractor strive to keep PII maintained by the Department and the Contractor
secure and to use reasonable administrative, technical, and physical safeguards to do so. The Department
and the Contractor will maintain and update incident response plans that establish procedures to follow in
case a breach occurs. The Department and the Contractor will also identify individuals within their
respective organizations responsible for implementing incident response plans if a breach should occur.
If the Department or the Contractor determine that a breach has occurred, when there is a reasonable risk
of identity theft or other harm, or where otherwise required by law, the Department or the Contractor, as
applicable, will notify affected parties as promptly as possible, and will cooperate as needed to enable
compliance with all breach of confidentiality laws.
The Department and Contractor employee are required to report as promptly as possible to the Department
leadership (or his or her designee) and persons responsible for managing their respective organization’s
incident response plan any incident or threatened incident involving unauthorized access to or acquisition of
PII of which they become aware. Such incidents include any breach or hacking of the Department’s
electronic data system or any loss or theft of data, other electronic storage, or paper. As used herein,
"Electronic Data System" means all information processing and communications hardware and software
employed in the Department's or the the Contractor's business, whether owned by the Department or the
Contractor, or operated by its employees or agents in performing work for the Department or the Contractor.
F. Personnel Security Policy Overview
The Department and the Contractor shall mitigate the risks posed by users of the Department- or the
Contractor-maintained PII by:
1. Performing appropriate background checks and screening of new Department and Contractor
personnel, in particular those who will have access to the Department-maintained PII;
2. Obtaining agreements from the Department and the Contractor internal users covering confidentiality,
nondisclosure and authorized use of PII, including PII contained in or accessed through the
Department; and
Florida Standards Assessments, ITN 2014-47
135
3. Providing training to support awareness and policy compliance for new hires and annually for all
Department and Contractor personnel.
V. ENFORCEMENT
The Department and the Contractor will consistently enforce this Policy with appropriate discipline for its own
employees. The Department and the Contractor, as applicable, will determine whether violations of this Policy
have occurred and, if so, will determine the disciplinary measures to be taken against any director, officer,
employee, agent or representative who violates this Policy.
The disciplinary measures may include counseling, oral or written reprimands, warnings, probation or
suspension without pay, demotions, reductions in salary, or termination of service or employment, as well as
criminal referral to law enforcement, if appropriate.
Persons subject to disciplinary measures may include, in addition to the violator, others involved in the
wrongdoing such as (a) persons who fail to use reasonable care to detect a violation, (b) persons who withhold
material information regarding a violation, and (c) supervisors who approve or condone the violations or attempt
to retaliate against employees or agents or representatives of the Department or the Contractor for reporting in
good faith violations or violators.
The Department also may take appropriate actions authorized under contract or by law regarding the
Contractor’s failure to comply with the terms of this Policy. It is noted that if the U.S. Department of Education
finds that the Contractor has violated FERPA requirements related to disclosure, the Contractor may be
debarred from access to PII from the Department for at least five (5) years.
Florida Standards Assessments, ITN 2014-47
136
APPENDIX J
REQUIREMENTS FOR SCORING PERFORMANCE ITEMS
This appendix supplements the ITN description of the procedures required for scoring constructed response items on
Florida Standards Assessments in English Language Arts/Literacy (ELA/L) and Mathematics. The ITN requires that
student responses for constructed response items ELA/L tests in grades 3 and 10 and Algebra 1 be scored by at
least two readers (100% second scored) using imaging technology or other technological applications that are
equally responsive to the test administration and reporting schedule. As a quality control measure, 20% of all other
constructed response items for ELA/L and Mathematics tests will require at least two readers’ scores. The
Department desires to implement processes that are reliable and valid as well as efficient in terms of time and
expenditures. Therefore, respondents should, at a minimum, follow the processes described in this appendix.
Enhancements to these processes are acceptable when approved by the Department. For example, respondents
may propose an alternate method or formula for second reads to ensure quality control of scoring constructed
response items.
J.1
Overview of Handscoring of English Language Arts/Literacy and Mathematics Constructed
Response Items
Respondents should submit scoring rubrics to be used in conjunction with other training materials for
scoring constructed-response items for all grades and assessments. Field testing of items occurs in the
spring of each year with items embedded within the operational tests. Near the end of operational scoring,
contractor staff will begin selecting responses from the field test sample that represent a full range of
student work. These responses will be presented to educator committees in mid-spring at field test
rangefinding meetings. From these meetings, complete training and qualifying sets are created to train
scorers who will score a sufficient number of responses to yield reliable statistical data for each constructedresponse item. Every effort must be made to expedite scoring of the field test responses in order to have
the item statistics available to construct and produce the following year’s operational tests. Based on the
number of constructed response items field tested each spring, the Department will consider an additional
fall field test rangefinder meeting for the purpose of preparing training materials for constructed response
items that were not scored in the summer. The statistics for these items will be banked for possible use on
future operational tests. After all rangefinder meetings are completed, a final scoring material review
meeting between the Department and contractor staff will be conducted to finalize training, qualifying, and
initial validity materials for scorer training.
Depending on the measurement model for the ELA/L assessment, the Department will consider a standalone field test for ELA/L extended-response tasks. Respondents who propose a stand-alone field test
should prepare field test sampling specifications detailing the sampling methodology. The Department
prefers that such a field test, if proposed, take place in the December time frame of each year of
administration.
J.2
Conduct Field-Test and Operational Rangefinding Meetings for ELA/L and Mathematics
Prior to the scoring of constructed-response items field tested during the spring administrations, the
contractor will be responsible for organizing and implementing rangefinding meetings to select training
papers for the tasks. The meetings take place in Tallahassee and will involve Florida educators,
Department staff, and the contractor’s scoring directors and assistant directors over a period of up to three
weeks each spring and each fall for field-test rangefinder meetings and each fall for operational rangefinder
meetings. For all educator meetings, the Department and contractor staff will hold concurrent meetings for
subjects and grades whenever possible.
The contractor’s subject area lead scoring directors, subject/grade scoring directors, and assistant scoring
directors will participate in meetings with up to 10 Florida educators and a Department staff member who
will serve as chair.
Florida Standards Assessments, ITN 2014-47
137
The contractor’s staff will maintain a record of all scores and will keep notes during the rangefinder selection
process to document the rationale for assigning scores. Following the meeting, the contractor’s staff will
meet with Department staff to finalize the student responses to be used as rangefinders and to identify
others that may be used as training, qualifying, and validity responses. The contractor’s staff is also
responsible for providing written annotations in a format approved by the Department for the training
materials. Requirements for selecting validity sets are identified in the following section. Contractor staff is
also responsible for developing as part of the scoring guides any additional scoring notes or criteria required
to conduct accurate scoring of the constructed-response tasks. The Department will approve all selections,
annotations, or other materials to be used in training. See Section 7.5.4.5 for associated tasks.
J.3
Conduct Scoring Material Review Meetings
Meetings to review proposed scoring materials for operational tests will be conducted in Tallahassee at least
three months prior to the beginning of handscoring training for the spring test administrations. The
contractor’s subject area lead scoring director and the subject/grade scoring directors will participate in
meetings with Department staff to review scoring criteria and rangefinder papers that were used to score the
tasks during field test scoring, or during the previous operational use.
The purpose of each review meeting is to review the criteria, supplement the initial set of rangefinder papers
with additional papers, if necessary, and to ensure that the contractor’s scoring directors and Department
staff share the same, detailed understanding of the scoring criteria for the operational constructed-response
tasks. The scoring standards established by the initial field-test rangefinder selection will be maintained
during the subsequent review. See Section 7.5.4.5 for associated tasks.
J.4
Produce Scorer Training Materials
Department staff will work closely with the contractor to prepare scoring materials. Frequent communication
between Department and contractor staff can be expected following rangefinder selection and scoring
material review meetings to initiate and complete the development of scoring materials. All scoring material
will be submitted to the Department for review and approval according to an agreed-upon schedule.
Scoring materials must be approved at least four weeks prior to the beginning of training and scoring. The
contractor will be responsible for developing a detailed schedule to be included in the handscoring
specifications identifying steps and schedules in the development of scoring materials. The contractor is
responsible for producing the following scoring materials for each operational and field-test constructedresponse task.
•
•
•
•
•
•
Scoring guides
Training sets
Qualifying sets
Validity sets
Daily calibration sets
Targeted calibration
The contractor will develop a system for cataloging and storing all scoring materials developed during the
course of the project.
Scoring Guides - Scoring guides will be produced for each reader to use during training and scoring. The
scoring guides will contain the rubrics, scoring criteria, constructed-response tasks, passages (if applicable),
glossary of key terms, and other scoring guidelines designated by the Department and the contractor. The
contractor will be responsible for writing annotations for the scoring guides, and the Department will give
final approval to all annotations.
Training Sets – Contractor must develop a sufficient number of training (practice) sets for ELA/L and
Mathematics constructed-response items for scorer training to ensure valid and reliable scores for the
duration of the project. The Department will work with contractor staff to determine the number and types of
Florida Standards Assessments, ITN 2014-47
138
training sets to be used. The contractor will be responsible for writing the annotations for the training papers
for all content areas, and the Department will give final approval to the annotations. The respondent should
propose the process for administering the training sets.
Validity Papers - Initial validity responses are selected from student responses to field-test administrations
and do not appear in any other scoring materials. These responses allow the Department and contractor to
monitor scoring accuracy and to maintain a consistent focus on the established rubrics and scoring
guidelines. A sufficient number of initial validity responses must be approved in time to be loaded into the
electronic scoring system before the scoring session begins. Validity responses are administered to readers
online as part of the electronic scoring system and must be indistinguishable from live responses. The
validity insertion rate will be agreed upon during the negotiation phase; however, for cost purposes,
respondents should assume a validity insertion rate of 1 in 10 for extended ELA/L items and 1 in 25 for all
other constructed response items. Once handscoring begins, scoring directors must refresh the validity pool
during live scoring by escalating proposed validity responses to the Department for approval. All proposed
validity papers for constructed-response items must have two agreement scores assigned. These scores
may be generated by two scorers or by one scorer and a back-reading score. The frequency distribution of
approved validity papers in the pool throughout the project should mirror the operational frequency
distribution to the greatest extent possible. Responses that are approved by the Department will be moved
into the electronic scoring system for random distribution to scorers. To avoid validity responses being
repeatedly seen by the same scorers, adding and deleting validity papers within the electronic system is a
continuous process throughout the project.
The contractor will describe the estimated number of readers to be employed and plans for the
administration of validity responses.
Daily Calibration Sets – In addition to review of selected papers from the Scoring Guide at the start of each
shift, daily calibration sets will be sent via the electronic scoring system at least once a day – usually after
lunch or break. These 2-3 paper sets will be proposed by scoring directors and approved by the
Department. The purpose of daily calibration is to highlight scoring issues or remind scorers of what to look
for at a certain score point. Following the scoring of the calibration sets, the scoring directors will conduct a
short discussion to further explain the scoring standards therein.
Targeted Calibration Sets – Targeted calibration sets consist of 2-3 responses that target an individual’s
scoring trend and may be used as individual training/remediation for scorers whose quality metrics are
below the standard for any reason.
At the completion of scoring, the contractor will securely ship to the Department an electronic file of all
scoring materials prepared for and utilized during scoring.
J.5
Scoring Directors for Handscoring
The contractor will assign its most qualified scoring staff to be scoring directors for Florida’s Standards
Assessments. All scoring directors must have worked in scoring director roles on a regular, continuing
basis. The project and site scoring directors must be highly qualified to serve in these positions. They must
have an appropriate educational background and extensive experience in directing state-level performance
scoring projects as members of the contractor’s regular scoring staff. The contractor will submit resumes of
all handscoring staff to the Department for approval before assigning them to the project.
The contractor will appoint a full-time project director to serve as overall director for Florida Standards
Assessment handscoring activities. The project director must be available on a daily basis to discuss issues
with the site scoring directors and the Department either in person, by e- mail, by phone or fax throughout
the training and scoring sessions.
The contractor must provide at least one full-time lead scoring director for each content area with expertise
in their respective subject. This contractor staff will be available as liaison for Department content specialists
for all handscoring activities.
Florida Standards Assessments, ITN 2014-47
139
The contractor will provide a site scoring director at each site who is dedicated solely to scoring of Florida’s
tests during the scoring period. Site scoring directors will be on site throughout the training and scoring
sessions and will personally assist the scoring directors during the training of team leaders and readers and
throughout the scoring sessions.
The project director must be a senior scoring manager for the contractor. Each site scoring director must
have served as a site scoring director for at least two projects in the past three years involving projects
similar in size and scope to Florida’s.
The scoring directors who will direct the scoring for each subject/grade combination must be highly qualified
to serve in this position. They must have an appropriate educational background and must have had
experience serving as scoring directors at this level for previous scoring projects conducted by the
contractor. The scoring directors must participate in the field test and operational rangefinder and scoring
material review meetings. At least two scoring directors will be required for each grade and subject in order
to be available in the event of the scheduling of two shifts or two scoring rooms. For the ELA/L extended
constructed response items, it may be necessary to assign three scoring directors for this project.
At least one assistant scoring director will be appointed to assist each scoring director in all aspects of
directing scoring. The assistant scoring directors must also have an appropriate educational background
and must have had experience serving as scoring directors or extensive experience serving as team leaders
for previous scoring projects conducted by the respondent. The assistant scoring directors must also
participate in all rangefinder meetings.
Lead scoring directors, scoring directors, and assistant scoring directors must have their résumés approved
by the Department prior to assignment to the project.
As stated above, Florida’s handscoring involves selection of rangefinders for field-test tasks at the same
time that responses to the operational test are being scored. The scoring director and the assistant scoring
director for each subject/grade will work together as a team to ensure that field test rangefinder selection
and ongoing direction of operational scoring are both conducted at the highest levels of quality. It may be
required that contractor staff assigned to the project be present at field test rangefinding meeting while other
contractor staff remains at the scoring site to oversee completion of operational scoring.
For each assessment, the project and site scoring directors will ensure that all additional training required
for scoring is conducted prior to team leader training. Respondents should describe how they intend for this
training to occur.
J.6
Team Scoring Leaders
The contractor will hire one team leader for every 10 to 12 readers. The team leaders may be especially
recruited by the contractor or identified through the reader recruitment process. Team leaders must go
through the same screening process as readers. The Department requires that team leaders have previous
experience as readers and as team leaders if at all possible. At a minimum, team leaders must be
experienced readers.
Extensive training sessions approximately four days in length will be conducted for team leaders before
readers are trained. The scoring directors and assistant scoring directors for each set of constructedresponse tasks will conduct the training. The team leader training sessions will include the successful
completion of a minimum of three training sets and two qualifying rounds. For ELA/L extended constructedresponse writing tasks, Florida currently requires five training sets and three qualifying rounds.
The initial training will include the training and qualifying procedures that will be used for the readers. In
addition, there will be training on managing interaction with scorers, interpreting scoring reports, completing
forms, and following all procedures required by the contractor and the Department.
Florida Standards Assessments, ITN 2014-47
140
J.7
Recruit and Hire Readers
Respondents should include an analysis of the number of people that must be recruited, hired, and
subsequently qualified as readers to complete the scoring within the time required to return reports to
districts by the dates designated in the project schedule. The analysis must include the number and percent
of people who will be expected to qualify, expected reader-attrition rate during each scoring session, and
production required by reader and, if necessary, by shift at each site to complete the work. The respondent
should describe the number of team leaders needed for each subject/grade. This detailed analysis should
be completed in the reply for the Florida Standards Assessments in English Language Arts/Literacy and
Mathematics. Respondents should also indicate expected changes in reader numbers across successive
administrations.
All readers are required to have earned a bachelor's degree in mathematics or the area of English
Language Arts, or some related field. Readers of mathematics constructed-response tasks must have
either been certified to teach the level of mathematics being assessed, have a bachelor’s degree in
mathematics education or mathematics, or must have completed a degree program with a sufficiently strong
emphasis in mathematics to have the mathematics knowledge needed to effectively score responses to
mathematics items.
Potential readers in Florida may include retired Florida teachers or substitute teachers in the state of Florida
provided that they will not teach in Florida’s public schools (or work as a substitute teacher therein) for the
remainder of the current school year. The Respondent is responsible for conveying this information and
obtaining signed verification that potential readers agree to adhere to this stipulation.
The contractor must work with the Department to implement a screening process for hiring readers to satisfy
the Department’s requirements. Potential readers must document that they meet the minimal education
requirements during the screening process. Respondents should describe the screening process to select
scorers.
Potential readers with related degrees are those with degrees that involve course work and experience
related to the content areas being scored. Potential candidates for training and qualifying to score
mathematics shall have degrees in fields such as engineering, computer science, or liberal arts with an
emphasis in mathematics. Likewise, for scoring English Language Arts/Literacy, candidates shall have
degrees in fields such as English, English education, elementary education, reading education, history,
psychology, or journalism. Potential readers will provide information about degree, certification, and course
work taken. Minimum college coursework that represents a sufficiently strong emphasis in mathematics
needed to score in this area is college-level credit in at least three courses to score at the high school level,
two courses for the middle-school level, and one course to score at the elementary level. The contractor will
provide for an independent, third-party verification that the potential reader has earned the identified degree.
It has been the Department’s experience that obtaining third-party verifications is more expensive and timeconsuming for potential readers holding degrees conferred by foreign universities and degrees conferred by
U.S. colleges and universities fifteen or more years ago. The contractor must take this into consideration
when hiring potential readers. Potential readers whose degrees cannot be verified by a third party before
the start of live scoring must be dismissed. No scores must be entered or maintained in the scoring system
from readers with unverified qualifications.
Potential readers also include those with a bachelor’s degree or higher who have successfully completed a
scoring project in the content area and at the elementary, middle, or high-school level being scored,
qualifying and performing at levels required by Florida’s program. Potential readers will ultimately
demonstrate their qualification to score through successful completion of training and meeting the qualifying
standard on scoring qualification rounds, as indicated in the Handscoring Specifications document.
The contractor will document qualifications of readers assigned to be trained in an electronic file (the
“Qualification File”) specifying degrees earned; relevant teaching, educational, or work experience; and
previous scoring experience relevant to scoring Florida’s assessments. The Qualification Files will not
Florida Standards Assessments, ITN 2014-47
141
include personally identifiable information such as name, address, and telephone number. Documentation
will be provided to Department staff, as needed, as it is accumulated throughout the hiring and training
period. After scoring has begun, the contractor will only maintain Qualification Files for readers who
successfully qualify and score student responses to Florida’s assessments.
The Department will monitor adherence to the screening process, review reader files as needed, approve
acceptable related degrees, and provide general guidance on the screening process. Typically, each
scoring site will be monitored by at least one representative from the Department for two to three weeks
beginning with training. Prior to and during this on-site monitoring, the Department will provide oversight
and monitoring of both potential Team Leader and reader qualifications. This monitoring will include a
review for acceptability of potential readers for assignment to the handscoring program,
In the month prior to handscoring training, weekly conference calls between the contractor and the
Department will provide discussions surrounding site readiness, hiring/staffing, system readiness, and other
pertinent topics related to handscoring. After recruitment of potential readers is complete, the Department
staff will review a sample of potential readers’ qualifications. Staff review of qualifications will be conducted
utilizing a random sample of the Qualification Files. Staff will select additional potential scorer IDs for review
as deemed necessary, e.g., for candidates who are having little success in scoring accurately during
training. All qualifications for team leaders and a minimum of one in five candidates’ documentation will be
reviewed (e.g., for an initial hiring of 150 candidates for training, approximately 30 sets of documentation will
be requested). Once scoring training begins, daily conference calls between the contractor and the
Department will be conducted for project management purposes.
As part of the screening process, all potential readers must sign a statement indicating that they understand
the following conditions:
• Applicants who do not successfully complete the training and qualifying requirements, they will not be
hired as readers.
•
Hired readers may be dismissed if, after being trained to score, their scoring performance does not
meet the requirements of the Department or the respondent acting on behalf of the Department.
•
Readers who are absent from scoring for two or more consecutive scoring days must repeat the
training and qualifying procedures described above.
•
Applicants must agree that they will not teach or work as a substitute teacher in Florida’s public
schools for the remainder of the current school year.
Applicants hired for training must also sign an agreement with the Department that they will maintain the
security of Florida’s test materials in addition to security agreements required by the contractor.
J.8
Training and Qualifying of Readers
The contractor will conduct separate training sessions for each constructed-response task or task set.
Training is provided separately for each grade level. The scoring director will conduct training with the
assistance of team leaders, under the direction of site scoring directors. The purpose of the training is to
ensure that each person who scores Florida’s student responses has met the Department's standards for
scoring. The training process is essential for ensuring that scores assigned to performance task responses
provide valid and reliable information. During training, the trainees will be provided with a scoring guide,
highlighters and pens or pencils to make notations on training materials. At the conclusion of training,
qualified readers will be taught how to use the alert system to identify students whose responses indicate
the need for intervention. The contractor is responsible for developing training procedures in consultation
with the Department. The Department will have final approval on all training techniques.
To qualify as a reader, candidates must meet the established standards for the qualification sets approved
by the Department.
Assignment of scorers to items will be done in consultation with Department staff.
Florida Standards Assessments, ITN 2014-47
142
J.9
Handscoring Reports
The contractor will collect reader performance statistics and provide electronic access to daily and
cumulative reader handscoring reports to team leaders and above, including Department staff. The reports
must include both real-time and partial- to full-day reports. Real-time reports must include reader inter-rater
reliability and validity reports. The same reports are required for both operational and field-test scoring. The
Department will determine, in consultation with the contractor, which of these reports will be available to the
Department in real-time. The reports listed below are described generically. The contractor and the
Department will work together initially to determine the format of this information that should be used for
reporting to the Department. An annual review of the format and function of the reports will be conducted
and some modifications may be required prior to the next test administration requiring handscoring based
upon that review. The Department will have final approval on the design and user access level of each
report.
At a minimum, reports containing the following information will be produced and will include both numbers
and percentages, as appropriate.
1. Cumulative Training Report by Reader. This report identifies, by performance task or performance
task sets, by reader, by team, and by site (subject/grade combination), performance on training sets.
2. Cumulative Training Report by Document. This report identifies and summarizes, by performance task
or performance task set, by team, and by site, performance on documents.
3. Cumulative Qualifying Report by Reader. This report identifies, by performance task or performance
task set, by reader, by team, and by site, performance on qualifying sets.
4. Cumulative Qualifying Report by Document. This report summarizes, by performance task or
performance task set, by team, and by site, performance on documents.
5. Cumulative Reader Performance Report. This report includes the reader’s scores on each training set,
qualifying set, and validity set; daily inter-rater reliability statistics; and daily score point distribution.
This report will be updated daily. The report will be accessed on request of the scoring director or the
team leader.
6. Daily and Cumulative Score Point Distribution. This report summarizes, by performance task or
performance task set, by team, by site, and by rubric (0-2, 0-4, and 0-6), daily and cumulative
distributions of score-point assignments.
7. Daily and Cumulative Validity. This report summarizes, by performance task or performance task set,
by team, and by site, daily and cumulative performance on validity sets.
8. Daily and Cumulative Inter-rater Reliability. This report summarizes, by performance task or
performance task set, by site, by team, and by rubric, daily and cumulative levels of inter-rater
agreement statistics.
9. Score Point Distribution for Standard Curriculum Students. This report summarizes, by team and by
site, the cumulative distribution and mean of score-point assignments for standard curriculum students,
for writing performance tasks only.
10. Daily and Cumulative Score Point Distribution and Inter-rater Reliability by Reader. This report
identifies and summarizes, by performance task or performance task set, by site, by team, by rubric, by
reader, daily and cumulative score-point assignment distributions and levels of inter- rater agreement
statistics (including perfect, adjacent, and disparate agreement; high/low scoring summaries; and
required resolutions).
11. Daily and Cumulative Validity by Reader. This report summarizes, by performance task or
performance task set, by site, by team, by rubric, by reader, statistics indicating daily and cumulative
Florida Standards Assessments, ITN 2014-47
143
levels of agreement with validity set scores (including perfect, adjacent, and disparate agreement and
high/low scoring summaries).
12. Daily and Cumulative Validity by Validity Set. This report summarizes, by site, by team, by validity set,
by reader, statistics indicating daily and cumulative levels of agreement with validity set scores
(including a listing of validity set scores with readers assigned scores; perfect, adjacent, and disparate
agreement; and high/low scoring summaries).
13. Daily and Cumulative Read-Behind Reports. This report summarizes, for image scoring applications,
by performance task or performance task set, by team, and by site, the result of online read-behinds by
team leaders or scoring directors.
14. Daily and Cumulative Timing Report. This report summarizes, by site, by team, by performance task
or performance task set, the daily and cumulative number of readers engaged in scoring and statistics
on actual time spent scoring performance tasks or performance task sets (including mean and range).
15. Daily and Cumulative Completion Report. This report, by site and by team, summarizes the daily and
cumulative number of student responses read and the proportion of student responses remaining to be
read. This report also compares on a daily and cumulative basis the amount of readings completed in
comparison to projected completion targets.
By the time of the first handscoring planning meeting with the Department, the contractor will provide an
annotated list and samples of all reports, including reports used internally by the contractor to monitor the
quality and pace of the scoring session. The Department may request that some internal reports be used
for Florida and available to Department staff.
At the completion of operational and field-test scoring, the contractor will provide the Department with final
summaries of all cumulative handscoring reports. The summary reports are to be made available as
electronic files and will be posted to a secure transfer file system maintained by the contractor and shared
with the Department to access, manage, and transfer files as needed throughout the project. The contractor
will produce a technical report that summarizes the score reports and provides details related to the
reliability and validity of the field-test and operational handscoring. These materials will be reviewed at an
annual handscoring debrief.
J.10
Scoring Student Responses
The rubrics used for all constructed-response tasks must be approved by the Department and will be used
in conjunction with approved scoring guides and additional training materials throughout the handscoring of
student responses. Constructed-response items for ELA/L tests in grades 3 and 10 and Algebra 1 will be
scored by at least two readers (100% second scored) using imaging technology or other technological
applications that are equally responsive to the test administration and reporting schedule. As a quality
control measure, 20% of all other constructed response items for ELA/L and Mathematics tests will require
at least two readers’ scores. Responses receiving first and second reads of an individual student’s response
to an item will be randomly distributed across all readers designated to score that item. Assignment of
responses across schools must be randomized to the extent that an individual reader scores responses
from several schools within the same time period and does not score responses from the same school in
succession.
The Department will provide the contractor with a specific set of scoring rules for resolving disagreement
between first and second readings with third and, infrequently, fourth readings and for calculating final
scores. The following should be noted:
•
•
•
Third, independent readings are conducted by team leaders.
Complex resolutions and resolutions involving fourth readings and unscorable decisions are resolved
by scoring directors.
All discrepancies for short-response tasks shall be resolved by a third reader
Florida Standards Assessments, ITN 2014-47
144
Attachment 1
ITN 2014-47 Florida Standards Assessments
RESPONDENT:_________________________________________________
We propose to provide the services being solicited for ITN 2014-47 Florida Standards Assessments. All work shall be performed in accordance with the specifications of this ITN, which has been reviewed and
understood. The below prices are all inclusive. Payment for services will be made based on the deliverables and dates specified. There shall be no additional costs charged for work performed
under this ITN, unless changes are made during the negotiation phase.
Respondents may submit bids on any individual assessment components (A, B, C or D), any group of assessment components, or to provide all assessment components (A, B, C and D). The prices on this
sheet will be used for evaluation purposes. All accommodations and deliverable costs should be included in each component price. In addition, Respondents should complete the pricing per deliverable in
SECTION 7.8, for each assessment component, which will be used for negotiation purposes and any resulting contract.
PRICING
ASSESSMENT COMPONENTS
YEAR 1
YEAR 2
YEAR 3
TOTAL ORIGINAL
CONTRACT TERM
(YEARS 1 - 3)
GRAND TOTAL
RENEWAL YEAR RENEWAL YEAR RENEWAL YEAR
(ORIGINAL TERM + ALL
1
2
3
RENEWALS)*
A. Computer-based and paper-based assessments in English
Language Arts/ Literacy in Grades 3-8 per schedule in
Appendix A.
B. Computer-based and paper-based assessments in
Mathematics in Grades 3-8 per schedule in Appendix A.
C. Computer-based and paper-based assessments in English
Language Arts/Literacy in Grades 9-11 per schedule in
Appendix A.
D. Computer-based and paper-based End-of-Course (EOC)
assessments in Algebra 1, Geometry, and Algebra 2 per
schedule in Appendix A.
E. All assessments (A-D) per schedule in Appendix A.
* Evaluation points will be calculated on these costs.
Vendor Name:
Street Address:
City, State, Zip Code:
Telephone #:
Fax:
Authorized Agent (typed):
Authorized Agent (Signature):
145
ATTACHMENT 2
DRUG-FREE WORKPLACE
(will be considered in case of identical tie replies)
Preference shall be given to businesses with drug-free workplace programs. Whenever two or more bids which are
equal with respect to price, quality, and service are received by the state or by any political subdivision for the
procurement of commodities or contractual services, a bid received from a business that certifies that it has
implemented a drug-free workplace program shall be given preference in the award process. Established
procedures for processing tie bids will be followed if none of the tied vendors have a drug-free workplace program. In
order to have a drug-free workplace program, a business shall:
1) Publish a statement notifying employees that the unlawful manufacture, distribution, dispensing, possession, or
use of a controlled substance is prohibited in the workplace and specifying the actions that will be taken against
employees for violations of such prohibition.
2) Inform employees about the dangers of drug abuse in the workplace, the business's policy of maintaining a drugfree workplace, any available drug counseling, rehabilitation, and employee assistance programs, and the
penalties that may be imposed upon employees for drug abuse violations.
3) Give each employee engaged in providing the commodities or contractual services that are under bid a copy of
the statement specified in subsection (1).
4) In the statement specified in subsection (1), notify the employees that, as a condition of working on the
commodities or contractual services that are under bid, the employee will abide by the terms of the statement
and will notify the employer of any conviction of, or plea of guilty or nolo contendere to, any violation of Chapter
893 or of any controlled substance law of the United States or any state, for a violation occurring in the
workplace no later than five (5) days after such conviction.
5) Impose a sanction on, or require the satisfactory participation in a drug abuse assistance or
rehabilitation program if such is available in the employee's community, by any employee who is so
convicted.
6) Make a good faith effort to continue to maintain a drug-free workplace through implementation of this
section.
As the person authorized to sign the statement, I certify that this firm complies fully with the above
requirements.
Vendor's Signature
Florida Standards Assessments, ITN 2014-47
146
ATTACHMENT 3
DISCLOSURE
STATEMENT
PARTNERSHIP OR INDIVIDUAL
I hereby certify that I, if an individual, or each of us, if a partnership, doing business as of
(Name of Individual or Partnership)
,
(am)(is) not now involved in nor have I ever engaged in any private business venture or enterprise, directly or indirectly, with
the Commissioner of Education, the Deputy Commissioner of Education, any Associate Commissioner of Education,
Division Director, or Bureau Chief within the Florida Department of Education.
I further certify that neither I, nor any partner, if a partnership, nor anyone acting in my or our behalf has requested that any
of the above designated persons or any other employee of the Department of Education exert any influence to secure the
appointment of
under this proposed agreement.
(Name of Individual or Partnership)
(1)
Signature
Signature
(1) If partnership, each partner must sign and execute.
Signature
COMPANY OR CORPORATION
I hereby certify that neither I nor any owner, officer, director, or shareholder of
(Name of State of Inc.)
(Name of Corporation/Company)
,a
(1) corporation, licensed to do business in Florida, is presently involved in or has
been engaged in any private business venture or enterprise, directly, or indirectly, with the Commissioner of Education, the
Deputy Commissioner of Education, any Associate Commissioner of Education, Division Director, or Bureau Chief within the
Florida Department of Education.
I further certify that neither I nor any owner, officer, director, or shareholder of this corporation or anyone acting on behalf of
this corporation or any of its owners, officers, directors, or shareholders has requested that any of the above designated
persons or any other employee of the Department of Education exert any influence to secure the appointment of
under this proposed agreement.
(Company / Corporation)
(2)
Signature
Title
(1) If company is not incorporated, insert "not incorporated" in this space.
(2) If incorporated, this statement is to be executed by same person who will execute Contract, if awarded.
Florida Standards Assessments, ITN 2014-47
147
ATTACHMENT 4
REFERENCES
RESPONDENT NAME: _____________________________________
PROVIDE THE FOLLOWING REFERENCE INFORMATION FOR A MINIMUM OF THREE (3) BUSINESSES WHERE
SERVICES OF SIMILAR SIZE AND SCOPE HAVE BEEN COMPLETED.
BUSINESS NAME:
ADDRESS:
CONTACT PERSON:
PHONE NUMBER:
FAX NUMBER:
E-MAIL ADDRESS:
DATE AND DESCRIPTION
OF SERVICES:
BUSINESS NAME:
ADDRESS:
CONTACT PERSON:
PHONE NUMBER:
FAX NUMBER:
E-MAIL ADDRESS:
DATE AND DESCRIPTION
OF SERVICES:
BUSINESS NAME:
ADDRESS:
CONTACT PERSON:
PHONE NUMBER:
FAX NUMBER:
E-MAIL ADDRESS:
DATE AND DESCRIPTION
OF SERVICES:
Florida Standards Assessments, ITN 2014-47
148
ATTACHMENT 5
MINORITY SUB CONTRACTORS UTILIZATION SUMMARY
STATE OF FLORIDA DEPARTMENT OF EDUCATION
The Department’s Supplier Diversity initiative strives to ensure the promise of Florida’s future is shared by all of its
residents, regardless of race, ethnicity, disability, neighborhood or background. To that end, the Department is
dedicated to support, track and increase its small, minority-, women-, and service-disabled veteran business
enterprise spending with Prime Contractors and subcontractors. This form was developed to assist in these efforts.
The Prime Contractor shall report all small, minority-, women-, and service-disabled veteran business enterprise
Subcontractors, identifying the Name, Address, Type of Certification and Dollar Amount on the form below. The
Prime Contractor shall submit this form with each invoice submitted for payment, whether or not funds have been
spent with a small, minority-, women-, and service-disabled veteran business enterprise subcontractor for the period
covered by the invoice. The Office of Supplier Diversity, Florida Department of Management Services will assist in
furnishing names of qualified minorities. The Office of Supplier Diversity can be reached at (850) 487-0915; the
Internet Web address is http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd .
PRIME CONTRACTOR:
CONTRACT NO.:
CONTRACT TITLE:
MBE CONTRACTORS
Full Name, Address, Telephone Number
State
Certified
NonCertified
NonProfit
Dollar
Amount
Total Amount $
Certified True and Correct by:
Submit Report to:
_________________________________________
Prime Contractor
_________________________________________
Title
_________________________________________
Date
Mrs. Janice Brown
Bureau of Contracts, Grants & Procurement
Management Services
325 West Gaines Street
332 Turlington Building
Tallahassee, Florida 32399-0400
For additional information, you may call Mrs. Brown at (850) 245-0819, or e-mail janice.brown@fldoe.org
Florida Standards Assessments, ITN 2014-47
149
ATTACHMENT 6
VENDOR CERTIFICATION REGARDING
SCRUTINIZED COMPANIES LISTS
Respondent Vendor Name: _____________________________________________________
Vendor FEIN: ___________________
Vendor’s Authorized Representative Name and Title: ___________________________________
Address: _____________________________________________________________________
City: _____________________ State: _____________________________ Zip: ______________
Phone Number: ____________________________________
Email Address:
_
Section 287.135, Florida Statutes, prohibits agencies from contracting with companies, for goods or services over
$1,000,000, that are on either the Scrutinized Companies with Activities in Sudan List or the Scrutinized Companies
with Activities in the Iran Petroleum Energy Sector List. Both lists are created pursuant to section 215.473, Florida
Statutes.
As the person authorized to sign on behalf of Respondent, I hereby certify that the company identified above in the
section entitled “Respondent Vendor Name” is not listed on either the Scrutinized Companies with Activities in Sudan
List or the Scrutinized Companies with Activities in the Iran Petroleum Energy Sector List. I understand that pursuant
to section 287.135, Florida Statutes, the submission of a false certification may subject company to civil penalties,
attorney’s fees, and/or costs.
CERTIFIED BY: __________________________________________________________,
Who is authorized to sign on behalf of the above referenced company.
Authorized Signature Print Name and Title: _________________________________________
Florida Standards Assessments, ITN 2014-47
150
ATTACHMENT 7
CONTRACT STANDARD TERMS AND CONDITIONS
STATE OF FLORIDA DEPARTMENT OF EDUCATION
I.
II.
III.
IV.
Pursuant to S. 287.058(1), Florida Statutes (“F.S.”):
A. Bills for fees or other compensation for services or expenses shall be submitted in detail sufficient for a proper preaudit and
postaudit thereof.
B. Travel expenses will be reimbursed only if expressly authorized by the terms of the Contract. Bills for any travel expenses
shall be submitted in accordance with s. 112.061, F.S.
C. The Department may unilaterally cancel this Contract if the Contractor refuses to allow access by members of the public to all
documents, papers, letters and materials made or received in conjunction with the Contract that are subject to Chapter 119,
F.S., and are not exempt from public inspection by s 119.071, F.S., or by other provisions of general or special law.
D. The Deliverables specified in the Contract must be received and accepted in writing by the Department’s Contract Manager
before Contractor is entitled to payment.
E. To complete this Contract, all services must be performed and/or goods received on or before the date(s) specified in the
Contract.
F. If this Contract is expressly renewable, it may be renewed for a period that may not exceed three years or the term of the
original contract, whichever is longer. The renewal price for the contracted service is set forth in the bid, proposal, reply. Cost
for renewal shall not be changed. Renewals shall be contingent on satisfactory performance evaluations by the Department
and subject to the availability of funds. Exceptional purchase contracts pursuant to s. 287.057(3)(a) and (c), F.S., may not be
renewed.
The Contractor shall prepare an invoice for the amount due and mail it to the Department of Education Comptroller after having
delivered the products and services required under this Contract to the Contract Manager. The invoice shall set forth details
sufficient for a proper pre-audit and post-audit including, where applicable, the products and services delivered and completion
dates. Upon receipt of the invoice, the Department of Education Comptroller will request confirmation from the Contract Manager
that the delivered products and services are satisfactory and payment is due. If for any reason they are not satisfactory, payment
will be withheld until the unsatisfactory condition or conditions are corrected. Upon receipt of the Contract Manager’s approval, the
Department of Education Comptroller shall process each invoice in accordance with the provisions of s. 215.422, F.S.
A. Contractor agrees to submit invoice within thirty (30) days of the Department’s acceptance of deliverables. It is understood
that should Contractor fail to submit invoice within thirty (30) days following the Department’s acceptance of the deliverables,
the Department shall not be responsible for payment thereof under this contract or quantum meruit.
Section 215.422, F.S., provides that agencies have five (5) working days to inspect and approve goods and services, unless bid
specifications or the Contract specifies otherwise. With the exception of payments to health care providers for hospital, medical, or
other health care services, if payment is not available within forty (40) days, measured from the latter of the date the invoice is
received or the goods or services are received, inspected and approved, a separate interest penalty set by the Comptroller pursuant
to s. 55.03, F.S., will be due and payable in addition to the invoice amount. To obtain the applicable interest rate, please contact the
Department’s Fiscal s. at 850/245-0401 or Purchasing Office at 850/245-0735. Payments to health care providers for hospitals,
medical, or other health care services, shall be made not more than thirty-five (35) days from the date of eligibility for payment is
determined, and the daily interest rate is .02740 percent. Invoices returned to a vendor due to preparation errors will result in a
payment delay. Invoice payment requirements do not start until a properly completed invoice is provided to the agency. A Vendor
Ombudsman, whose duties include acting as an advocate for vendors who may be experiencing problems in obtaining timely
payment(s) from a State Agency, may be contacted at 866/352-3776 or by calling the Chief Financial Officer’s Hotline, 800/3422762.
As used in this Contract, the term “Deliverable” refers to tangible “commodities”, as defined in s. 287.012(5), F.S., which the
Contractor provides pursuant to the Contract and to reports or other tangible or documentary evidence which demonstrate that the
Contractor has performed the services required by the Contract. The following provisions govern Deliverables, as applicable:
A. Each Deliverable must be physically delivered to the Department’s Contract Manager, or to a person designated by the
Contact Manager. If delivery is made to a designee, the Contractor shall give written notice to the Contract Manager of the
delivery. A Deliverable is not received until the Contract Manager has physical control of deliveries or has written notice that
the designee has physical control.
B. In each case in which the approval of a Deliverable is dependent upon tests being conducted by the Department or Contractor,
independently or jointly, the Department’s inspection and approval of the Deliverable shall not be subject to the five (5) day
provision in s. 215.422, F.S., but shall be governed by the terms and conditions of the acceptance testing plan as stated in
Attachment A, until approved in accordance with the plan.
C. In each case of a Deliverable of information technology, as defined at s. 287.012(14), F.S., unless specified otherwise in
Attachment A, the acceptance testing plan is deemed to include as a minimum the reliable performance of the information
technology in accordance with its design specifications in:
1. a test environment that simulates the production environment as much as is reasonably possible; and
2. the production environment for which it is intended for a period of time sufficient for the information technology to have
experienced the major foreseeable exigencies of the production functions.
D. The Department’s inspection, including testing when applicable, shall determine whether or not the Deliverables appear to be
in compliance with the Contract. The Contractor shall be notified in writing of any apparent deficiency. The written notice shall
Florida Standards Assessments, ITN 2014-47
151
V.
VI.
VII.
VIII.
IX.
X.
XI.
detail the specific action required by the Contractor to correct the deficiency. The Contractor shall timely correct such
deficiency and resubmit the deliverable for acceptance.
The Contractor represents and agrees that information submitted in support of its requests for payment is the basis of payment and
is true and accurate to the best of knowledge of the responsible signatory. A violation of this provision shall subject the violator to
the provisions of s. 68.082, F.S., pertaining to false claims against the State, and/or s. 837.06, F.S., pertaining to false official
statements.
This paragraph applies if this Contract expires in a fiscal year subsequent to the fiscal year in which the Contract is entered. The
State of Florida’s fiscal year comprises July 1 through June 30. The Department’s and State of Florida’s performance and obligation
to pay under this Contract is contingent upon an annual appropriation by the Legislature. If the Legislature fails to make the
necessary appropriation, the Department will determine if there are other unencumbered funds which are available and which can
be lawfully expended to pay for the Department’s obligations hereunder. If the Department determines that there are no such funds,
the Department shall promptly notify the Contractor. The giving of notice shall be deemed to have cancelled this Contract by mutual
consent, with the date of notice being the date of cancellation.
Notwithstanding anything to the contrary contained in a State Term Contract, Contractor warrants that all commodities, as defined in
s. 287.012, F.S., shall meet the specifications of the Contract and shall be merchantable and fit for the particular purposes intended
by the Contract.
The Contractor further warrants that as to each Deliverable produced pursuant to this Contract, Contractor’s production of the
Deliverable, and the Department’s use of the Deliverable, will not infringe on the copyrights of any third party. This provision applies
to each work of authorship in which copyrights subsist pursuant to 17 U.S.C. Sections 102-105 and to each exclusive right
established in 17 U.S.C. Section 106. In furtherance of this provision the Contractor additionally warrants that:
A. As to each work of software or other “information technology”, as defined in s. 287.012(15), F.S., in which copyrights subsist,
the Contractor has acquired the rights by conveyance or license to any third party software or other information technology,
which was used to produce the Deliverable;
B. As to each image and sound recording incorporated into a Deliverable, the Contractor has acquired the necessary rights,
releases, and waivers from the person whose image or sound is included, or from the holder of the copyrights subsisting in the
literary, musical, dramatic, pantomime, choreographic, pictorial, graphic, sculptural, motion pictures, audiovisual work or sound
recording from which the included image or sound recording was taken.
The Contractor further warrants that the Contractor shall not disclose to any third party, without the express, prior, written approval
of the Department, any personally identifiable information about any student. This applies to information which came from any
record or report of a Florida public education institution or from any education record which is subject to the Family Educational
Rights and Privacy Act, 20 U.S.C. Section 1232g. The terms “record a report” and “student” shall have the meanings prescribed in
s. 1002.22(2)(c) and (d), F.S. The term “educational record” shall have the meaning prescribed in 20 U.S.C. Section 1232g(a)(4).
In the event that the Governor and Cabinet are required to impose a mandatory reserve on appropriations, the Department shall
amend this Contract to place in reserve the amount determined by the Department of Education to be necessary because of the
mandatory reserve. Such amendments may provide for adjustments in the Deliverable products and services as may be necessary.
Intellectual property is subject to following additional provisions:
A. Anything by whatsoever designation it may be known, that is produced by, or developed in connection with, this Contract shall
become the exclusive property of the of the State of Florida and may be copyrighted, patented, or otherwise restricted as
provided by Florida or federal law. Neither the Contractor nor any individual employed under this Contract shall have any
proprietary interest in the product.
B. With respect to each Deliverable that constitutes a work of authorship within the subject matter and scope of U.S. Copyright
Law, 17 U.S.C. Sections 102-105, such work shall be a "work for hire" as defined in 17 U.S.C. Section 101 and all copyrights
subsisting in such work for hire shall be owned exclusively by the Department pursuant to s. 1006.39, F.S., on behalf the State
of Florida.
C. In the event it is determined as a matter of law that any such work is not a "work for hire", Contractor shall immediately assign
to the Department all copyrights subsisting therein for the consideration set forth in the Contract and with no additional
compensation.
D. The foregoing shall not apply to any preexisting software, or other work of authorship used by Contractor, to create a
Deliverable but which exists as a work independently of the Deliverable, unless the preexisting software or work was
developed by Contractor pursuant to a previous Contract with the Department or a purchase by the Department under a State
Term Contract.
E. The Department shall have full and complete ownership of all software developed pursuant to the Contract including without
limitation:
1. The written source code;
2. The source code files;
3. The executable code;
4. The executable code files;
5. The data dictionary;
6. The data flow diagram;
7. The work flow diagram;
8. The entity relationship diagram; and
Florida Standards Assessments, ITN 2014-47
152
XII.
XIII.
XIV.
XV.
XVI.
XVII.
XVIII.
XIX.
XX.
XXI.
XXII.
XXIII.
XXIV.
XXV.
9. All other documentation needed to enable the Department to support, recreate, revise, repair, or otherwise make use of
the software.
The Department reserves the right, at its option, to issue a change order to delete work tasks reducing the total Contract amount by
up to 10%. An addition of work tasks within the scope of the Contract, an increase in the total Contract amount, or a decrease of
more than 10% of the total Contract amount, shall be implemented only by a Contract amendment signed by both the Department
and the Contractor.
Pursuant to s. 216.347, F.S., no funds awarded under this Contract may be used for the purpose of lobbying the Legislature, the
judicial branch, or a State agency.
The Contractor shall grant access to all records pertaining to the Contract to the Department’s Inspector General, General Counsel
and other agency representatives, the State Auditor General, the Office of Program Policy and Government Accountability, and the
Chief Financial Officer.
The Contractor agrees to permit onsite visits by designated Department employees or agents to conduct audits to ensure
compliance with Section 20.055, Florida Statutes. These audits may require Department access to records and data, computers and
communications devices, and other materials whether owned or operated by the Contractor. Access may include, but is not limited
to, user level and/or system level access to any computing or communications device; access to information (electronic, hardcopy,
etc) that may be produced, transmitted or stored on the Contractor's equipment or premises; access to work areas; and access to
interactively monitor and log traffic on the Contractor's networks.
The Contractor must carry general liability insurance, which shall include errors and omissions coverage. The amount of coverage
shall be a minimum of $1,000,000 or the aggregate total of all contractual agreements between the Contractor and the agencies and
political subdivisions of the State of Florida, whichever is greater. The Contractor shall add the Department as an additional insured
on the general liability coverage. The insurance shall cover all of the Contractor’s operations under this Contract and shall be
effective throughout the Term of this Contract, as well as any renewals or extensions thereto. It is not the intent of this Contract to
limit the types of insurance otherwise required by this Contract or that the Contractor may desire to obtain or be required to obtain
by law. The Contractor must submit a Certificate of Insurance indicating coverage for general liability purposes and additional
insured coverage, and shall maintain and pay for same throughout the Term of this Contract. A Certificate of Insurance indicating
adequate coverage shall be submitted to the Department prior to the time the Contract is entered. Any and all insurance policies
shall be through insurers qualified to do business in Florida.
The Contractor agrees to provide the Department upon execution of this Contract with a performance bond or other security
deposited with the Department in the total amount of the Contract or another amount if specified in the procurement specifications or
Attachment A, guaranteeing that the Contractor will perform all work according to this Contract, within the time and price specified in
the Contract. A performance bond shall be issued from a surety company, qualified to do business in Florida.
The Contractor may not assign or subcontract all or any portion of this Contract without the advance written consent of the
Department.
In all cases in which the Contractor, with the advance written consent of the Department, assigns or subcontracts, all or any portion
of the Contract:
A. The Contractor shall monitor the subcontractor or assignee and establish controls to avoid or mitigate risks identified by the
Department or the Contractor; and
B. The Contractor shall allow the Department to monitor subcontractor or assignee activity and compliance, and the Contractor
shall require the subcontractor or assignee to promptly submit to the Department, at the Department’s request, complete and
accurate documentation pertaining to the subcontract or the Contract.
The Contractor shall coordinate with and assist the Department’s Contract Manager in the performance of the latter’s
responsibilities, which include without limitation:
A. Monitoring the activities of the Contractor;
B. Receiving and reviewing the reports of the Contractor to determine whether the objectives of the Contract are being
accomplished;
C. Receiving and reviewing the invoices for payment of funds to assure that the requirements of the Contract have been met and
that payment is appropriate;
D. Evaluating the process used by the Contractor to monitor the activities of any subcontractor or assignee; and
E. Accessing, directly, the subcontractors and assignees, as the Contract Manager deems necessary.
This Contract may not be modified unless in writing signed by the Department and the Contractor.
The Department and the Contractor waive application of the principle of contract construction that ambiguities are to be construed
against a contract’s drafter, and agree that this Contract is their joint product.
The Department and the Contractor acknowledge that they have had their respective attorneys review and approve this Contract or
that they have had the opportunity to do so.
This Contract shall be governed by the laws of the State of Florida, and venue for purposes of any action brought to enforce or
construe the Contract shall lie in Leon County, Florida.
Failure of the Department to declare any default immediately upon the occurrence or knowledge thereof, or delay in taking any
action in connection therewith, does not waive such default. The Department shall have the right to declare any such default at any
time and take such action as might be lawful or authorized under the Contract, at law, or in equity. No Department waiver of any
term, provision, condition or covenant of the Contract shall be deemed to imply or constitute a further Department waiver of any
other term, provision, condition or covenant of the Contract, and no payment by the Department shall be deemed a waiver of any
default under the Contract.
Florida Standards Assessments, ITN 2014-47
153
XXVI.
XXVII.
XXVIII.
XXIX.
XXX.
XXXI.
XXXII.
XXXIII.
XXXIV
Time is of the essence with regard to each and every obligation of the Contractor contained in the Contract. Each such obligation is
deemed material, and a breach of any such obligation (including a breach resulting from the untimely performance thereof) shall
constitute a material breach.
The Contractor shall indemnify and hold harmless the Department, its attorneys, agents and employees, from and against any and
all third party claims, suits, debts, damages, and causes of action, whatsoever, whether arising in law or in equity, arising out of or
relating to Contractor performance or failure to perform under this Contract. The indemnification shall include reasonable attorney
fees and costs incurred by the Department, its attorneys, agents and employees, in the defense of any such claim, suits or causes
of action, as aforesaid.
This Contract may be cancelled by written agreement of the Department and the Contractor specifically referencing this Contract.
Such agreement shall specify the remaining measures necessary to be taken by each party.
The Department reserves the right to cancel this contract without cause by giving the Contractor thirty (30) days written notice.
Should Contractor fail to perform to Contract terms and conditions, Contractor shall be notified in writing, stating the nature of the
failure to perform and providing a time certain (which shall be not less than ten (10) days following receipt of such notice) for
correcting the failure. Such failure to perform shall otherwise be dealt within accordance with Rule 60A-1.006, F.A.C.
A person or affiliate who has been placed on the convicted vendor list following a conviction for a public entity crime may not submit
a bid on a contract to provide any goods or services to a public entity, may not submit a bid on a contract with a public entity for the
construction or repair of a public building or public work, may not submit bids on leases of real property to a public entity, may not be
awarded or perform work as a contractor, supplier, subcontractor, or consultant under a contract with any public entity, and may not
transact business with any public entity in excess of the threshold amount provided in s. 287.017, F.S., for CATEGORY TWO for a
period of thirty-six (36) months from the date of being placed on the convicted vendor list.
The employment of unauthorized aliens by any contractor is considered a violation of Section 274A(e) of the Immigration and
Nationality Act. If the contractor knowingly employs unauthorized aliens, such violation shall be cause for unilateral cancellation of
the contract. In addition, pursuant to State of Florida Executive Order No. 11-116, Contractor shall utilize the U.S. Department of
Homeland Security’s E-Verify system to verify the employment of all new employees hired by the Contractor during the contract
term. Also, Contractor shall include in related subcontracts a requirement that subcontractors performing work or providing services
pursuant to the Contract utilize the E-Verify system to verify employment of all new employees hired by the subcontractor during the
contract term.
Certification Regarding Debarment, Suspension, Ineligibility and Voluntary Exclusion – Lower Tier Covered Transactions
A. The prospective lower tier participant certifies, by submission of this proposal, that neither it nor its principals is presently
debarred, suspended, proposed for debarment, declared ineligible, or voluntarily excluded from participation in this transaction
by any Federal department or agency.
B. Where the prospective lower tier participant is unable to certify to any of the statements in this certification, such prospective
participant shall attach an explanation to this proposal. The Department may cancel this contract if an attached explanation is
not acceptable to the Department or the Federal government.
MyFloridaMarketPlace
A. MyFloridaMarketplace Vendor Registration
Each Vendor doing business with the State of Florida for the sale of commodities or contractual services as defined in section
287.012, Florida Statutes, shall register in MyFloridaMarketPlace, in compliance with Rule 60A-1.030, Florida Administrative
Code, unless exempt under Rule 60A-1.030(3) Florida Administrative Code.
B. MyFloridaMarketplace Transaction Fee
The State of Florida, through the Department of Management Services, has instituted MyFloridaMarketPlace, a statewide
eProcurement system. Pursuant to section 287.057(23), Florida Statutes (2010), all payments for commodities and/or
contractual services as defined in Section 287.012, Florida Statutes, shall be assessed a Transaction Fee of one percent
(1.0%), which the Vendor shall pay to the State, unless exempt under Rule 60A-1.032, Florida Administrative Code.
Notwithstanding the provisions of Rule 60A-1.030, et seq., the assessment of a transaction fee shall be contingent upon
Federal approval of the transaction fee assessment program and continued payment of applicable federal matching funds.
For payments within the State accounting system (FLAIR or its successor), the Transaction Fee shall, when possible, be
automatically deducted from payments to the Vendor. If automatic deduction is not possible, the Vendor shall pay the
Transaction Fee pursuant to Rule 60A-1.031(2), Florida Administrative Code. By submission of these reports and
corresponding payments, Vendor certifies their correctness. All such reports and payments shall be subject to audit by the
State or its designee.
The Vendor shall receive a credit for any Transaction Fee paid by the Vendor for the purchase of any item(s) if such item(s)
are returned to the Vendor through no fault, act, or omission of the Vendor. Notwithstanding the foregoing, a Transaction Fee
is non-refundable when an item is rejected or returned, or declined, due to the Vendor’s failure to perform or comply with
specifications or requirements of the agreement.
Failure to comply with these requirements shall constitute grounds for declaring the Vendor in default and recovering
reprocurement costs from the Vendor in addition to all outstanding fees. VENDORS DELINQUENT IN PAYING
TRANSACTION FEES MAY BE EXCLUDED FROM CONDUCTING FUTURE BUSINESS WITH THE STATE.
Florida Standards Assessments, ITN 2014-47
154
|