Item 02 Accountability and Continuous Improvement CALIFORNIA STATE BOARD OF EDUCATION Nancy Brownell Senior Fellow, Local Control and Accountability State Board of Education Keric Ashley Deputy Superintendent District, School, and Innovation Branch California Department of Education Jenny Singh, Administrator Analysis, Measurement, and Accountability Reporting Division July 13, 2016 1 Brief Context CALIFORNIA STATE BOARD OF EDUCATION • March SBE Meeting – Architecture of Accountability and Continuous Improvement System • May SBE Meeting – Determination of a balance of local and state measures and plans for a single, coherent local, state, federal system • July SBE Meeting – Standards and performance expectations 2 • SBE Approved Design for the LCFF Evaluation Rubrics • A set of state indicators; CALIFORNIA STATE BOARD OF EDUCATION • A methodology for calculating performance as a combination of status (outcome) and change (improvement) for the state indicators in order to differentiate performance at the LEA and school levels, and for student subgroups; • A component that supports the use of local data; and • Concepts for a top-level display. • The SBE also directed staff to prepare a recommendation for establishing standards for the LCFF priorities that are not addressed by the state indicators and options for incorporating college and career readiness, local climate surveys, and an English learner composite measure into the overall LCFF evaluation rubrics design. 3 CALIFORNIA 4=Pupil Achievement STATE BOARD OF 5=Pupil Engagement EDUCATION 6=School Climate 7=Course Access 8=Other pupil outcomes 1=Basic Resources 2=Standards Implementation 3=Parental Involvement 4 Item 02 Attachments 1. Proposed Standards for Graduation Rate, scores on the CASPP, Suspension Rates, Progress of English learners Toward English CALIFORNIA STATE BOARD OF Proficiency, and College and Career EDUCATION Readiness 2. Proposed Standards for the Local Control Funding Formula (LCFF) Priority Areas not Addressed by the Approved State Indicators 3. Additional Components of the LCFF Evaluation Rubrics – Draft Statements of Model Practices 4. Additional Components of the LCFF Evaluation Rubrics – Top-Level Summary Data Display 5 5. Relevant CA Education Codes • CALIFORNIA STATE BOARD OF EDUCATION Recommendations for Action Attachments lay the foundation for action today and final approval of the evaluation rubrics in September • Item recommends methodology for determining standards for each state priority as required by statute (EC 52064.5) • Use of the rubrics supports local planning and determines levels of tiered technical assistance by County Superintendents, State Superintendent, and CA Collaborative for Educational Excellence (CCEE) 6 State Board of Education Meeting Item 2 Attachment 1 Analysis, Measurement, and Accountability Reporting Division July 13, 2016 CALIFORNIA DEPARTMENT OF EDUCATION Tom Torlakson, State Superintendent of Public Instruction Topics TOM TORLAKSON State Superintendent of Public Instruction • Overview of the California Model • Methodology for the California Model • Indicators—Setting the Distributions • The College and Career Standards Model 8 TOM TORLAKSON State Superintendent of Public Instruction Overview of the California Model Overview of the California Model TOM TORLAKSON State Superintendent of Public Instruction • The California Model uses percentiles to create a 5 by 5 grid (giving 25 results) that combine “Status” and “Change” that are equally weighted to make an overall determination for a “Performance Category” (represented by a color) for each indicator. 10 Overview of the California Model (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • Status is based on the current year performance. • Change is the difference between performance from the prior year and current year, or between the current year and a multi-year average—if available. 11 Overview of the California Model (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • The model will be applied to all local educational agencies (LEAs), traditional schools, and student groups with 30 or more students. • The model is aligned with the Local Control and Accountability Plans (LCAPs) and will support local improvement efforts. (Note: Because a separate accountability system is being developed 12 for alternative schools, their data were excluded from the process.) Overview of the California Model (Cont.) State Superintendent of Public Instruction Example: An LEA or school with a “High” Status and an “Increased” in Change will receive an overall performance of Green for most indicators. Change Status TOM TORLAKSON 13 TOM TORLAKSON State Superintendent of Public Instruction Methodology for the California Model 14 Methodology TOM TORLAKSON State Superintendent of Public Instruction • To determine the performance category, the percentile distributions for Status and Change were examined and cut points were selected separately for each indicator (i.e., each indicator has their own unique set of cut points) 15 Methodology (Cont.) TOM TORLAKSON State Superintendent of Public Instruction – For Status, LEAs and/or schools were ordered from highest to lowest and four cut points were selected based on the distribution of all LEAs and/or schools. These cut points created five “Status Levels.” 16 Methodology (Cont.) TOM TORLAKSON State Superintendent of Public Instruction – For Change, LEAs and/or schools were ordered separately from highest to lowest for positive change and lowest to highest for negative change. Cut points were determined separately for positive and negative change. A total of four cuts were selected which created five “Change Levels.” 17 Methodology (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • Cut points will remain in place for a select number of years (e.g., 3 to 5 years), as determined by the State Board of Education (SBE). 18 TOM TORLAKSON State Superintendent of Public Instruction Indicators Based on Local Educational Agency Distributions 19 Graduation Rate Indicator TOM TORLAKSON State Superintendent of Public Instruction • The Graduation Rate Indicator uses the four-year cohort graduation rate. • Cut points were set using the LEA distribution data, which were applied to both LEAs and traditional schools. 20 Graduation Rate Indicator (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • Change was determined by comparing the most recent fouryear cohort graduation rate data (2014–15) to a three-year average generated by using prior four-year cohort graduation rates (2011–12, 2012–13, and 2013–14). 21 Graduation Rate Indicator (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • The Every Student Succeeds Act (ESSA) allows for the use of a fiveyear cohort graduation rate for accountability. Staff will work with the Technical Design Group (TDG) to determine the best methodology for incorporating a five-year cohort rate into the California Model, for inclusion in the State’s Consolidated Plan. 22 College and Career Indicator TOM TORLAKSON State Superintendent of Public Instruction • The College and Career Indicator (CCI) methodology is applied to all students in the four-year graduation cohort. • Cut points were set using the LEA distribution data, which were applied to all LEAs and traditional schools 23 College and Career Indicator (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • Change was determined by comparing the results of the current four-year cohort graduation class (2014–15) to the results of the prior four-year cohort graduation class (2013–14). 24 English Learner Indicator TOM TORLAKSON State Superintendent of Public Instruction • The English Language Indicator (ELI) combines performance on the English language test (i.e., currently the California English Language Proficiency Assessment [CELDT]) with reclassified student data. 25 English Learner Indicator (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • Cut points were set using the LEA distribution, which were applied to all LEAs and traditional schools • Change was determined by comparing the current year CELDT data (2014–15) to the prior year CELDT data (2013–14) • Reclassification data are from the prior school year (2013–14) 26 Long-Term English Learners TOM TORLAKSON State Superintendent of Public Instruction • In 2015, SB 750 was signed into law which redefines the definition of Long-term English learners (LTELs). Therefore, 2015–16 LTEL data will not be comparable to the prior two years of LTEL data (Note: the 2015–16 LTEL data, using the revised definition, will be available in mid-August.) 27 Long-Term English Learners (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • In addition, using LTEL data for school level accountability may not be appropriate. However, LTEL data may be more relevant at the LEA level. • Department staff will convene a work group to explore methodologies for incorporating LTEL data into the ELI as outlined in the June Information Memorandum to the SBE. 28 TOM TORLAKSON State Superintendent of Public Instruction Indicators Based on School Distributions 29 Academic Indicator TOM TORLAKSON State Superintendent of Public Instruction • Only one year of data is currently available (2015 Smarter Balanced Assessments). Therefore, for this indicator only, Performance Categories (i.e., colors) were based on “Status” using the Smarter Balanced Assessment levels of ‘standards met or exceeded’. 30 Academic Indicator (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • Cut points were set separately for: – ELA and mathematics – LEAs and schools • Cut points for schools were set using assessment results for grades 3 through 8 • LEA cut points were set using all assessment results (i.e., grades 3 31 through 8 and grade 11) Academic Indicator (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • The Department expects to receive the 2016 California Assessment of Student Performance and Progress (CAASPP) results in August. Having two years of assessment data will allow staff to pursue options for replacing ‘standard met or exceeded’ as the measure for this indicator. • In addition, the TDG will continue to 32 explore student-level growth models. School Suspension Rate Indicator TOM TORLAKSON State Superintendent of Public Instruction • Cut points were set separately for each school type (elementary, middle, and high). • Change was determined by comparing the current year data (2013–14) to prior year data (2014–15). 33 District Suspension Rate Indicator TOM TORLAKSON State Superintendent of Public Instruction • Staff will work with the TDG to develop the LEA Suspension Rate Indicator for the SBE to consider at the September meeting. 34 TOM TORLAKSON State Superintendent of Public Instruction The College and Career Standards Model 35 College and Career Standards Model TOM TORLAKSON State Superintendent of Public Instruction • Staff recommend that the SBE approve: – The methodology for the College and Career Standards Model – The inclusion of the CCI as a state indicator – Modifying the Academic Indicator (ELA and mathematics) to remove grade 11 assessment scores to avoid “double counting” 36 College and Career Standards Model (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • The Department will conduct two stakeholder webinars (July 20th and 22nd) to obtain input on the different measures and their placement across the four performance levels (i.e., well prepared, prepared, approaching prepared, and not prepared) identified on the chart in Attachment 1. 37 College and Career Standards Model (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • The information from these sessions will be shared with the TDG for consideration in setting the final placement of measures across the four performance levels. • TDG recommendations will be brought to the SBE in September 38 for final approval. College and Career Standards Model (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • The initial CCI simulations combined students who are “well prepared” and students who are “prepared” to determine performance levels. The department will explore other methodologies for consideration in 2017-18. 39 College and Career Standards Model (Cont.) TOM TORLAKSON State Superintendent of Public Instruction • However, LEA and school-level reports will be generated to reflect their performance on all four CCI performance levels. 40 TOM TORLAKSON State Superintendent of Public Instruction Questions 9 ?9 I 41 Item 02-Attachment 2 This attachment summarizes the recommended approach for establishing standards for the following LCFF priorities without state-level data: CALIFORNIA STATE BOARD OF  Priority 1 (Appropriately Assigned Teachers, Access EDUCATION to Curriculum-Aligned Instructional Materials, and Safe, Clean and Functional School Facilities)  Priority 2 (Implementation of State Academic Standards)  Priority 3 (Parent Engagement)  Priority 7 (Access to a Broad Course of Study)  Priority 8 (Outcomes in a Broad Course of Study)  Priority 9 (Coordination of Services for Expelled Students)  Priority 10 (Coordination of Services for Foster Youth) 42 • CALIFORNIA STATE BOARD OF EDUCATION Item 02-Attachment 2for each remaining LCFF The staff proposed standards priority on prior slide are described in the attachment using the following structure: • the proposed statement of the standard; • the evidence for assessing progress relative to the standard; and • criteria that LEAs would use to assess progress toward meeting the standard (i.e., [Met / Not Met / Not Met for Two or More Years]). • This approach emphasizes the importance of these LCFF priorities, also begins to establish a baseline of locally reported information to inform future policymaking. • Staff also propose including a standard for Priority 6 (School Climate) related to the use of local climate surveys to complement the suspension rate 43 state indicator. Item 02-Attachment Priority 2: Implementation 2 of State Academic Standards Standard: LEA completes a self-assessment* included within LCFF evaluation rubrics and reports the results CALIFORNIA STATE BOARD OF in the evaluation rubrics. EDUCATION Evidence: LEAs would determine whether they completed the self-assessment and reported the results, including ratings on each prompt and the overall rating, through the local data selection option in the evaluation rubrics. Criteria: LEAs would assess their performance on a [Met / Not Met / Not Met for Two or More Years] scale. *Note: The self-assessment instrument would address the LEA’s implementation of adopted state academic standards. LEAs would rate their performance (e.g., using a 1- to 5-point scale) on distinct aspects of implementation. Staff will present the proposed self-assessment at the September 2016 SBE meeting, after further consultation with stakeholders.44 Item 02-Attachment 3 LCFF Evaluation Rubrics Design CALIFORNIA STATE BOARD OF EDUCATION     Top-Level Summary Data Display Data Analysis Tool Statements of Model Practices Links to External Resources DRAFT Statements of Model Practices  June 2016 information memorandum (http:// www.cde.ca.gov/be/pn/im/documents/memo-sbejun16item02.doc  Descriptions of research-supported and evidence-based practices organized to correspond to the indicators from the data analysis tool for optional use by LEAs as a complementary component 45 Item 02-Attachment 4 Top-Level Summary Data Display CALIFORNIA STATE BOARD OF EDUCATION • Three statutory (EC 52064.5) purposes for the LCFF evaluation rubrics: – to support LEAs in identifying strengths, weaknesses and areas for improvement; – to assist in determining whether LEAs are eligible for technical assistance; – to assist the Superintendent of Public Instruction in determining whether LEAs are eligible for more intensive state support/intervention. • Primary users of the evaluation rubrics • Importance of ensuring that students, parents, and other stakeholders and the 46 public can access information on LEA- or Proposed Design Features for Top-Level Summary Data Display. g?fwigp LEA/so-mmro (could include basic cleimg?qolic info) - . lnoicainis All Sudeit Perfonmme 531 .5 . Irrproved (Optional for State -- ELA Assessment . . l, 5~ - model practices 4 Smi?cantly Indicators) and resources Math Assessment High 2,3~ CALIFORNIA . . . . . . (indicator applies only STATE BOAR 0 =4 English Learner ProfiCIency Intermediate Maintained to Engish Learners) DUCATI I 5 Graduation Rate (912) Low Inrproved - l~ None 5 (mm Abse(K_8) ?tee'sm Very Low Maintained 1, 4, 8, 9~ Suspension Rate . . A 6 Local Climate Survey Low Maintained 6,9 College Career . 7' 8 Readiness (912) H'gh Sgnificantly None Basics (Teachers, (Summarize Self- 1 Instructional Materials, Met Assessment Facilities) Results) of A 2 Academc ?andards Not Met for One Year A 3 Parent Engagerr?ent Met A Note: The following symbols correspond to the Performance Category noted in parentheses for All Student Performance and within the Equity Report: (Blue); (Green); - (Yellow); A (Orange); (Red). 1 The Equity Report identifies any student subgroup, with valid n-size, that is in the Red or Orange level of performance on the indicator. Users can generate more detailed reports showing performance for all student subgroups. The Equity Report would include the specific student subgroups listed in Education Code 52052: Socioeconomically disadvantaged pupils; English learners; Foster youth; Pupils with disabilities; Homeless youth; and racial/ethnic student subgroups currently reflected in standard reporting (American Indian/Native Alaskan; Asian; Black/African-American; Filipino; Hispanic/Latino; Islander; Two or more races; and White). This mock-up identifies student subgroups by number for illustrative purposes only. 1. CALIFORNIA STATE BOARD OF EDUCATION Recommendations for Action Approve a measure of college and career readiness, as specified in Attachment 1, including: a. Adopting the College and Career Indicator (CCI), which combines Grade 11 test scores on English Language Arts (ELA) and Math and other measures of college and career readiness, as a state indicator (formerly called “key indicators”) b. Using the CCI to establish standards for Priority 7 (Access to Broad Course of Study) and Priority 8 (Outcomes in a Broad Course of Study) based on the approved methodology 48 CALIFORNIA STATE BOARD OF EDUCATION Recommendations for Action c. Modifying the state indicator for student test scores on ELA and Math (Priority 4 – Pupil Achievement), approved at the May 2016 Board meeting, to remove the Grade 11 scores, in order to avoid doublecounting those test scores in two state indicators; and d. Directing staff to prepare a recommendation for the September 2016 Board meeting on the final technical specifications for the CCI. 49 Recommendations for Action 2. Approve a methodology for establishing CALIFORNIA STATE BOARD OF EDUCATION standards, as specified in Attachment 2 for: a. Priority 1 (Appropriately Assigned Teachers, Access to Curriculum-Aligned Instructional Materials, and Safe, Clean and Functional School Facilities), b. Priority 2 (Implementation of State Academic Standards), c. Priority 3 (Parent Engagement), d. County Office of Education (COE) Priority 9 (Coordination of Services for Expelled Students), and e. COE Priority 10 (Coordination of Services for Foster Youth). 50 Recommendations for Action 3. Approve inclusion of a standard CALIFORNIA STATE BOARD OF EDUCATION for the use of local climate surveys to support a broader assessment of performance on Priority 6 (School Climate), as specified in Attachment 2. 4. Approve inclusion of an Equity Report, which identifies instances where any student subgroup is in the two lowest performance categories (currently Red or Orange) on a state indicator, within the top-level summary data 51 display, as specified in CALIFORNIA STATE BOARD OF EDUCATION Questions? 52