2017 PART C FFY 2015 SPP/APR INDICATOR ANALYSIS BOOKLET TABLE OF CONTENTS Indicator 1 – Timely Receipt of Services .................................................................................1 Prepared by the Early Childhood Technical Assistance Center (ECTA) Indicator 2 - Settings ................................................................................................................3 Prepared by the Early Childhood Technical Assistance Center (ECTA) Indicator 3 – Early Childhood Outcomes ................................................................................5 Prepared by the Early Childhood Technical Assistance Center (ECTA) Indicator 4 – Family Outcomes ................................................................................................9 Prepared by the Early Childhood Technical Assistance Center (ECTA) Indicator 5 – Child Find Birth To One ....................................................................................12 Prepared by the Early Childhood Technical Assistance Center (ECTA) Indicator 6 – Child Find Birth To Three .................................................................................14 Prepared by the Early Childhood Technical Assistance Center (ECTA) Indicator 7 – 45-Day Timeline .................................................................................................16 Prepared by the Early Childhood Technical Assistance Center (ECTA) Indicator 8 – Early Childhood Transition...............................................................................18 Prepared by the Early Childhood Technical Assistance Center (ECTA) Indicator 9 and 10 – Dispute Resolution ...............................................................................21 Prepared by the Consortium for Appropriate Dispute Resolution in Special Education (CADRE) Indicator 11 – State Systemic Improvement Plan .................................................................24 Prepared by the Early Childhood Technical Assistance Center (ECTA) in collaboration with the Center for IDEA Early Childhood Data Systems (DaSy), the National Center for Systemic Improvement (NCSI), and the IDEA Data Center (IDC) INDICATOR 1: TIMELY RECEIPT OF SERVICES Completed by ECTA Indicator 1: Percent of infants and toddlers with IFSPs who receive the early intervention services on their IFSPs in a timely manner. INTRODUCTION Indicator 1, Timely Receipt of Services, is a compliance indicator with a target of 100%. Each state defines what constitutes timely services. The indicator refers to the percentage of children for whom all services are timely, not the percentage of services that are timely; if one or more of the services for a child are not delivered within the defined timeline, then the child would not be counted in the percentage of children receiving timely services. The analysis of Part C Indicator 1 is based on data from FFY 2015 Annual Performance Reports (APRs) for 56 states. For the purpose of this report, the term “state” is used for both states and jurisdictions. DATA SOURCES In responding to this indicator, states could use data from monitoring or the state data system. METHODOLOGY & MEASUREMENT APPROACHES Defining Timely Services States are required to provide the criteria used to determine which infants and toddlers received IFSP services in a timely manner. The data are based on the actual number of days between parental consent or the date specified on the IFSP for the initiation of services and the provision of services. The number of days states use to define timely services varies across states. States are also allowed to count delays due to family circumstances as timely, although not all states collect and report delays attributable to family circumstances. PERFORMANCE TRENDS Figure 1 illustrates current data for Indicator 1 and trend data over the last six reporting years, from FFY 2010 to FFY 2015. For each reporting year, the number of states represented within each ten-percentage point range is shown in the chart, and the table below the chart shows the national mean, range, and number of states included. 1 Figure 1 Percent Receiving Timely Services Trends - Six Years of Indicator C1 Data Percent Receiving Timely Services 100 MIStates 46 IStates 45: States 48: States 48: States 47: States 90 8 States 7 States 9 States 7 States 4 States 5 States 80 2 States 2 States 1 State - 2 States 2 States 70 i - - - 1 --State 1 State 1 -_State 1 State 1 State 60 1 - - State - - - - State 30 - - 20 10 0 FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 94 94 95 96 95 94 Highest 100 100 100 100 100 100 Lowest Data 1 0 0 0 1 0 INDICATOR 2: SETTINGS Completed by ECTA Indicator 2: Percent of infants and toddlers with IFSPs who primarily receive early intervention services in community-based or home settings. INTRODUCTION Indicator 2 documents the extent to which early intervention services are provided in natural environments. “Natural environments” are settings that are either home-based or community-based. Settings that are not considered natural environments include hospitals, residential schools, and separate programs for children with delays or developmental disabilities. This summary of Indicator 2 is based on data from FFY 2015 APRs for 56 states. For the purposes of this report, the term “state” is used for both states and territories. DATA SOURCES The data for this indicator are from the 618 IDEA Part C Child Count and Settings data collection. In this data collection, states report the primary setting of the child’s services for all children enrolled in Part C on a state-designated date, between October 1 and December 1, 2015. “Primary setting” is the service setting in which the child receives the largest number of hours of Part C early intervention services. Determination of primary setting should be based on the information included on the IFSP in place on the Child Count date. PERFORMANCE TRENDS Figure 1 illustrates current data for settings and trend data over the last six reporting years, from FFY 2010 to FFY 2015. For each reporting year, the number of states represented within each ten-percentage point range is shown in the chart, and the table below the chart shows the national mean, range, and number of states included. 3 Figure 1 Percent of infants and toddlers receiving services in home or community settings Trends - Six Years of Indicator C2 Data Percent of infants and toddlers receiving services in home or community settings 100 45 49 States States 51 States Stat 47 States 51 54 States 90 States 10 States States States States State 80 State State State State State FFY 2010 FFY 2012 95 100 33 0 FFY 2013 97 100 74 0 FFY 2014 97 100 74 1 FFY 2015 97 100 76 0 Mean Highest Lowest Data INDICATOR 3: INFANT & TODDLER OUTCOMES Prepared by ECTA Indicator 3: Percent of infants and toddlers with IFSPs who demonstrate improved: A. Positive social-emotional skills (including social relationships); B. Acquisition and use of knowledge and skills (including early language/communication); and C. Use of appropriate behaviors to meet their needs. INTRODUCTION Indicator 3 reports the percentage of infants and toddlers with IFSPs who demonstrate improved outcomes during their time in Part C. This summary is based on information reported by 56 states and jurisdictions in their FFY 2015 Annual Performance Reports (APRs). For the purposes of this report, the term ‘state’ is used for both states and jurisdictions. For this indicator, states report data on two summary statements for each of the three outcome areas. The summary statements are calculated based on the number of children in each of five progress categories. The child outcomes summary statements are:   Summary Statement 1: Of those children who entered the program below age expectations in each outcome, the percent who substantially increased their rate of growth by the time they turned three years of age or exited the program (progress categories c+d/a+b+c+d). Summary Statement 2: The percent of children who were functioning within age expectations in each outcome by the time they turned three years of age or exited the program (progress categories d+e/a+b+c+d+e). DATA SOURCES & MEASUREMENT APPROACHES States and jurisdictions continue to use a variety of approaches for measuring child outcomes, as shown in Table 1. Table 1 Child Outcomes Measurement Approaches (N=56) Type of Approach Number of States (%) Child Outcomes Summary (COS) process 42 (75%) One statewide tool 8 (14%) Publishers’ online analysis 3 (5%) Other approaches 3 (5%) PERFORMANCE TRENDS Figures 1 through 6 illustrate trends for the two summary statements for each of the three outcome areas over the last six reporting years (FFY 2010 to FFY 2015). For each reporting year, the number of states within each ten-percentage point range are shown 5 in the charts, and the tables below each chart show the national mean, range, and number of states included each year. Figure 1 Figure 2 6 Figure 3 Trends - Six Years of Indicator CS Data (Outcome B, Summary Statement 1) 100 4: 3 States 3 States 4 States 4 States 2 States 3 States 90 11 States 11 States 10 States 13 States 10 States 10 States Inc- 80 17 States 17 States 19 States 18 States 23 States 18 States States 10 States 12 States 7 States 11 States 15 States 60 7 States 9 States 6 States 9 States 8 States 7' States 2 50 5 States 3 States 2 States 4 States 2 States 2 States 3 40 -- 1 State 3 States 3 States 1 State - 1 State FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 70 70 70 71 71 71 Highest 100 100 100 100 94 100 Lowest Data 1 0 0 0 0 0 Figure 4 Trends - Six Years of Indicator CS Data (Outcome B, Summary Statement State 90 -- 1 State 1 -- State 1 State 1 State 1 State - .9 -- 80 2 States 1 State 2 States 2 States 1 State 1 State a. 70 12 States 10 States 8 States 7 States 8 States 6 States 3 60 .E f, 18 States 19 States 20 States 21 States 20 States 24 States 3 50 an .5 12 States 16 States 15 States 15 States 18 States 12 States '2 40 5 7 States 6 States 6 States 9 States 4 States 8 States 30 5 1 State 1 State 2 States - 3 States 3 States ?5 20 5 2 States 1 State 1 State - - 1 State 10 - 1 State 1 State 1 State 1 State - 0 FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 51 51 50 51 50 50 Highest 81 85 87 88 80 94 Lowest Data 1 0 0 0 0 0 Figure 5 Trends - Six Years of Indicator CS Data (Outcome C, Summary Statement 1) 100 4 States 5 States 4 States 5 States 5 -- States 4 States 90 .C 10 States 10 States 11 States 13 States 11 States 10 States 5 80 "5 21 States 21 States 19 States 15 States 18 0 States 23 States 33 70 E. 9 States 11 States 13 States 13 States 14 States 10 States 60 4.. 8 States 5 States 5 States 7 States 5 States 5 States 50 1 State 1 State 1 State - 1 State 2 States 40 .2 1 State 1 State 1 State 2 States 1 State 2 States +2 30 a -- 1 State 2 States 2 States 1 State 1 State - -- -- FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 71 72 71 72 73 72 Highest 94 100 100 100 94 94 Lowest Data 1 0 0 0 0 0 Figure 6 Trends - Six Years of Indicator CS Data (Outcome C, Summary Statement 2) 100 2 States 1 State - - - 1 State -- 90 -- -- 5 1 State 3 States 4 States 3 States 4 States 1 State '45: 80 8 11 States 7 States 3 States 7 States 9 States 9 States 3 70 E: 12 States 17 States 20 States 18 States 13 States 14 States 60 14 States 16 States 15 States 17 States 20 States 18 States N. 50 .E 3g 9 States 8 States 6 States 6 States 6 States 10 States 5 40 i 4 States 2 States 5 States 3 States 2 States 2 States 5 30 -- 1 State 1 State 2 States 1 State 1 State 1 State 5 20 -- -- 1 State 1 State 1 State 1 -- State 1 State - 10 0 FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 59 59 57 58 59 59 Highest 100 92 87 86 87 94 Lowest Data 1 0 0 0 0 0 INDICATOR 4: FAMILY OUTCOMES Completed by ECTA Indicator 4: The percent of families participating in Part C who report that early intervention services have helped the family: (A) Know their rights; (B) Effectively communicate their children's needs and; (C) Help their children develop and learn. INTRODUCTION Indicator C4 is a performance indicator, with states allowed to set performance targets each year. The data used for this report reflect information reported by 56 states and jurisdictions in their FFY 2015 APRs. States and jurisdictions are referred to as “states” for the remainder of this summary. DATA SOURCES AND MEASUREMENT APPROACHES All states use survey methodology to report on this indicator. In cases where a state did not report which survey was used and did not report a change to survey type for FFY 2015, the survey type reported in previous years was used. States reported using four main survey approaches to collect data for this indicator. Of the 56 states, 18 used the NCSEAM Family Survey (32%), 17 states (30%) used the revised ECO Family Outcomes Survey (2011), 12 (21%) used a state-developed survey, and nine used the original (2006) ECO Family Outcomes Survey (16%). Some states tailored their surveys by removing questions not required for APR reporting, adding survey questions specific to their state, and/or making wording and formatting changes. Scoring metrics and indicator thresholds varied among states as well. Forty-three states (77% of states) reported a response rate. Among these 43 states, the average was 37%, with response rates ranging from 9.2% to 100%. PERFORMANCE TRENDS Figures 1, 2, and 3 show the current data for each of the three family outcome subindicators and trend data over the last six reporting years (FFY 2010 to FFY 2015). For each reporting year, the number of states represented within each ten-percentage point range is shown in each chart, and the tables below the charts show the national means, ranges, and number of states included. 9 Figure 1 Trends - Six Years at Indicator 4A Data Early intervention services have helped the family know their rig 1(1) 28 States 2? States 30 States 31 States 31 I States 31 I States 90 13 0 States 16 0 States 12 0 States 13 States 17 States 17 States 30 9 States 9 States 9 States 9 States 7 States 6 States F0 4 States 4 States 4 States 3 -- States 1 State 2 -- States a 2 States - 1 State - - - FFY 2010 FFY 2011 2012 FFY 2013 FFY 2014 FFY 2015 Mean 86 8? 87 38 90 89 Highest 10CI 100 100 100 Ill! 100 Lowest Data 0 0 0 (l 0 0 Figure 2 Trends - Six Years at Indicator 4B Data Ea 1y Intervention helped the family effectively communicate their children's needs 1(1) 32 States 30 States 33 States 33 States 35 I States 35 I States 90 0 0 9 States 14 States 11 States 14 States 13 States 13 States 30 10 States 8 States 7 States 5 States 5 States 5 States 1, 4 States 3 States 4 States 3 States 3 -. States 3 States a- 60 -- I- 1-. State 1 State 1 State 1 State - - FFY 2010 FFY 2011 2012-13 FFY 2013 FFY 2014 FFY 2015 Mean 87 88 88 39 90 90 Highest 10CI 100 99 100 Ill! 100 Lowest Data Figure 3 Average Percent Reported Trends - Six Years of Indicator 4C Data Early lnterventlon has helped the famlty help their children develop and learn 100 35 States 3? States 35 States 37 States 37 States 38 I States 90 14 States 15 States 15 States 14 States 16 States 16 States 80 6 States 4 States 5 States 5 J. States 3 -- States 2 States F0 1 State so -- - - 1 State - - - FFY 2010 FFY 2011 2012 FFY 2013 FFY 2014 FFY 2015 Mean 90 91 90 91 92 92 Highest 100 1CD 100 100 1CD 100 Lowest Data INDICATOR 5: CHILD FIND (BIRTH TO ONE) Prepared by ECTA Indicator 5: Percent of infants and toddlers birth to one with IFSPs compared to national data. INTRODUCTION Indicator 5 reports state performance in the identification of eligible infants during their first year of life. The summary of the analysis of Indicator 5 is based on data from FFY 2015 APRs from 56 states. For the purposes of this report, the term “state” is used for both states and jurisdictions. DATA SOURCES States must use data collected and reported under Section 618 (Annual Report of Children Served) on the number of infants, birth to age one, who were identified and served on a state-determined date (generally December 1), and to calculate the percentage of the state’s birth to one population that number represents. For Indicator 5, OSEP provided states with pre-populated data from the Section 618 Child Count data, and comparison data from the U.S. Bureau of the Census “2015 State Population Estimates by Age, Sex, Race, and Hispanic Origin" (http://www.census.gov/popest). These two numbers are used to calculate the percentage of children served. For jurisdictions for which US Census data were not available, states may submit population data from an alternate source for the purpose of calculating the percentage served. The Part C Early Intervention Numbers and Percentages: Indicator C5 & C6 data file (https://osep.grads360.org/#communities/pdc/documents/9795) reports the national percentage of infants and toddlers ages birth to age one receiving early intervention services under IDEA, using data from 50 states and the District of Columbia. For Part C, the 2015 national percentage is 1.20%. This is the number to which all states must compare their data. Although jurisdictions and territories are not included in the 618 Child Count mean, they also compare their data to this mean. PERFORMANCE TRENDS Figure 1 illustrates current data for child find and trend data over the last six reporting years. For each reporting year, the number of states represented within each onepercentage point range is shown in the chart; the table below the chart shows the mean, range, and number of states included. The mean shown in the chart below for FFY 2015 (1.37%) includes all 56 states, counting each state equally regardless of population. This average (1.37%) therefore differs slightly from the national mean calculated from the 618 Child Count data for FFY 2015 (1.20%) explained above. 12 Figure 1 Percent of infants and toddlers birth to one with IFSPs Trends - Six Years of Indicator C5 Data Percent of infants and toddlers birth to one with IFSPs 10.00 9.00 8.00 7.00 6.00 5.00 - 1-- State 1 --State 4.00 - 1 State 1 State 2 States 2 States 1 State 3.00 5 TStates 4 States 3 States 4 States 4 States 5 States 2.00 25 States 26 States 26 States 26 States 26 States 27 States J. 26 States 25 States 26 States 24 States 23 States 22 States 0.00 FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 1.15 1.16 1.21 1.26 1.28 1.37 Highest 2.73 3.00 3.20 3.97 4.60 4.57 Lowest 0.50 0.27 0.40 0.44 0.36 0.56 No Data INDICATOR 6: CHILD FIND (BIRTH TO THREE) Prepared by ECTA Indicator 6: Percent of infants and toddlers birth to three with IFSPs compared to national data. INTRODUCTION Indicator 6 reports state performance in the identification of eligible infants and toddlers from birth to age three. This summary of the analysis of Indicator 6 is based on APR data for FFY 2015 from 56 states. For the purposes of this report, the term “state” is used for both states and jurisdictions. DATA SOURCES States must use data collected and reported under Section 618 (Annual Report of Children Served) on the number of infants and toddlers, birth to age three, who were identified and served on a state-determined date (generally December 1), and to calculate the percentage of the state’s birth to one population that number represents. For Indicator 6, OSEP provided states with pre-populated data from the Section 618 Child Count data, and comparison data from the U.S. Bureau of the Census “2015 State Population Estimates by Age, Sex, Race, and Hispanic Origin" (http://www.census.gov/popest). These two numbers are used to calculate the percentage of children served. For jurisdictions for which US Census data were not available, states may submit population data from an alternate source for the purpose of calculating the percentage served. The Part C Early Intervention Numbers and Percentages: Indicator C5 & C6 data file (https://osep.grads360.org/#communities/pdc/documents/9795) reports the national percentage of infants and toddlers ages birth through 2 receiving early intervention services under IDEA, Part C is 3.00% (based on 50 states and the District of Columbia). This is the number to which all states, including jurisdictions and territories, must compare their data. PERFORMANCE TRENDS Figure 1 illustrates current data for child find and trend data over the last six reporting years. For each reporting year, the number of states represented within each onepercentage point range is shown in the chart; the table below the chart shows the mean, range, and number of states included. The mean shown in the charts below for FFY 2015 (3.15%) includes all 56 states (i.e. including jurisdictions and territories), and therefore differs slightly from the Part C Child Count and Settings Table C1 national mean for FFY 2015 (3.00%). 14 Figure 1 Percent of infants and toddlers birth to three with IFSPs Trends - Six Years of Indicator C6 Data Percent of infants and toddlers birth to three with IFSPs 10.State 9.00 - - - 1 State - 8.00 - - 1 State 1 State - - 7.00 1 State 1 --State 1 State 2 States 2 States 2 States 6.00 2 States 3 States 2 States - 3 States 3 States 5.00 5 States 5 States 5 States 6 States 5 States 5 States 4.00 13 States 13 States 11 States 12 States 8 States 10 States 3.00 23 TStates 23 States 22 States 24T States 29 States 27 States 2.00 12 LStates 11 iStates 14? States 11L States 6 States 8 States 1.00 - - - - 2 States 0.00 FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 2.91 2.92 2.94 2.97 3.08 3.15 Highest 6.96 6.70 7.18 7.94 8.89 9.05 Lowest 1.50 1.20 1.64 1.19 0.93 1.06 No Data INDICATOR 7: 45-DAY TIMELINE Prepared by ECTA Indicator 7: Percentage of eligible infants and toddlers with IFSPs for whom an initial evaluation and initial assessment and an initial IFSP meeting were conducted within Part C’s 45-day timeline. INTRODUCTION Indicator 7 is a compliance indicator with a performance target of 100%. Part C regulations specify that the initial evaluation and initial assessments of the child and family, as well as the initial IFSP meeting must be completed within 45 days from the date the lead agency or provider receives the referral. For this indicator, states have the option to identify and count as timely those delays that are the result of exceptional family circumstances. This summary is based on data from Annual Performance Reports (APRs) submitted by 56 states and jurisdictions for FFY 2015. For the remainder of the summary, the term “state” is used to refer to both states and jurisdictions. DATA SOURCES Data for reporting on this indicator may be gathered from a state’s data system and/or local monitoring practices, including sampling files for review, onsite verification visits, or reviews of self-assessment results. ACTUAL PERFORMANCE Figure 1 illustrates current data for Indicator 7 and trend data over the last six reporting years (FFY 2010 through FFY 2015). For each reporting year, the number of states represented within each ten-percentage point range is shown in the chart, and the table below the chart shows the national mean, range, and number of states included. 16 Figure 1 Percent meeting 45?day timeline Trends - Six Years of Indicator C7 Data Percent meeting 45-day timeline 100 States 52 States 51 50 States 50 States 50 States 90 States States States States States States 80 State State States State 70 State FFY 2010 95 100 13 1 FFY 2011 96 100 22 0 FFY 2012 97 100 83 0 FFY 2013 97 100 76 0 FFY 2014 96 100 65 0 FFY 2015 96 100 72 0 ea Highest Lowest Data 17 INDICATOR 8: EARLY CHILDHOOD TRANSITION Completed by ECTA Indicator 8: The percentage of toddlers with disabilities exiting Part C with timely transition planning for whom the Lead Agency has: A. Developed an IFSP with transition steps and services at least 90 days, and at the discretion of all parties, not more than nine months, prior to the toddlers third birthday; B. Notified (consistent with any opt-out policy adopted by the State) the State education agency (SEA) and the lead education agency (LEA) where the toddler resides at least 90 days prior to the toddler’s third birthday for toddlers potentially eligible for Part B preschool services; and C. Conducted the transition conference held with the approval of the family at least 90 days, and at the discretion of all parties, not more than nine months, prior to the toddler’s third birthday for toddlers potentially eligible for Part B preschool services. INTRODUCTION Indicator 8 is a compliance indicator with a performance target of 100%. Each of the three sub-indicators of Indicator 8 corresponds to specific Part C regulations. This analysis of Part C Indicator 8 is based on data from FFY 2015 Annual Performance Reports (APRs) for 56 states and jurisdictions. For the purposes of this report, the term “state” is used for both states and jurisdictions. One state had data determined by OSEP to be not valid and reliable for Indicator 8B. DATA SOURCES/ MEASUREMENT APPROACHES States use a variety of data sources for reporting on this indicator, including monitoring data (e.g. file review and self-assessment), the state’s data system, or combinations of these approaches. There is also variability among states regarding use of census vs. sampling methodologies for reporting on this indicator. A census approach is defined as reporting on all children for the entire reporting period or all children in a specific time frame (e.g. all children transitioning in one quarter of the calendar year). A majority of states use census data for all three sub-indicators. PERFORMANCE TRENDS Figures 1, 2, and 3 illustrate the current data for each of the three transition subindicators and trend data over the last six reporting years. For each reporting year, the number of states represented within each ten-percentage point range is shown in each chart. The tables below the charts show the national mean, range, and number of states included. 18 Figure 1 Average Percent Reported Trends - Six Years of Indicator 8A Data Percent of Exiting Children with Transition Steps and Services on the IFSP 1CD S1 0 States 52 0 States so 0 States 430 States SST States 51LT States 90 3 States 1 State 2 States 5 States 2 States 4 States 80 - 2 States 2 States 1 State 1-- State 70 1 State 1 State 1 State - - 60 - 1 State 1 --State - - 50 - - 1 State - . 40 30 20 10 0 FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mea 97 97 95 96 97 97 Highest 100 100 100 100 100 100 Lowest Data 1 1 0 0 0 19 Figure 2 Trends - Six Years of Indicator BB Data Percent of Exiting Children with Notification to the LEA 100 - 52 States 52 TStates dsI States 444) States AQI States States 90 3 States 1 State 4 States 7 States 4 States 5 States 80 - 1 State - 1 State 1 State 4% 70 -- a - 1 --State - 2 States - 1 State State - - 30 . . - 1 State - 20 10 3 States 1 State 1 State - 0 FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 98 98 91 95 94 97 Highest 100 100 100 100 100 100 Lowest 81 64 0 0 69 No Data Trends - Six Years of Indicator 8C Data Percent of Exiting Children who Received 3 Transition Conference 100 45 0 States 45 0 States 49 0 States 450 States AQT States 48: States 90 8 States 6 States 4 States 8 States States 6 States 80 .- 1 State 4 thates 2 States 3 States - 2 States 70 1-- State . - . State - - - - - FFY 2010 FFY 2011 FFY 2012 FFY 2013 FFY 2014 FFY 2015 Mean 94 95 95 95 96 95 Highest 100 100 100 100 100 100 Lowest Data 1 1 0 0 0 0 INDICATORS C9 & C10: DISPUTE RESOLUTION Prepared by the Center for Appropriate Dispute Resolution in Special Education (CADRE) INDICATOR 9: RESOLUTION SESSIONS Indicator 9: Percent of hearing requests that went to resolution sessions that were resolved through resolution session settlement agreements (applicable if Part B due process procedures are adopted). INDICATOR 10: MEDIATION Indicator 10: Percent of mediations held that resulted in mediation agreements. INTRODUCTION The IDEA requires States receiving grants under Part C to make available four dispute resolution processes, and to report annually to the U.S. Department of Education Office of Special Education Programs (OSEP) on their performance.1 The processes include signed written complaints, mediation, due process complaints, and, in states where Part B due process complaint procedures have been adopted, resolution meetings. The following is a report and brief summary of States’ Federal Fiscal Year (FFY) 2015 Annual Performance Reports (APRs) for Indicators C9 (Resolution Meetings Resulting in Written Settlement Agreements) and C10 (Mediations Resulting in Written Agreements).2,3 DATA SOURCES AND METHODOLOGY Data sources for this report include FFY 2015 APRs and Section 618 data, currently available through the GRADS360 OSEP portal. These analyses are specific to State performance on Indicators C9 and C10, and do not present a complete picture of dispute resolution activity. SUMMARY BY INDICATOR Indicator C9: Resolution Meetings Resulting in Written Settlement Agreements Indicator C9 documents the percentage of resolution meetings that result in written 1 For the purposes of this report, the terms “States” is used to refer to all 56 Part C grant recipients (i.e., the Fifty States, the District of Columbia, Puerto Rico, the Virgin Islands, American Samoa, Guam, and the Northern Mariana Islands). 2 The reporting period (July 1, 2015-June 30, 2016) began during FFY 2015. 3 These indicators were reported as C12 and C13 in previous years’ APRs. 21 settlement agreements. This indicator applies only to States that have adopted Part B due process complaint procedures. States are required to report any activity relating to performance Indicator C9 but are not required to set or meet a performance target if fewer than 10 resolution meetings are held in a single year. Because due process complaints continue to be a seldom used dispute resolution option in Part C programs, resolution meetings rarely occur. Historically, in only one reporting year (2012-13) has national data reflected more than two resolution meetings held. Twenty states reported that they use Part B due process procedures according to their 2015 APR. Nationally, only one resolution meeting was held leading to a successful written settlement agreement. Indicator C10: Mediations Resulting in Written Agreements Indicator C10 is a performance indicator that documents the percentage of mediations resulting in written mediation agreements. As with Indicator C9, states are required to report any activity relating to Indicator C10, though they are not required to set or meet a performance target if fewer than ten mediations are held in a single year. The bands in Figure 1 reflect state performance on Indicator C10 over a six year period. The blue diamonds on each performance band in Figure 1 indicate the mean, or average, rate of agreement across states for that year.4 Figure 1 4 For this “average of state agreement rates”, all states contribute equally to the calculation regardless of the level of activity. 22 Note: “No data” indicates the number of states reporting no activity or lacking valid/reliable data. Nationally in 2015-16, eight states held a total of 57 mediation sessions which is a reduction of 34.5% as compared to 2014-15 national data. Two large population states accounted for 50 (one state held 41 and one state held 9) of the 57 mediation sessions held, or 87% of all mediation sessions held nationally in 2015-16. Fifty of the fifty-seven mediation sessions held in 2015-16 resulted in agreements – yielding a national agreement rate of 88%. This contrasts with the average state reported mediation agreement rate of 85% because in this calculation all states are counted equally regardless of level of activity. CONCLUSION Nationally, the use of mediation sessions and resolution meetings among Part C programs continues to be very low. This may be the result of the collaborative, familycentered nature of Part C programs and the short time of service as transition to Part B programs occurs before the child’s third birthday. It is recommended that Lead Agencies continue to educate parents about their rights, and the full continuum of dispute resolution options available to them should conflict occur. 23 INDICATOR 11: STATE SYSTEMIC IMPROVEMENT PLAN, PHASE III This report was prepared by the Early Childhood Technical Assistance Center (ECTA) in collaboration with the Center for IDEA Early Childhood Data Systems (DaSy), the National Center for Systemic Improvement (NCSI), and the IDEA Data Center (IDC). Indicator 11: The State’s SPP/APR includes a State Systemic Improvement Plan (SSIP) that meets the requirements set forth for this indicator. INTRODUCTION In Indicator 11 of the State Performance Plan and Annual Performance Report (APR), the Office of Special Education Programs (OSEP) requires states to develop and implement a three-phase State Systemic Improvement Plan (SSIP). The SSIP is a comprehensive, ambitious yet achievable, multiyear plan for improving results for infants and toddlers with disabilities and their families. Parents of infants and toddlers with disabilities, early intervention service (EIS) programs and providers, the State Interagency Coordinating Council, and other stakeholders are critical partners in improving results for infants and toddlers and their families. States are required to include broad representation of stakeholders in implementing, evaluating, and revising each phase of the SSIP. In Phase I of the SSIP [federal fiscal year (FFY) 2013], each state established baseline data for Indicator 11. Baseline data were expressed as a percentage and aligned with the State Identified Measurable Result(s) [SIMR(s)] for infants and toddlers and their families. Each state also established measurable and rigorous targets, expressed as percentages, for each of the five years from FFY 2014 through FFY 2018. These data were submitted to OSEP in April 2015. In Phase II (FFY 2014), each state developed and submitted a plan to build state capacity to support EIS programs and/or EIS providers to implement evidence-based practices (EBPs) that would lead to measurable improvement in the SIMR. States developed a plan that included activities, steps, and resources needed to implement the coherent improvement strategies, with attention to the research on implementation, timelines for implementation, and measures to evaluate implementation and impact on the SIMR. This plan was submitted to OSEP in April 2016. In Phase III - Year 1 (FFY 2015), states began implementation and evaluation of their plans. In their annual reports due to OSEP in April 2017, states were required to report on their progress made during the first year of SSIP implementation, consistent with the evaluation plan developed in Phase II. States were expected to include data and analysis on the extent to which they made progress on and/or met their short-term and long-term objectives for implementing the SSIP and progress on achieving the SIMR(s). In addition, states were required to describe how the data from their evaluation informed 24 their decisions about continuing SSIP implementation without modifications or instead providing rationale for revisions made or revisions to be made. Finally, states were required to describe how stakeholders were included in the decision-making process. The data in this report are based on an analysis of the FFY 2015 SSIP reports submitted by 55 of 56 states and jurisdictions. (One state had not yet submitted its SSIP report at the time of this analysis.) Submissions were analyzed by technical assistance providers, and the results were summarized for this report. States and jurisdictions are referred to as “states” in the remainder of this report. FFY 2015 SIMR DATA In the FFY 2015 SPP/APR, states were required to report that year’s progress data, expressed as a percentage and aligned with the SIMR for infants and toddlers with disabilities and their families. The FFY 2015 progress data were compared with the FFY 2015 measurable and rigorous targets, also expressed as a percentage. FFY 2015 data were also compared with the SIMR progress data reported in FFY 2014 (see “Progress Toward Achieving Intended Improvements”). Child and Family Outcomes Identified in the SIMRs Each state has identified a child and/or family outcome as the focus of its SIMR. Fortynine of the 55 states that submitted an SSIP in FFY 2015 selected a single outcome for their SIMR and reported one percentage for their FFY 2015 SIMR data. Five states selected multiple child and/or family outcomes as their SIMR (either all within Part C or a combination of child outcomes across Part C and Part B Section 619). For one state, it was not clear whether the focus of the SIMR was a single outcome or multiple outcomes. States that selected multiple outcomes for the focus of their SIMR opted to either combine the data into a single percentage or report more than one percentage (one percentage for each child and/or family outcome included in the SIMR). More than half of the states (28 of 55, 51%) continued to focus on greater than expected growth in children’s positive social-emotional skills (C3A-SS1) (Figure 1). Two states included Part B 619 child outcomes in their measurement: both included preschool children’s knowledge and skills (B7B) and one of these states also included preschool children’s positive social-emotional skills (B7A). Two states identified SIMRs that were not equivalent to an APR indicator measurement (labeled as “Other Family and Other Child” in Figure 1). While most SIMRs were focused on child outcomes, five states (9%) included at least one family outcome in their SIMR. Three states’ SIMRs included measuring early intervention to determine the extent that it helped families help their child develop and learn (C4C). One state’s SIMR focused on whether early intervention helped families effectively communicate their children’s needs (C4B). In the fifth state, the family focus was not equivalent to an APR indicator. The numbers in Figure 1 do not add to 55 because some states reported multiple outcomes for their SIMR. 25 Figure 1 Number of States Using Each SIMR Measurement (n = 54) C3A - SS1 C3B- SS1 C3B-SS2 C3C-SS1 C4C C3A - SS2 B7B-SS1 Other Family Other Child C4B B7A-SS1 C3C-SS2 28 10 7 3 3 3 C3A – EI child outcome: social-emotional skills C3B – EI child outcome: knowledge and skills C3C – EI child outcome: action to meet needs B7A – 619 child outcome: social-emotional skills B7B – 619 child outcome: knowledge and skills SS1 – % of children that made greater than expected growth SS2 – % of children exited the program within age expectations C4C – family outcome: EI helped families help their child develop and learn C4B – family outcome: EI helped families effectively communicate their children’s needs Other Family – family outcome not aligned with an APR indicator Other Child – child outcome not aligned with an APR indicator 2 1 1 1 1 1 0 5 10 15 20 25 30 Note: The count of indicator measurements on this graph is greater than 55 because some states used more the one indicator for their SIMR measurement. STAKEHOLDER ENGAGEMENT States were expected to engage stakeholders throughout the year in the implementation of the SSIP as well as in the evaluation and report on this involvement in SSIP Phase III. For example, stakeholders might support the implementation of activities, review evaluation data, and make decisions about adjustments or additions to existing plans. In addition to the broad data on stakeholder involvement, the specific roles stakeholders played in implementation and evaluation are described in later sections of this report. The stakeholder groups most commonly involved in SSIP Phase III were state implementation teams (87%), State Interagency Coordinating Councils (85%), subgroups focusing on specific content (67%), and local implementation teams (51%) (Figure 2). A small number of states (16%) had subgroups focused specifically on evaluation of the SSIP. Other reported stakeholder groups included special education advisory committees, provider groups, parent groups, and a family advisory committee. 26 Figure 2 Types of Stakeholder Groups Involved in SSIPs (n = 55) State implementation teams 87% Interagency Coordinating Councils 85% Subgroups focusing on specific content 67% Local implementation teams 51% Subgroups focusing on evaluation 16% Other 11% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Note: Percentages do not add to 100% because states reported multiple types. States engaged stakeholders in decision-making to differing degrees (Figure 3). The majority of states reported that stakeholders either primarily made recommendations (42%) or made recommendations in some areas and made decisions in others (44%). A small number of states (5%) reported that stakeholders were only recipients of information and did not provide input. Another small group of states (5%) reported that stakeholders almost always made decisions. 27 Figure 3 Stakeholder Involvement in Decision-Making (n = 55) 5% Made recommendations in some areas, made decisions in others 4% 5% Primarily made recommendations 44% Received information only; were not involved in making decisions Almost always made decisions 42% Unspecified States reported a broad variety of examples of the types of decisions stakeholders made:  Selected EBPs.  Determined training timelines and developed training competencies.  Developed practice guides.  Identified specific processes and procedures for implementing EBPs.  Developed criteria for and chose pilot sites.  Identified outcome measures for use in both SSIP evaluation activities and child evaluation and assessment.  Participated in hiring staff to implement EBPs.  Influenced decision to focus on families and culturally appropriate implementation of practices.  Developed online and web-based communication strategies for families to respond and engage and not just be informed. SUMMARY OF CHANGES MADE TO SSIP PHASE II PLANS In Phase III, states were required to report on how the data from their evaluation shaped their decisions about continuing SSIP implementation without modifications or instead providing a rationale for revisions made or revisions to be made. They were also required to describe how stakeholders were included in the decision-making process. 28 Accordingly, each state described the changes made to its SSIPs, including the theory of action, logic model, improvement plan, and/or evaluation plan. Changes Made to SIMRs Fifty-four of the 55 states (98%) that submitted a SSIP report did not make changes to their SIMRs in FFY 2015. One state did not include its SIMR in its report, nor did it mention whether any changes were made. Revisions to SIMR Baseline and Targets States were required to review their baseline and targets for FFY 2015 through FFY 2018. On the basis of stakeholder input and justification, states were able to revise their baseline and targets. Nine states reported in Phase III they had revised their SIMR baseline data since the FFY 2014 SSIP report. In seven of these nine states, the new baseline was lower than the baseline reported in FFY 2014. In one state, the new baseline was higher than that reported in FFY 2014. One state that revised its baseline had a clustered SIMR with four baselines; two of the baselines were lower, and two were higher. Eight states provided rationales for the revisions to the baseline:  Five states revised it because of improved data quality.  One state revised it because of a new data collection process.  One state revised it to include data only from the demonstration sites.  One state revised it to include data from an additional region not in the FFY 2014 baseline data. One state did not provide a rationale for changing its baseline. Of the 55 states that submitted an SSIP in FFY 2015, 50 states reported targets in both FFY 2014 and FFY 2015 (four states did not report targets in FFY 2015, and one state did not report targets in FFY 2014). Ten of the 50 states (20%) had revised their targets since the FFY 2014 report. In half these states the revised FFY 2015 targets were lower than FFY 2014 targets, and in the other half of states they were higher. The states provided the following rationales for these revisions:  Seven states revised the targets to align with their revised baselines.  Two states revised the targets to better reflect expected progress.  One revised the targets because of issues with data quality. Changes to Theories of Action and Logic Models In Phase I of the SSIP, states were required to develop a graphic illustration of the rationale of how implementing their coherent set of improvement strategies would increase the state’s capacity to lead meaningful change in EIS programs and/or EIS providers and achieve improvement in the SIMR (a theory of action). During Phase II, a number of states chose to develop a logic model that further defined the relationship of inputs, activities, outputs, and outcomes to help develop their evaluation plan and ensure the evaluation plan aligned with their improvement plan. 29 In Phase III of the SSIP, eight states (15%) reported modifications to the previously submitted theory of action and 12 states (22%) reported changes to the logic model. Most of the eight states (63%) made changes that elaborated on, or clarified existing activities in their theories of action, whereas two states (25%) reported deleting improvement activities or reducing the number of outcomes in the theory of action (Figure 4). Similarly, changes to logic models were principally modifying activities (58%) and outputs (50%) (Figure 5). States reported that initial data collection and evaluation activities led to adjusted expectations for future implementation. Figure 4 Nature of Changes States Made to Theories of Action Since Phase II (n = 8) Modified existing improvement activities [VALUE] (n = 5) Added improvement activities [VALUE] (n = 2) Condensed or deleted improvement activities [VALUE] (n = 2) Streamlined or reduced number of outcome(s) [VALUE] (n = 2) Unspecified [VALUE] (n = 1) 0% 10% 20% 30% 40% 50% 60% 70% Note: Percentages do not add to 100% because states reported multiple changes. 30 Figure 5 Nature of Changes States Made to Logic Models Since Phase II (n = 12) Activities [VALUE] (n = 7) Outputs [VALUE] (n = 6) Intended outcomes [VALUE] (n = 5) Inputs [VALUE] (n = 1) Unspecified [VALUE] (n = 3) 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple changes. Changes to Improvement Plans Whereas only a small number of states reported changes to their theories of action and logic models, 45 states (82%) reported changing their improvement plan since Phase II. The SSIP improvement plan, intended to lead to better results in the SIMR for infant and toddlers with disabilities and their families, contains specific activities and steps the state will take to enhance results. Most states (78%) that made changes revised timelines for the activities in the plan based on the first year of implementation (Figure 6). Eighteen states (40%) changed the manner of implementing activities, and 15 (33%) added improvement activities. Fewer states consolidated or deleted improvement activities (27%), added specificity to activities (18%), and/or changed collaborative partners (13%). 31 Figure 6 Nature of Changes States Made to Improvement Plans Since Phase II (n = 45) Revised timelines 78% Changed manner of implementing activities 40% Added improvement activities 33% Consolodated or deleted improvement activities 27% Added specificity to activities 18% Changed collaborative partner programs/agencies 13% Unspecified 2% 0% 10% 20% 30% 40% 50% 60% 70% 80% Note: Percentages do not add to 100% because states reported multiple changes. Changes to Evaluation Plans In Phase II, states were required to report on how they would evaluate their SSIP, including the methods to be used to collect and analyze data related to measuring implementation and outcomes. The evaluation plan was expected to specify how the state would use the evaluation results to determine the effectiveness of the implementation of the SSIP and the progress on achieving intended improvements. In addition, the evaluation results would inform potential modifications to the SSIP, including modifications to the evaluation plan. In Phase III, 34 states (62%) reported revising their SSIP evaluation plan. States made different types of changes to their evaluation plans (Figure 7). Twenty-two of those 34 states (65%) revised timelines for evaluations, and 17 states (50%) revised or identified new measurement strategies, data collection methods, or tools. Sixteen states (47%) added, deleted, or reworded outcomes and/or performance indicators. Four states (12%) reported that they made changes to their evaluation plans but did not specify the nature of the changes. 32 Figure 7 Nature of Changes Made to State Evaluation Plans Since Phase II (n = 34) Revised timelines 65% Revised or identified new measurement strategies, data collection methods, tools 50% Added, deleted, or reworded outcomes and/or performance indicators 47% Revised or developed data analysis methods 6% Updated or developed communication and/or reporting strategies 6% Other 3% Unspecified 12% 0% 10% 20% 30% 40% 50% 60% 70% Note: Percentages do not add to 100% because states reported multiple changes. Stakeholder Engagement in Changes States that made changes to their SSIP were required to involve stakeholders in the decision-making process. The type of stakeholder engagement varied across states (Figure 8). Fifty-two of the 55 states (95%) reported that stakeholders reviewed information and provided input on needed changes, and 36 states (65%) reported that stakeholders provided input on how to make the changes. Five states (9%) reported that stakeholders received information but did not otherwise describe the stakeholders’ role. 33 Figure 8 Stakeholder Involvement in Changes to States' SIMR, Baseline, Targets, and Improvement Plans (n = 55) Reviewed information and provided input on changes 95% Provided input on how to make changes 65% Provided feedback on changes 58% Received information 9% Not applicable (no changes this reporting year) 2% Unspecified 2% 0% 20% 40% 60% 80% 100% Note: Percentages do not add to 100% because states reported multiple roles. PROGRESS IN IMPLEMENTING THE SSIP In Phase III, states were required to report progress in implementing the SSIP and were encouraged to include the following information:  A description of the extent to which the state carried out its planned activities with fidelity—what was accomplished, what milestones were met, and whether the intended timeline was followed.  The intended outputs accomplished as a result of the implementation activities. In developing their plan, states had been encouraged to use concepts of implementation science in adopting and implementing their selected evidence-based practices. Effective implementation of EBPs occurs over time and follows five implementation stages: exploration, installation, initial implementation, full implementation, and expansion and scale-up. Use of Implementation Stages in Implementing Evidence-Based Practices Forty-eight states (87%) reported activities reflective of the implementation stages (Figure 9). Some states reported activities in the exploration stage including 16 states that identified the state leadership team (33%) and 14 states (29%) that secured agency and cross-agency leadership support. More states reported activities in the installation 34 stage; 38 (79%) implemented infrastructure changes, 27 (56%) established implementation sites and teams, and 16 (33%) completed written implementation plans. Most states reported activities and steps in the initial implementation stage. Forty-one states (85%) reported that they trained and coached local personnel, and 21 states (44%) reported that state leadership teams and local implementation teams used feedback loops to adjust system supports and that local implementation teams supported practitioners in implementing EBPs. Ten states (21%) reported that the state leadership team and local implementation team had evaluated fidelity of implementation and emerging outcomes. Very few states reported activities in the full implementation and expansion and scaleup stages. Four states (8%) reported that policies and practices were functioning in implementation sites, and two states (4%) reported that they used feedback loops to sustain fidelity of practices and system functioning. Three states (6%) reported that they selected new sites and began implementation in those sites. 35 Figure 9 Use of Implementation Stages This Reporting Year (n = 48) EXPLORATION Identified state leadership team (SLT) 33% Secured leadership support 29% INSTALLATION Developed system supports and infrastructure changes Established sites and local implementation teams (LITs) 79% 56% Completed written implementation plans 33% INITIAL IMPLEMENTATION Trained and coached local personnel 85% LITs supported practitioners to implement EBPs 44% SLT and LITs used feedback loops to adjust system 44% SLT and LITs evaluated fidelity and outcomes 21% FULL IMPLEMENTATION Selected new sites and began implementation process 6% EXPANSION AND SCALE UP Policies and practices functioned in implementation sites Used feedback loops to sustain fidelity of practices 8% 4% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Note: See ECTA Center’s Guide to Implementation Process: Stages, Steps and Activities. Percentages do not add to 100% becuase states reported multiple phases and steps. 36 Progress Implementing Activities Related to Infrastructure Improvements Fifty-three of the 55 states reported in Phase III that they implemented infrastructure improvement activities in the FFY 2015 reporting year. One state did not report implementing infrastructure improvement activities, and for one state reviewers could not tell from the information provided whether infrastructure activities were implemented. The components of state infrastructure selected for improvement varied across states (Figure 10). Most states (94%) improved the Professional Development and Technical Assistance, state or local Governance (70%), and Data System (66%) components of the infrastructure. To a lesser extent, infrastructure improvements focused on the Accountability and Quality Improvement (43%), Quality Standards (30%), and Finance (28%) components. Figure 10 State Infrastructure Components Selected for Improvement (n = 53) Professional Development and Technical Assistance 94% Governance 70% Data System 66% Accountability and Quality Improvement 43% Quality Standards 30% Finance 28% 0% 20% 40% 60% 80% 100% Note: Percentages do not add to 100% because states reported multiple components. For the Professional Development and Technical Assistance component of the infrastructure, 50 states reported implementing activities (Figure 11). Eight states (16%) had fully implemented all or almost all planned improvement activities in FFY 2015. Thirty-four states (68%) had fully implemented some activities with others in process, and in eight states (16%) activities were all still in process. 37 Figure 11 Professional Development and Technical Assistance: Implementation Status of SSIP Improvement Activities (n = 50) 16% 16% All or almost all planned improvement activities were completed Some improvement activities were completed, some are in process 68% Improvement activities are in process For the Governance component of the infrastructure, 37 states reported implementing activities (Figure 12). Ten states (27%) had completed all or almost all planned improvement activities in FFY 2015. Twenty states (54%) had completed some activities with others in process. In four states (11%), activities were all still in process. Reviewers could not determine the implementation status in three states (8%). 38 Figure 12 Governance: Implementation Status of SSIP Improvement Activities (n = 37) All or almost all planned improvement activities were completed 8% 11% 27% Some improvement activities were completed, some are in process Improvement activities are in process 54% Unspecified For the Data System component, 34 states reported implementing activities (Figure 13). Nine states (26%) had completed all or almost all planned improvement activities in FFY 2015. Fourteen states (41%) had completed some activities with others in process, and in ten states (29%) activities were all still in process. Reviewers could not determine the implementation status in one state. 39 Figure 13 Data System: Implementation Status of Improvement Activities (n = 34) 3% All or almost all planned improvement activities were completed 26% 29% Some improvement activities were completed, some are in process Improvement activities are in process 41% Unspecified For the Accountability and Quality Improvement component of the infrastructure, 23 states reported implementing activities (Figure 14). Four states (17%) had completed all or almost all planned improvement activities in FFY 2015. Thirteen states (57%) had completed some activities with others in process, and in five states (22%) activities were all still in process. Reviewers could not determine the implementation status in one state. 40 Figure 14 Accountability and Quality Improvement: Implementation Status of SSIP Improvement Activities (n = 23) All or almost all planned improvement activities were completed 4% 17% 22% Some improvement activities were completed, some are in process Improvement activities are in process 57% Unspecified For the Quality Standards component of the infrastructure, 16 states reported implementing activities (Figure 15). Five states (31%) had completed all or almost all planned improvement activities in FFY 2015. Six states (38%) had completed some activities with others in process. In four states (25%), activities were all still in process. Reviewers could not determine the implementation status in one state. 41 Figure 15 Quality Standards: Implementation Status of SSIP Improvement Activities (n = 16) All or almost all planned improvement activities were completed 6% 31% 25% Some improvement activities were completed, some are in process Improvement activities are in process 38% Unspecified For the Finance component of the infrastructure, 15 states reported implementing activities (Figure 16). Six states (40%) had completed all or almost all planned improvement activities in FFY 2015. Another six states (40%) had completed some activities with others in process. In two states (13%), activities were all still in process. Reviewers could not determine the implementation status in one state. 42 Figure 16 Finance: Implementation Status of SSIP Improvement Activities (n = 15) [VALUE] (n=1) All or almost all planned improvement activities were completed [VALUE] (n-2) [VALUE] (n=6) Some improvement activities were completed, some are in process Improvement activities are in process [VALUE] (n-6) Unspecified Progress Implementing Evidence-Based Practices States were expected to report on SSIP activities that supported the implementation of EBPs. Because states were in various stages of implementation and evaluation of practices, the Phase III analysis identified work that states had completed as well as work they had planned for the upcoming year. Forty-nine states (89%) reported implementing activities to improve practices (Figure 17). Two states (4%) had completed all or almost all planned improvement activities in FFY 2015. Thirty-one states (63%) had completed some activities with others in process. In 15 states (31%), activities were all still in process. Reviewers could not determine the implementation status in one state. 43 Figure 17 Practices: Status of Improvement Activities Planned for This Reporting Year (n = 49) 4% 2% All or almost all planned improvement activities were completed Some improvement activities were completed, some are in process 31% 63% Improvement activities are in process Unspecified Several figures below provide detail on the types of activities to improve practices, the practices and models being used, and the specific evidence-based practices that states identified implementing. To improve practices (Figure 18), most states trained providers in EBPs (92%), disseminated information to providers or administrators (78%), and coached providers in implementing EBPs (53%). States also trained administrators in EBPs (29%), provided data to providers or administrators on fidelity of implementation (24%), and trained providers (16%) and administrators (14%) in interpreting and using data. 44 Figure 18 Types of Activities States Implemented to Improve Practices (n = 49) Trained providers in EBPs 92% Disseminated information to providers or administrators 78% Coached providers in EBPs 53% Trained administrators in EBPs 29% Provided implementation data to providers or administrators 24% Trained providers in interpreting and using data 16% Trained administrators in interpreting and using data 14% 0% 20% 40% 60% 80% 100% Note: Percentages do not add to 100% because states reported multiple types of activities. States also reported on the specific EBPs and models that they implemented (Figure 19). The most common selected were DEC Recommended Practices (44%), family support and engagement practices (38%), primary coach approach/coaching practices (31%) and routines-based intervention practices (29%). Less frequently identified EBPs and models were Pyramid Model/social-emotional practices (20%); Mission and Key Principles, Agreed Upon Practices (15%), Family Guided RBI/Caregiver Coaching (7%); MIECHV (Maternal, Infant, and Early Childhood Home Visiting Program) home visiting model (4%); primary provider model (4%); practices for children with autism (2%); and early literacy practices (2%). Nine states (16%) reported various other practices not shown in Figure 19, including specific early childhood curricula and frameworks such as Promoting First Relationships, Help Me Grow, Natural Learning Activities, and Collaborative Problem Solving. 45 Figure 19 Evidence-Based Practices and Models Implemented by States (n = 55) Selected DEC Recommended Practices 44% Family support/engagement practices 38% Coaching in Natural Learning Environments, coaching practices 31% Routines-based intervention (RBI) practices 29% Pyramid Model, social emotional practices 20% Mission and Key Principles, Agreed Upon Practices 15% Family Guided RBI, Caregiver Coaching 7% MIECHV home visiting model 4% Primary provider model 4% Practices for children with autism 2% Early literacy practices 2% Other 16% Unspecified 7% 0% 10% 20% 30% 40% 50% Note: Percentages do not add to 100% because states reported multiple practices and models. The DEC is “an international membership organization for those who work with or on behalf of young children with disabilities and other special needs and their families”. Its Recommended Practices were developed “to provide guidance to practitioners and families about the most effective ways to improve the learning outcomes and promote the development of young children, birth through age five, who have or are at-risk for developmental delays or disabilities.” The purpose is to “help bridge the gap between research and practice by highlighting those practices that have been shown to result in better outcomes for young children with disabilities, their families, and the personnel who serve them” (http://www.dec-sped.org/dec-recommended-practices). 46 The DEC Recommended Practices are organized in eight topic areas, and SSIP Phase III reviewers categorized states’ implementation into those areas for those states that reported implementing them (24 states). The most implemented practices were related to the following topic areas: Family (58% of states reported implementing), Instruction (42%), Assessment (38%), Teaming and Collaboration (38%), and Environment (33%). Interaction practices (25%), Transition practices (13%) and Leadership practices (8%) were less frequently identified as being used by states. Figure 20 DEC Recommended Practices Implemented by States (n = 24) Family 58% Instruction 42% Assessment 38% Teaming and Collaboration 38% Environment 33% Interaction 25% Transition 13% Leadership 8% Unspecified 21% 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple practices. Progress Implementing Evaluation Activities In Phase III, states reported for the first time on their implementation of evaluation activities to measure progress on their improvement activities and achievement of intended outcomes including the SIMR. Additional information on the data collected through the evaluation activities is reported under “Data on Implementation and Outcomes.” Fifty-five states reported implementing activities in their evaluation plans (Figure 21). Seven states (13%) completed all or almost all planned evaluation activities in FFY 2015. Thirty-one states (56%) completed some activities with others in process. In 12 states (22%), activities were all still in process. Reviewers could not determine the evaluation activity status in five states (9%). 47 Figure 21 Implementation Status of Evaluation Activities Planned for This Reporting Year (n = 55) All or almost all evaluation activities were completed 9% 13% 22% Some activities were completed, some are in process Evaluation activities are in process 56% Unspecified Barriers to Implementation In FFY 2015, 48 states (87%) reported barriers to implementing their improvement plans (Figure 22). The most common barriers were changes in state leadership or support (50%), personnel shortages or unprepared personnel (48%), plans that exceeded fiscal resources (46%), and overambitious timelines that the state was unable to meet (42%). Additional barriers reported include changes in leadership or support at the local level (25%) and limits on data system capacity (17%). Several states reported other (19%) barriers, including individual state contextual barriers (such as geographic or cultural barriers) and the need for reorganization or coordination of activities, which may have been due to one of the other barriers. 48 Figure 22 Barriers to Implementing Improvement Plans (n = 48) Changes in state leaderhip or support 50% Personnel shortages/ unprepared personnel 48% Plans exceeded fiscal resources 46% Overambitious timelines 42% Changes in local leadership or support 25% Limits on data system capacity 17% Other 19% 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple practices. Forty-seven states reported on barriers to implementing activities to improve practices (Figure 23). The most common barriers were personnel shortages or unprepared personnel (62%), lack of fiscal resources to implement plans (43%), and overambitious timelines that the state was unable to meet (36%). Additional barriers reported included changes in state leadership or support (30%), limits on data system capacity (23%), and changes in leadership or support at the local level (21%). Other reasons (11%) primarily included individual state contextual barriers (such as geographic or cultural barriers). 49 Figure 23 Barriers to Implementing Activities to Improve Practices (n = 47) Personnel shortages/unprepared personnel 62% Plans exceeded fiscal resources 43% Inability to meet timelines 36% Changes in state leadership or support 30% Limits on data system capacity 23% Changes in local leadership or support 21% Other 11% 0% 10% 20% 30% 40% 50% 60% 70% Note: Percentages do not add to 100% because states reported multiple barriers. States also reported barriers to implementing evaluation plans. Twenty-nine states (53%) reported barriers (Figure 24). The barriers were very similar to the barriers to implementing practice improvement plans, including overambitious timelines (55%), limits on data system capacity (52%), lack of personnel with evaluation expertise (45%), and plans that exceeded available fiscal resources (17%). 50 Figure 24 Barriers to Implementing Evaluation Plans (n = 29) Overambitious timelines 55% Limits on data system capacity 52% Lack of availability of personnel with evaluation expertise 45% Plans exceeded fiscal resources 17% 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% since some states reported multiple barriers. Stakeholder Engagement and Decision-Making in Implementation Activities Regarding stakeholder engagement and decision-making in implementation of activities to improve the infrastructure, 48 states (87%) reported that stakeholders reviewed information and provided input on changes (Figure 25). Thirty-nine states (71%) reported that stakeholders provided input on how to make changes, and 35 states (64%) reported that stakeholders provided feedback on changes the state proposed. Five states (9%) reported that stakeholders reviewed information but otherwise did not identify how stakeholders were involved in decision-making. One state reported other ways of involving stakeholders, and two states did not provide details on how stakeholders were involved. 51 Figure 25 Stakeholder Involvement in Infrastructure Improvements (n = 55) Reviewed information and provided input 87% Provided input on how to make changes 71% Provided feedback on changes 64% Received information Other Unspecified 9% 2% 4% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Note: Percentages do not add to 100% because states reported multiple ways stakeholders were involved. DATA ON IMPLEMENTATION AND OUTCOMES States were required to report data that reflected whether or not they achieved the intended outputs and outcomes for FFY 2015 as related to the infrastructure and practices. States were also required to report on how stakeholders were engaged in reviewing this evaluation data and their involvement in decision-making. Data on Progress in Infrastructure Improvements Fifty-three of 55 states (96%) reported implementing infrastructure improvement activities in the FFY 2015 reporting year. Reviewers categorized state-reported progress in six infrastructure components, and the percentage of states reporting progress on each component was as follows:  Professional Development and Technical Assistance – 94%.  Governance – 70%.  Data System – 66%.  Accountability and Quality Improvement– 43%.  Quality Standards – 30%.  Finance – 28%. 52 Professional Development and Technical Assistance  Fifty of the 53 states (94%) reported implementing improvement activities related to professional development and technical assistance infrastructure improvements.  Twenty-one of 50 states (42%) reported collecting pre-implementation baseline data to document the beginning status of the Professional Development and Technical Assistance component (Figure 26). Of the variety of tools and methods available to collect baseline data, most states used the status of outputs (67%), state-developed surveys (48%), and the ECTA System Framework SelfAssessment (19%). Figure 26 Professional Development and Technical Assistance: Tools and Methods States Used to Collect Baseline Data (n = 21) Status of outputs 67% State-developed survey 48% ECTA System Framework Self-Assessment 19% Interviews/focus group 5% Stakeholder group perceptions 5% Other 10% 0% 10% 20% 30% 40% 50% 60% 70% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Thirty-three of 50 states (66%) reported collecting data to obtain evidence on whether the Professional Development and Technical Assistance component was changing or had changed (Figure 27). To collect evidence of change, most states used the status of outputs (76%), state-developed surveys (33%), and stakeholder group perceptions as documented in stakeholder meeting minutes or notes from a stakeholder group process activity (15%). Only two states (9%) reported using interviews or focus groups, and one state reported using the ECTA System Framework Self-Assessment to measure professional development and technical assistance improvements. 53 Figure 27 Professional Development and Technical Assistance: Tools and Methods States Used to Collect Change Data (n = 33) Status of outputs 76% State-developed survey 33% Stakeholder group perceptions 15% Interviews or focus groups 9% ECTA System Framework Self-Assessment 3% Other 3% Unspecified 6% 0% 10% 20% 30% 40% 50% 60% 70% 80% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Twenty-eight of the 33 states (85%) provided evidence of progress during the reporting year on the Professional Development and Technical Assessment infrastructure component (Figure 28). Seven states (25%) achieved all or almost all of the planned outputs and outcomes for the FFY 2015 reporting year, and 18 states (64%) achieved some of the planned outputs and outcomes. Two states (7%) achieved none or few of the planned outputs and outcomes related to professional development and technical assistance. 54 Figure 28 Professional Development and Technical Assistance: Status of Outputs and Short-Term and Intermediate Outcomes for FFY 2015 (n = 28) Achieved all or almost all outputs and outcomes 4% 7% 25% Achieved some outputs and outcomes Achieved no or very few outputs and outcomes 64% Unspecified Governance  Thirty-seven of the 53 states (70%) reported implementing improvement strategies related to governance.  Twelve of 37 states (32%) reported collecting baseline data to document the beginning status of the state or local Governance area of the infrastructure (Figure 29). Of the variety of tools and methods available to collect baseline data, most states used the status of outputs (50%), state-developed surveys (33%), the ECTA System Framework Self-Assessment (17%), and stakeholder group perceptions (17%). One state used interviews/focus groups to evaluate its governance improvements. 55 Figure 29 Governance: Tools and Methods States Used to Collect Baseline Data (n = 12) Status of outputs 50% (n-6) State-developed surveys 33% (n-4) ECTA System Framework Self-Assessment 17% (n-2) Stakeholder group perceptions 17% (n-2) Interviews/focus group 8% (n-1) Other 17% (n-2) 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Twenty-eight of the 37 states (77%) collected data to provide evidence on whether the Governance area of the infrastructure was changing or had changed (Figure 30). To collect evidence of change, most of these states (89%) reported on accomplished outputs. Fewer states reported using state-developed surveys (14%) and stakeholder group perceptions (7%) to measure improvements in governance. One state (4%) reported that it used extant referral, caseload, and evaluation data from a regular state and local data collection. 56 Figure 30 Governance: Tools and Methods States Used to Collect Change Data (n = 28) Status of outputs 89% State-developed surveys 14% Stakeholder group perceptions 7% Other 4% Unspecified 7% 0% 20% 40% 60% 80% 100% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Twenty-three of 28 states (82%) provided evidence of progress during the reporting year on governance (Figure 31). Nine states of these states (39%) achieved all or almost all of the planned outputs and outcomes for the FFY 2015 reporting year, and 13 states (57%) achieved some of the planned outputs and outcomes. 57 Figure 31 Governance: Status of Outputs and Short-Term and Intermediate Outcomes (n = 23) 4% Achieved all or almost all outputs and outcomes 39% Achieved some outputs and outcomes 57% Achieved no or very few outputs and outcomes Data System  Thirty-five of the 53 states (66%) reported implementing improvement strategies related to the Data System infrastructure component.  Fifteen of 35 states (43%) reported collecting baseline data to document the beginning status of the Data System component (Figure 32). Of the tools and methods available to collect baseline data, most states used the status of outputs (67%), the ECTA System Framework Self-Assessment (27%), state-developed surveys (20%), and stakeholder group perceptions (13%). Three states (20%) reported that they used other data collection tools and methods, including child outcome “completeness” data, the BDI-2 Fidelity Checklist, the Child Outcomes Measurement System Framework, and the ENHANCE survey. 58 Figure 32 Data System: Tools and Methods States Used to Collect Baseline Data (n = 15) Status of outputs 67% (n = 10) ECTA System Framework Self-Assessment 27% (n = 4) State-developed survey 20% (n = 3) Stakeholder group perceptions 13% (n = 2) Other 20% (n = 3) 0% 10% 20% 30% 40% 50% 60% 70% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Twenty-five of the 35 states (71%) collected data to provide evidence on whether the Data System component of the infrastructure was changing or had changed (Figure 33). To collect evidence of change, 23 states (92%) used the status of outputs, three states used state-developed surveys (12%), and one state used stakeholder group perceptions. One state used other extant data from the child outcomes data collection for the APR. 59 Figure 33 Data System: Tools and Methods States Used to Collect Change Data (n = 25) Status of outputs 92% State-developed survey 12% Stakeholder group perceptions 4% Other 4% Unspecified 4% 0% 20% 40% 60% 80% 100% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Twenty-one of the 25 states (84%) provided evidence of progress during the reporting year on the data system (Figure 34). Seven of these states (33%) achieved all or almost all of the planned outputs and outcomes for the FFY 2015 reporting year, and 12 states (57%) achieved some of the planned outputs and outcomes. 60 Figure 34 Data System: Status of Outputs and Short-Term and Intermediate Outcomes (n = 21) Achieved all or almost all outputs and outcomes 10% 33% Achieved some outputs and outcomes 57% Unspecified Accountability and Quality Improvement.  Twenty-three of the 53 states (43%) reported implementing improvement strategies related to accountability and quality improvement.  Six of these 23 states (26%) reported collecting baseline data to document the beginning status of the Accountability and Quality Improvement component of the infrastructure (Figure 35). Of the tools and methods available to collect baseline data, most states used the status of outputs (67%) and state-developed surveys (17%). Two states (33%) used other data sources including extant Battelle Developmental Inventory (BDI) and child outcomes data and a report from the state’s data system. 61 Figure 35 Accountability and Quality Improvement: Tools and Methods States Used to Collect Baseline Data (n = 6) Status of outputs 67% (n = 4) State-developed survey 17% (n = 1) Other 33% (n = 2) 0% 10% 20% 30% 40% 50% 60% 70% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Sixteen of the 23 states (70%) collected data to provide evidence on whether the Accountability and Quality Improvement component of the infrastructure was changing or had changed (Figure 36). Of the tools and methods available to collect evidence of change, most states (81%) used the status of outputs. Statedeveloped surveys, interviews or focus groups, and stakeholder group perceptions were each reported by six percent of states. Two states (13%) used other data including child outcomes data, videos, fidelity measurements, and child outcomes “completeness” data. 62 Figure 36 Accountability and Quality Improvement: Tools and Methods States Used to Collect Change Data (n = 16) Status of outputs 81% State-developed survey 6% Interviews/focus group 6% Stakeholder group perceptions 6% Other 13% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Thirteen of 16 states (81%) provided evidence of progress during the reporting year on accountability and quality improvement (Figure 37). Three of these states (23%) achieved all or almost all of the planned outputs and outcomes for the FFY 2015 reporting year, and six states (46%) achieved some of the planned outputs and outcomes. Two states (15%) achieved none or few of the outputs and outcomes related to accountability and quality improvement. 63 Figure 37 Accountability and Quality Improvement: Status of Outputs and Short-Term and Intermediate Outcomes (n = 13) [VALUE] (n=2) Achieved all or almost all outputs and outcomes [VALUE] (n=3) Achieved some outputs and outcomes [VALUE] (n=2) Achieved no or very few outputs and outcomes [VALUE] (n=6) Unspecified Quality Standards  Sixteen of the 53 states (30%) reported implementing improvement strategies related to quality standards.  Six of these 16 states (38%) reported collecting baseline data to document the beginning status of the Quality Standards component of the infrastructure (Figure 38). Of the tools and methods available to collect baseline data, most states used the status of outputs (50%), state-developed surveys (33%), the ECTA System Framework Self-Assessment (17%), and stakeholder group perceptions based on meeting minutes or notes (17%). Two states (33%) reported using other data sources such as general compliance and performance indicators and data from a statewide database. 64 Figure 38 Quality Standards: Tools and Methods States Used to Collect Baseline Data (n = 6) Status of outputs [VALUE] (n = 3) State-developed survey [VALUE] (n = 2) ECTA System Framework Self-Assessment [VALUE] (n = 1) Interviews and focus groups [VALUE] (n = 1) Other [VALUE] (n = 2) 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Ten of the 16 states (63%) collected data to provide evidence of whether the Quality Standards component of the infrastructure was changing or had changed (Figure 39). Of the tools and methods states available to collect evidence of change, most states used the status of outputs (90%), the ECTA Framework Self-Assessment (20%), or a state-developed survey (20%). 65 Figure 39 Quality Standards: Tools and Methods States Used to Collect Change Data (n = 10) Status of outputs 90% (n = 9) ECTA System Framework Self-Assessment 20% (n = 2) State-developed survey 20% (n = 2) Other 10% (n = 1) 0% 20% 40% 60% 80% 100% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Nine states provided evidence of progress during the reporting year on quality standards (Figure 40). Five of these states (56%) achieved all or almost all of the planned outputs and outcomes for the FFY 2015 reporting year, and four states (44%) achieved some of the planned outputs and outcomes. 66 Figure 40 Quality Standards: Status of Outputs and Short-Term and Intermediate Outcomes (n = 9) Achieved all or almost all outputs and outcomes [VALUE] (n-4) [VALUE] (n-5) Achieved some outputs and outcomes Finance  Fifteen of the 53 states (28%) reported implementing improvement strategies related to finance.  Seven of these15 states (47%) reported collecting baseline data to document the beginning status of the Finance component of the infrastructure (Figure 41). Of the tools and methods available to collect baseline data, most states used the status of outputs (43%), state-developed surveys (29%), and the ECTA System Framework Self-Assessment (14%). One state used child and provider data about monthly service hours. 67 Figure 41 Finance: Tools and Methods States Used to Collect Baseline Data (n = 7) Status of outputs [VALUE] (n = 3) State-developed surveys [VALUE] (n = 2) ECTA System Framework Self-Assessment [VALUE] (n = 1) Other [VALUE] (n = 1) 0% 10% 20% 30% 40% 50% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Nine of the 15 states (60%) collected data to provide evidence of whether the Finance component of the infrastructure was changing or had changed (Figure 42). Of the tools and methods states available to collect evidence of change, most states (89%) used the status of outputs, while others used state-developed surveys (11%) or stakeholder group perceptions (11%). 68 Figure 42 Finance: Tools and Methods States Used to Collect Change Data (n = 9) Status of outputs [VALUE] (n = 8) State-developed surveys [VALUE] (n = 1) Stakeholder group perceptions [VALUE] (n = 1) 0% 20% 40% 60% 80% 100% Note: Percentages do not add to 100% because states reported multiple tools and methods.  Nine states provided evidence of progress during the reporting year on the Finance component (Figure 43). Five of these states (56%) achieved all or almost all of the planned outputs and outcomes for the FFY 2015 reporting year, and three states (33%) achieved some of the planned outputs and outcomes. 69 Figure 43 Finance: Status of Outputs and Short Term and Intermediate Outcomes (n = 9) [VALUE] (n=1) Achieved all or almost all outputs and outcomes [VALUE] (n=3) [VALUE] (n=5) Achieved some outputs and outcomes Unspecified Data on Change in Practices In addition to reporting on changes to infrastructure, states reported on evidence collected to show change in practices. Forty-nine of the 55 states (89%) reported that they implemented activities directly related to improving practices. Of those 49 states, 30 (61%) collected baseline data to document the beginning status of improving implementation of practices. To collect baseline data on the status of EBPs, most states used state-developed tools and instruments (70%), tools or instruments developed by model developers (30%), or tools or instruments developed for DEC Recommended Practices (20%) (Figure 44). Other tools and instruments reported by five states (17%) included extant data and videos. 70 Figure 44 Improved Practices: Tools States Used to Collect Baseline Data (n = 30) State-developed tools and instruments 70% Tools and instruments developed by model developers 30% Tools and instruments developed for DEC Recommended Practices 20% Other 17% 0% 10% 20% 30% 40% 50% 60% 70% 80% Note: Percentages do not add to 100% because states reported multiple tools and instruments. Nineteen of 49 states (39%) collected data to provide evidence on whether the implementation of practices was changing or had changed. Most states used statedeveloped tools and instruments (58%), tool and instruments developed by model developers (47%), or tools and instruments developed for DEC Recommended Practices (16%) to collect evidence of change (Figure 45). 71 Figure 45 Improved Practices: Tools States Used to Collect Change Data (n = 19) State-developed tools and instruments 58% Tools and instruments developed by model developers 47% Tools and instruments developed for DEC Recommended Practices 16% Other 11% 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple tools and instruments. Of the 19 states that reported evidence of improving implementation of practices, 17 (89%) presented data showing that change or progress was achieved during the FFY 2015 reporting year (Figure 46). Five of those 17 states (29%) achieved all or almost all of the planned outputs and outcomes for the FFY 2015 reporting year, and nine states (53%) achieved some of the planned outputs and outcomes. Only a few states (6%) achieved none or very few of the planning outputs and outcomes this reporting year. 72 Figure 46 Improved Practices: Status of Outputs and Short-Term and Intermediate Outcomes (n = 17) Achieved all or almost all outputs and outcomes 12% 6% 29% Achieved some outputs and outcomes Achieved none or very few outputs and outcomes 53% Unspecified The forty-nine states that reported on improving implementation of practices also reported on the comparisons that they used or will use to determine change in practices (Figure 47). Most states did or will use a comparison of pre- and post-data (53%); a comparison to an established criteria, standard, or target (fidelity) (35%); or longitudinal change over time (24%) to determine the amount of change in practices. Some states reported that they will use only a post measure (6%) or a comparison group (6%). Twelve states (24%) did not clearly describe how they would do comparisons to evaluate progress in changing practices. 73 Figure 47 Comparisons States Used or Will Use to Determine Change in Practices (n = 49) Comparison of pre- to post-data 53% Comparison to criteria, standard, or target (fidelity) 35% Longitudinal change over time 24% Post measure only 6% Comparison group 6% Unspecified 24% 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple comparisons. Fifteen of the 49 states (31%) also reported that they had collected data and established criteria to determine the fidelity of practices (the extent to which practices were implemented as intended) in FFY 2015 (Figure 48). Most states used tools or instruments developed by model developers (73%) or state-developed tools or instruments (40%). Two states (13%) used tools or instruments developed for DEC Recommended Practices. For one state, reviewers were unable to determine the tool the state would use to evaluate fidelity. 74 Figure 48 Improved Practices: Tools States Used to Determine Fidelity of Practices (n = 15) Tools and instruments developed by model developers 73% (n = 11) State-developed tools and instruments Tools and instruments developed for DEC Recommended Practices Unspecified 40% (n = 6) 13% (n = 2) 7% (n = 1) 0% 10% 20% 30% 40% 50% 60% 70% 80% Note: Percentages do not add to 100% because states reported multiple tools and instruments. Figure 49 shows the results from the 15 states that measured fidelity of practices. One state implemented all or almost all of the targeted practices as intended. Six states (40%) implemented some targeted practices with fidelity, and two states (13%) implemented none or very few with fidelity. The remaining six states did not provide sufficient information for reviewers to determine the degree to which practices were implemented as intended. 75 Figure 49 Extent to Which Practices Were Implemented with Fidelity (n = 15) [VALUE] (n=1) All or almost all targeted practices Some targeted practices [VALUE] (n=6) [VALUE] (n-6) [VALUE] (n=2) None or very few targeted practices Unspecified Stakeholder Engagement and Decision-Making in Evaluation States were expected not only to inform stakeholders of the ongoing evaluation of the SSIP, but also to involve them in it. This entailed supporting stakeholders in reviewing evaluation data, expressing input or sharing ideas about the evaluation process and the meaning of the data, and contributing to the decision-making process including midcourse corrections to the improvement and evaluation plans. The 47 states that reported involving stakeholders in the evaluation of the SSIP in FFY 2015 did so in a variety of ways (Figure 50). Most stakeholders reviewed the evaluation results to provide input on modifications to the SSIP (55%), reviewed data to examine the effectiveness of implementation (53%), provided input on data collection strategies (47%), and reviewed data to examine progress on the SIMR (45%). A few stakeholder groups analyzed data (17%), while some received information only and were not involved in providing input or making decisions (15%). Some states (15%) did not provide sufficient detail for reviewers to determine the level of stakeholder engagement and decision-making in evaluation. 76 Figure 50 Stakeholder Involvement in Evaluation (n = 47) Reviewed evaluation results 55% Reviewed implementation data 53% Provided input on data collection strategies 47% Reviewed data to examine progress 45% Participated in data analysis 17% Received information 15% Unspecified 15% 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple ways stakeholders were involved. DATA QUALITY ISSUES In Phase III, states were asked to report any evaluation data quality limitations that affected reports of progress in implementing the SSIP and achieving the SIMR. Fortythree of 55 states (78%) reported concerns or limitations related to the quality or quantity of the data used to report progress or results. The types of data that were prone to quality issues are indicated in Figure 51. The most common data quality issues were with child outcomes data (72%), with many states working on improving the quality of their child outcomes measurement. States reported concerns about the quality of evaluation data for the SSIP, including data describing the status or changes to practices (37%), data documenting implementation of improvement activities (21%), data on fidelity of practices (21%), and data on status or changes to infrastructure (14%). A few states reported issues with the quality of their family outcomes data (16%). Other data quality issues reported were survey data being prone 77 to subjectivity and incomplete child service and Individualize Family Service Plan (IFSP) data. Figure 51 Types of Data with Quality Challenges (n = 43) Child outcomes data 72% Data describing status or changes to practices 37% Data documenting implementation of activities 21% Data on fidelity of practices 21% Family outcomes data 16% Data on changes to infrastructure 14% Other 7% 0% 10% 20% 30% 40% 50% 60% 70% 80% Note: Percentages do not add to 100% because states reported multiple types. PROGRESS TOWARD ACHIEVING INTENDED IMPROVEMENTS In Phase III, states reported their progress on achieving intended improvements in infrastructure, implementation of EBPs, and SIMR results (child and family outcomes). SSIP Phase III reviewers looked at each state’s reporting on infrastructure component improvement activities, change in practices, and outcomes to identify whether states had achieved the intended improvements planned for the reporting year. Progress on Improving System Infrastructure As described in “Data on Implementation and Outcomes,” states have made notable progress on achieving intended improvements in their systems and practices. Figures 28, 31, 34, 37, 40, and 43 present state data on achieving outputs and outcomes for enhancing state systems.  Twenty-eight states reported evidence of progress on enhancing their Professional Development and Technical Assessment system.  Twenty-three states reported evidence of progress on enhancing their Governance system.  Twenty-one states reported evidence of progress on enhancing their Data System. 78    Thirteen states reported evidence of progress on enhancing their Accountability and Quality Improvement system. Nine states reported evidence of progress on enhancing their Quality Standards component of the infrastructure. Nine states provided reported evidence of progress on enhancing their Finance system. Progress on Improving Practices Also described in “Data on Implementation and Outcomes,” states have started to make progress toward achieving improved practices. Figures 44 through 49 provide data on states’ progress on improved implementation of EBPs.  Forty-nine states have implemented activities to improve implementation of practices, and 19 of those states reported data that showed practices have improved.  Fifteen states have measured implementation fidelity, and seven of those states found that some or all practices were implemented as intended. Progress in Meeting SIMR Targets) for FFY 2015 States were required to report data collected for the SIMR to determine whether they made progress and whether they met the SIMR target for FFY 2015. Fifty of 55 states (91%) clearly reported FFY 2015 SIMR progress data. Of the five states that did not report,  Three states did not report any SIMR progress data.  One state reported data, but it was unclear which of the reported data were SIMR data.  One state reported SIMR progress data in a line graph but did not report SIMR progress data as a percentage. Fifty-one of 55 states (93%) included the FFY 2015 SIMR target in their Phase III report. Target and SIMR data were compared to determine whether the state met its FFY 2015 target. The one state that did not report SIMR progress data as a percentage but rather in a line graph did report target data for FFY 2015 and was included in the 51 states reporting targets. States were coded as meeting their targets if their actual FFY 2015 data were at or above their FFY 2015 targets for all outcomes associated with the SIMR. Twenty-six of the 51 states (51%) reported that they met their FFY 2015 targets for Indicator 11, and 25 states (49%) did not meet their FFY 2015 targets. Progress or Slippage in Improving the SIMR In determining whether states had progressed or slipped in improving their SIMR data (child and/or family outcomes data), reviewers compared the actual SIMR data reported for FFY 2015 and FFY 2014, available on GRADS 360. No progress meant that actual FFY 2015 SIMR data were less than last year's data. A state was determined to be making progress if its actual FFY 2015 data were the same as or above the SIMR data reported in FFY 2014 for all outcomes associated with the SIMR (for states with multiple outcomes). Data were available for 51 states. For the one state that did not report SIMR progress data as a percentage but rather in a line graph, reviewers were able to 79 determine if progress or slippage was made. A comparison could not be made for four states that did not clearly report FFY 2015 SIMR progress data. Twenty-three of the 51 states (45%) made progress in the SIMR between FFY 2014 and FFY 2015. Twenty-eight states (55%) did not make progress. Twelve states directly attributed improvement in the SIMR data to the implementation of SSIP activities. When reporting on the types of activities that improved the SIMR data, five states cited change in practices and the implementation of new practices. Ten states reported that changes in their infrastructure contributed to SIMR progress. ANTICIPATED BARRIERS AND TECHNICAL ASSISTANCE NEEDS FOR NEXT YEAR In the FFY 2015 SSIPs, states reported on anticipated barriers and technical assistance needs for FFY 2016. Anticipated Barriers Anticipated barriers for the next year included the following:  Providers’, families’, and local administrators’ attitudes and willingness to participate.  Personnel turnover and shortages.  Identifying staff to train other staff on evidence-base practices.  Funding.  Conducting evaluation activities, especially measuring fidelity and attributing changes to specific activities.  Need to increase capacity of data systems.  Adapting to changes in infrastructure.  Delays in timelines pushing back on other activities.  Changes in leadership. Technical Assistance Needs Of the 55 Phase III reports reviewed, 38 states (69%) reported the need for technical assistance in implementing their SSIPs next year (Figure 52). Specific areas of technical assistance identified by states included implementing EBPs (35%), SSIP evaluation (35%), improving infrastructure (25%), general SSIP technical assistance (22%), using implementation science principles (7%), and stakeholder involvement (2%). 80 Figure 52 Areas of Need for Technical Assistance (TA) (n = 55) Implementing EBPs 35% SSIP evaluation 35% Improving infrastructure 25% General need for SSIP TA 22% Using implementation science principles 7% Stakeholder involvement 2% No TA needs reported 31% 0% 10% 20% 30% 40% Note: Percentages do not add to 100% because states reported multiple areas. For the 18 states that reported a need for technical assistance on SSIP evaluation, reviewers collected further data on what the needs were. States reported a need for assistance with evaluation planning (50%), developing or adapting evaluation tools and measures (39%), data collection procedures (28%), data analysis (17%), local data use (11%), SIMR data quality (6%), state data use (6%), reporting the results of SSIP evaluation activities (6%), and other needs (33%) (Figure 53). Other needs reported were assistance in revising SIMR targets and broad evaluation technical assistance. 81 Figure 53 Aspects of SSIP Evaluation Identified for TA Support (n = 18) Evaluation planning 50% Developing or adapting evaluation tools/measures 39% Data collection procedures 28% Data analysis 17% Local data use 11% SIMR Data quality 6% State data use 6% Reporting results of SSIP evaluation 6% Other 33% Unspecified 11% 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple areas. Many states named specific technical assistance centers and providers in their plans for next year (Figure 54). States reported that they will access technical assistance from the National Center for Systemic Improvement (NCSI) (53%), the Early Childhood Technical Assistance Center (ECTA) (47%), the IDEA Data Center (IDC) (40%), and the Center for IDEA Early Childhood Data Systems (DaSy) (40%). Other centers mentioned (9%) included specific model developers, professional organizations, and regional technical assistance providers. 82 Figure 54 Centers from Which States Will Access TA Support (n = 55) NSCI 53% ECTA 47% IDC 40% DaSy 40% Other 9% Unspecified 7% None mentioned 29% 0% 10% 20% 30% 40% 50% 60% Note: Percentages do not add to 100% because states reported multiple centers. CONCLUSION This analysis describes how states reported on the implementation and evaluation of their SSIPs in FFY 2015 Phase III. Specifically, states reported on progress in implementing activities to improve their infrastructure and support implementation of evidence-based practices as well as their progress in accomplishing planned outputs and achieving intended outcomes including their SIMR. Many states reported making changes to their improvement and evaluation plans, whereas only some reported making changes to their theory of action and/or logic models. Almost all states reported that they implemented infrastructure improvements as well as activities to improve practices despite encountering implementation barriers. Most states reported having evidence of change in one or more infrastructure components, and some reported on collecting data that demonstrated changes in practices. Of those states that reported evidence of change in infrastructure and/or practices, most reported completing some of the planned outputs and outcomes they anticipated planning for FFY 2016. About half the states reported that FFY 2015 SIMR data met their target and they made progress on improving their SIMR from FFY 2014. Almost all states reported they engaged stakeholders in their SSIP during Phase III. Plans for next year and barriers anticipated for FFY 2016 were also identified in the SSIP, as were technical assistance needs for ongoing implementation of the SSIP. 83