Very Initial Observations on the 2016 California Smarter Balanced Results D. J. McRae, Ph.D. 08/24/16 Very briefly, here are some initial observations on the 2016 California Smarter Balanced results: First, the overall increases from 2015 to 2016 can be characterized as very good. The data on pages 2-3 below show that the average gains were 3.8 percentage points (for Percent Met and Above), which is a good grade point average in anyone’s book, a solid A- performance. Second, the gains for underserved students [low wealth, English Learners, Students with Disabilities] lagged behind gains for all other students (see Press Release Tables 5 & 6) indicating achievement gaps increased from 2015 to 2016. The acknowledged shallowness of Smarter Balanced item pools for measuring low-achieving students may have contributed to this result. Third, as noted in the Press Release, the increases in number of students being taught via the Common Core and the gains in experience with computer-administered tests both contributed to the gains reported; it is very difficult to say how much either of these factors contributed to the overall results. Fourth, the numbers of students counted as participating but not responding to a sufficient number of test questions to generate a score decreased significantly in 2016 compared to 2015; for ELA, the decrease indicated only 1/3 of the number of 2015 students not receiving scores, while for Math the decrease indicated only half the number of 2015 students not receiving scores. These data show the testing program is most likely moving in the right direction in terms of low scoring students being able to generate a legitimate score on the test. Finally, it is very obvious the Smarter Balanced ELA test is fundamentally easier than the Smarter Balanced Math test. Why this is the case is not clear, but most likely one can point to last minute changes in the Smarter Balanced standards-setting exercise (i.e., setting cut scores for the performance categories) in 2014 as at least a contributing reason for these differences. Readers should be very cautious 1attempting to assign meaning to the differences between ELA and Math Smarter Balanced scores.Smarter Balanced 2016 State-by-State Comparisons [Level 3 & Above Percents] Compiled by D. J. McRae, Ph.D. As of 08/24/16 English/Language Arts Grade 1 2 3 4 5 6 7 8 9 10 11 12 13 14 California 43 Connecticut Delaware 54 Hawaii Idaho Montana 48 Nevada New Hampshire North Dakota Oregon South Dakota 51 Vermont Washington West Virginia 48 3 4 5 6 7 8 Ave 44 54 56 49 56 60 48 59 52 48 55 53 48 55 54 56 48 49 52 52 51 50.0 50 50 51 52 53 51.2 54 48 57 51 60 46 57 48 59 47 60 36 46.7 55.8 54.8 57.8 48.0 Mathematics 1 2 3 4 5 6 7 8 9 10 11 12 13 14 California Connecticut 53 Delaware Hawaii Idaho Montana 49 Nevada New Hampshire North Dakota Oregon South Dakota 54 Vermont Washington 59 West Virginia 50 46 48 55 38 41 51 33 41 41 35 42 37 36 40 40 44 37 39 41 36 41.0 49 38 41 43 42 44.7 55 40 49 33 48 29 50 30 48 27 51.5 34.8 38 Smarter Balanced 2015-16 Gain Scores [Level 3 & Above Percents] 3 37.3 47.5 43.7 2015 2016 2015 2016 State 1 California 2 Connecticut 3 Delaware ELA 42.3 Math Tot Ave ELA Math Math Gain Gain 34.2 37.3 4.4 xx** 55.8 40.3 44.2 xx** 3.9 xx** 40.7 43.7 3.1 3.0 3.1 --- --- --- --- --- --- 51.7 54.8 Hawaii 47.7 42.2 5 Idaho 49.7 40.8 6 Montana* ------ 7 Nevada* ------ 8 New Hampshire 58.7 47.7 9 North Dakota* ------ ------ 10 Oregon 53.8 43.5 11 South Dakota 47.5 12 Vermont 53.7 13 Washington 55.5 50.0 ------ 41.0 ------ 51.2 41.2 44.5 3.1 Gain 46.7 4 14 West Virginia ELA 3.8 --- --- --- 3.7 3.3 3.5 43.2 57.8 49.8 51.5 45.5 48.0 30.8 2.3 34.8 1.7 2.5 2.0 4.0 3.3 Notes: All averages and gains are based on Grade 3-8 data only. Not all Smarter Balanced states uniformly administer Smarter Balanced tests for High Schools, and hence any HS results are not included. * Montana, Nevada, and North Dakota participated in Smarter Balanced testing in 2015, but all three states ran into technology difficulties that prevented generation of representative scores for the entire state. No scores are recorded for 2015 for these states, which prevents calculation of 2016 gain scores. ** Connecticut indicated they discontinued the PT portion of ELA tests in 2016, and recalculated 2015 data to reflect the CAT-only portion of the test. Connecticut reported an ELA gain score of 3.3 points using the recalculated 2015 scores, but it is unclear whether the Connecticut ELA gain calculation is comparable to gain calculations from other SBAC states so it is not recorded for now. 4 PARCC 2016 State-by-State Comparisons [Level 4 & Above Percents] English Language Arts Grade 1. 2. 3. 4. 5. 6. 7. Colorado 37 Dist Columbia Illinois Maryland New Jersey 48 New Mexico 24 Rhode Island 3 4 5 6 7 8 Ave 44 41 38 41 42 40.5 53 25 53 25 52 24 56 23 55 26 52.8 24.5 Mathematics Grade 1. 2. 3. 4. 5. 6. 7. Colorado 39 Dist Columbia Illinois Maryland New Jersey 52 New Mexico 30 Rhode Island 3 4 5 6 7 33 34 31 26 xx 32.6 47 23 47 25 43 20 39 17 xx xx 45.6 23.0 5 8 Ave PARCC 2015-16 Gain Scores [Level 4 & Above Percents] 2015 2016 2015 2016 State 1. 2. 3. 4. 5. 6. 7. Colorado Dist Columbia Illinois Maryland New Jersey New Mexico Rhode Island ELA 40.3 36.0 38.8 50.0 23.2 36.8 40.5 26.2 52.8 24.5 ELA Math ELA Math Math Gain 31.2 28.8 31.8 41.0 21.4 28.2 Tot Ave Gain Gain 32.6 25.3 0.2 1.4 0.8 45.6 23.0 2.8 1.3 4.6 1.6 3.7 1.5 Notes: All averages and gains reflect grades 3-8 only, and the xx under grade 8 means a state does not administer the Grade 8 PARCC test to all Grade 8 students. 6