Summer Scoring Training I for California Educators: Smarter Balapged I. I. English Language Arts/ Con?dential and Proprietary. Copyright 0 2015 Educatlonal Testing Service. All rights reserved. Acknowledgment 99-{Tasks and materials used in this I training are from the WestEd and Scale ?Building Educator Assessment iteracy? (BEAL) project. (Ego, Con?dential and Proprietary. Copyright 2015 Educational Tasting Sendce. All rights reserved, 7/3 1/2015 @awemm new 0! Student Performance and Progress Schedule - 8:30?9:30 am. Introductions and Training - 9:30?10:30 Performance Task #1 10:30?10:45 Break 10:45?Noon Performance Task #1 continued - Noon?1:00 pm. Lunch Performance Task #2 -.. 1:00?2:30 Break Performance Task#2 continued Additional tasks/rubrics Con?den?el and Prupnenry. Copyright 0 2015 Educational Testing Service. All rights reserved. 3 Amman: a Student Mom-nu and was. Agenda - Housekeeping Introductions Purpose of training -- Handscoring as professional development . How Smarter Balanced performance tasks (PTs) are i scored exercises Con?dential and Proprietary. Copyright 2015 Educah'unel Tesung Service. All rial-n: reserved. 4 7/31/2015 I cum: Assessment 0! 3mm Performance and Progress Introductions ETS Trainers - Donna Wolak, Assessment Specialist, K-12 Assessments - Lissa Cone, Assessment Specialist, K-12 Assessments 1 Ralph Morris, Assessment Director, K-12 Assessments Corcoran, Assessment Specialist, Higher Education Assessments instinaj?iobo, Assessment Specialist, Higher Education .. Con?dential and Proprietary. Copyright 0 2015 Educational Tesu'ng Service. All rights reserved. 5 Purposes and Objectives - Learn how the summative assessments are scored - Practice using standard handscoring tools and processes - Experience methods of applying standardized criteria to - the evaluation of student work Use scoring information to improve instruction in the ?rsroom Con?dential and Proprietary. Copyright 2015 Edurzt'ronal Tsung Service. All rights reserved. 6 7/31/2015 Writing ltem Types: Performance Tasks - Smarter Balanced PT item types Full writes: Narrative, Opinion/Argumentative, Informative/Explanatory Rese-arCh short?text items Focus of training: Qifarrative full-write (Grade 4) short-text (Grade 11) Baton full?write (Grade 11) full-write (Grade 3) Con?dential and hprletary. Cooyright 2015 Educational Tsting Service. All rights reserved. 7 seam-?Momma ma nguss How Does Training Benefit Educators? - Enhanced assessment literacy Deeper understanding of Smarter Balanced assessment claims and targets Better understanding of handscoring procedures Opportunity to take a closer look at rubrics range of score points 'g?o-nal Improvement ?sfor imp-roving instruction and evaluation em work in the classroom (Er/a - Opportunity to see examples of student work at - Con?dential and Proprietary. Copyright 2015 Educational Testing Service. All rights reserved. 8 7/3 1/2015 a 1 California ?seamen: of Student Perform and Progress Individual and Group Work - Opportunity for you to review student sample responses, apply rubrics, and score student samples Opportunity to share and discuss your scores, insights, and ideas with other educators (group) Con?dential and Prop?enry. Copyright 2015 Eduational Testing Service. All rights reserved. I Columbia Assessment arm 9mm and Progress Standard Handscoring Materials - Provided by Smarter Balanced - the same across all vendors - Ensure raters have the tools to apply the scoring criteria consistently - Include the following: I - PT sets: stimuli and prompts ,Scoring rubrics and exemplars -'Samples: responses with scores and annotations ECticei?gset: sample responses for you to score @7 Con?dential and Proprietary. Copyright 0 2015 Enucau?nnal Tsting Service. All rights reserved, 7/31/2015 Cam: 155an Security Consu??t?immwar Interim Materials: Materials Printed on White Paper This training relies on Smarter Balanced Interim Assessment system scoring guides and sample student responses. While these materials are not secure, they are not public. All participants are expected to maintain the integrity of the materials. Interim assessment materials include items, scoring guides, and student responses. These materials should mt be: QShared with students, parents, or the public gduced in any form in any public place, including online Con?dential and Proprietary. Copyright 0 2015 Educational Testing Service. All rights rmrvad. What Can Be Reproduced or Shared Verbally? - All material printed on colored paper: The PowerPoint Individual score sheets - The Handscoring Process .All material printed on white paper inn-rot leave the room and will be Con?dential and Proprietary. Copyright 0 2015 Eduutional Testing Service. All rights reserved. 7/31/2015 . SL?/n??tTVg?biTnmg?mTS?Scoring by Trained Raters Training?Summative scoring training is similar to this training. Certification?Prospective raters must take and pass a certi?cation test on a set of prescored responses. Calibration?Before raters start scoring, they must take and pass calibration set to demonstrate that they are ready to score ?Double Scoring?10% of responses are double scored to verify hat"raters are scoring consistently with the rubric. ?ack'5Rating?Scoring Leaders ?read behind? raters and score nSes raters have scored to provide another check. Listening. Lauri-ling. Learn-15' 13 (E737. Con?dential and Proprietary. Copyright 0 2015 Educational Testing Service. All rights reservod. i at 8am Perla-1m. and Progress 'Summative Scoring (cont) - Scoring Leaders: Provide feedback to raters for additional content support; help prevent score drift; serve to maintain scoring pace. .5 Mentor struggling raters and/or direct them to - sample sets for retraining. Con?denull and Proprietary. Copyright 6 2015 Educational Tasting Service. All rights reserved, 7/31/2015 I Scoring Terms - Stimuli: Includes passages, prompts, or short scenarios. - Source Material: Used to extract evidence in support of a claim or opinion or to provide an experience for a student. - Prompts: The questions to which students are responding, also called ?items.? - Rubric: A standardized approach to score student responses. Some rubrics are item-speci?c. Exemplar: An example of a correct answer. Samples: Student responses that are prescored and tated. Annotations are often just as important and "Mfui as the rubric. Con?dential and Proprietary. Copyright 2015 Educational Testing Service. All rights reservnd. 15 Sample Annotations - A-n annotation is the rationale given for why a response receives full, partial, or no credit. - The annotations written for samples provide further insight into how student responses were scored. (59, Con?dential and Proprietary. Copyright 2015 Educational Testing Service. All righls rservud. 15 7/31/2015 7/3 1/2015 I Cameo-Ive Assessment 0! Student Performence and Progress Guidelines for Group Participation - Create a collegial and productive atmosphere. - Attend closely to the materials provided, and encourage the engagement of others. working with interim tasks and responses, remember your responsibility to protect the "ntegrit'y of these resources. Con?dential and Proprietary. Copyright 0 2015 Educational Testing Service. All rights meme. 1".ll a 4 California Museum: 01 Student ?ufon'nance and Progress Appoint a Table Spokesperson Responsibilities: - Keep the group on track and focused. - Ensure that all voices are heard. - Help bring the group to consensus. Collect scores from everyone at the table and share with trainer and larger group. -- - . . (as) Con?dentjal and Proprietary. Copyright 6 2015 Eduutlcnal Testing Service. All rights reserved. 13 Group Scoring Process 1. Read stimulus and sources. 2. Read the prompt/item and rubric. 3. Individually read and score practice papers. Remember to use the rubric and samples when you assign scores. Chart your scores. iscuss with your group and try to come to gasps on the scores. Con?dential Il'ld Pmpriuary. Copyright 0 2015 Educational Tsung Sonics. All rights ISBNBU. How Do You Chart Your Scores? Ei' Con?dential and Proprietary. Copyright 2015 Educational Service. All rights m. 7/3 1/2015 10 7/31/2015 @aka?o ?smmoisrm Performance andProgress Group Discussion and Consensus - Record all individual scores. .-. Members with discrepant scores present rationale for scores to citing evidence from rubric and student response. - {Other group members can provide evidence/support for scores . based on rubric. Try to reach consensus on a single score for each practice response- imber?consensus may not be 100% agreement! it?s okay to disagree. Con?dential and Proprimry. Copyright 0 2015 Educational Testing Service. All rights reserved. 21 i annulment ?Student Penman and Progress Write Your Scores on Sticky Notes "Grll Research, Practice 1? 2 Con?dential and Proprietary. Copyright 0 2015 Edmond Testing Service. All rights reserved. 22 11 7/31/2015 Smarter Balanced Scores and Annotations - Spokespersons share scores and rationales from each table. - Master scores and rationales provided. - Questions and group discussion About the scoring process About the rubric or samples About the scores (59 Con?dential and Proprietary. Copyright 0 2015 Educa?nnal Tsung Service. All righls reserved. 23 ELA Scoring Demonstration with Hands-0n Practice - Performance Task Narrative Full-Write item, 10 pts. - Performance Task Research short?text item, 2 pts. @a Con?dential Ind Proprietary. Copyright 0 2015 Edua?onal Testing Service. All rights reserved. 24 12 r; California Assessment or Student Perturmance and Wu ELA Performance Task #1 Grade 4?Narrative Full Write . Objectives .4 Become familiar with the Smarter Balanced full-write :rubrics for Narrative writing and with scoring . materials and procedures. Understand the rubric score levels and how to apply rubrics to score student writing samples. hderstand the demands of the Narrative writing you learn- to reflect on your instruction and - Con?dential and Pmpnetary. Copyright 0 2015 Eduutimal Testing Servicn. All rights reserved. Gallium? Assessment of am Perimnce and Progtess ELA Performance Task #1 Grade 4 Full Write Claim #2 Students can produce effective writing for a range of purposes and audiences. - Grade-appropriate stimulus and source material -- to use as support Writing- task with a clearly stated writing topic, uestion, problem, etc., audience, purpose, and (rubrics and exemplarresponse) Con?dential and Proprietary. Copyright 2015 Edumuoml Testing Service. Ali rights reserved. 7/3 1/2015 13 7/31/2015 ELA Performance Task #1 Grade 4 Narrative - Read the sources and assignment. - Review rubric. - Review sample papers (exemplars). Listening. Learning. Leading.? Con?dential and Proprietary. Copyright 9 2015 Educational Testing Service. All rights reserved. 27 ELA Performance Task #1 Grade 4?Sources - Grade-Appropriate Source Material for Support Handouts: Three reading passages Con?dential and Proprietary. Copyright 0 2015 Educationll Testing Service. All rights reserved. 23 14 7/31/2015 a California assessment of Student Performance and Progress ELA Performance Task #1 Grade 4?Mode of Writing Narrative The students will use information in the sources to write a story or capture the essence of an experience. Con?dential Ind Proprietary. Copyright 2015 Educational Testing Service. All rights reserved. 29 i Calm: Assam of 32m ?ufm on!) Progress ELA Performance Task #1 Grade 4?Audience The designated audience will be appropriate for the grade level, purpose, and situation . for the task. for younger students, __-familiar audiences such as parents, - and principals) (53) Con?dential and Proprietary. Copyright 0 2015 Educational Testing Service. All n'ghs reserved. 30 15 ELA Performance Task #1 Grade 4?Task Assignment?see Handout Writing task with a clearly stated audience, purpose, and form Con?dential and Proprietary. Copyright 0 2015 Educational Tating Service. All rights reserved. 31 ELA Performance Task #1 Grade 4?Rubrics Full Writes Scoring Guides for 10?point items: 3 Traits (scoring criteria) 1. Organization/Purpose 4 points 2.. Development] Elaboration Evidence] Elaboration 4 points Con?dential and Proprietary. Copyright 2015 Edua?unal Testing Service. All rights reserved. 7/3 1/2015 16 7/31/2015 ?7 I California Msmmemoismdem FEW Scoring Guidelines - Each dimension on the scoring guide is characterized by a variety of descriptors that -_identify the qualities expected in a response that receives that score. Because there is a limited number of points on the scale each score point represents a ange of performance. expected that all the descriptors for a droppint will apply equally to a response. (?39 Con?dential and Proprietary. Copyright 0 2015 Edumtional Testing Service. All rights resumed. 33 1 633601116: Assam-rem of Student Performmce and Progress Grades 3?5 Narrative Scoring Guide?4 Points - Organization] Purpose Organization fully sustained Focus is clear and maintained throughout Effective plot Sense of unity/completeness (Level 4) Fetus adequate and generally maintained (Level 3) magnet sustained/ uneven focus (Level 2) "no focus'(Level 1) as. Confidential and ngrie?ary. Copyright 6 2015 Educational Testing Service. All righs rmrveu. 17 4 mm: assessment of Student hammer-cc am Prague: - Development] Elaboration Effective elaboration Relevant details, dialogue, and/or description Grades 3?5 Narrative Scoring Guide?4 Points elaboration and development 3) eiaboratioen and few or no details (Level 1) Con?dential and Proprietary. Copyright 6 2015 Educational Tsing Service. All right reserved. Thorough elaboration; effective development veg, cursory elaboration and development (Level 2) Lamina. mum. Lending.? 35 i Elaboration for Narrative Performance Task Full Writes: Definition from Smarter Balanced ?Thorough elaboration is guided by the purpose of the story, the point of view, and the needs of the audience.? Definition: The process of developing the character(s), events, and ideas he careful selection of pertinent, gdetails and the smooth integration Con?dential and Proprietary. Copyright 2015 Eduatinnal Testing Service. All rights merved. 7/31/2015 18 a wma Assessment 013mm Performance and Progress Elaboration (Continued) - Three Ds in Narrative Writing Details Description Dialogue 'LooSe interpretation of sources Con?dential and Proprietary. Copyright 0 2015 Eduatianal Testing Service. All rights reserved. 37 Cation?: Assessment at Student Performance and Progress Grades 3-5 Narrative Scoring Guide?2 Points - Conventions Sentence formation Punctuation Capitalization Grammar usage Spelling Score levels (2, 1, O) Adequate command command nocommand (ea. Con?dential and Proprietary. Copyright 2015 Eduatimal Tasting Servicu. All righs reserved. 33 7/3 1/2015 19 Full Writes Conventions Charts Grades 3?8 and High School - Spelling - Capitalization - Punctuation - Grammar usage a entence completeness Con?dential and Pmprioury. Copyright 0 2015 Educational Testing Service. All righs reserved. i m- Autumn-rt at Pm: and Progress Samples - Read samples and annotations (as. Con?dnnual and Proprietary. Copyright 2015 Eduut'ronal Tsting Servine. All rights reserved. - Discuss samples and annotations 7/31/2015 20 Can?dan?a! and Copyright 0 2015 Educational Tasting Sunrise. NJ righs reserved. Begin Group Work for Full Writes MW Leading! 41 Reconvene Large Group to Discuss Full Writes Scores Con?denull and Proprietary, Wm 2015 Educatioan rating Service. All reserved. 7/31/2015 21 :3 Comma Assessmem of Student Wis-ace and Progress ELA Performance Task #2 Grade 11?Research Task Objectives - Become familiar with the Smarter Balanced Research Task: constructed-response items, materials, and scoring procedures - Understand the rubric score levels and how to apply the rubrics to student writing samples ?derstand the demands of the Research short structed-response task at you learn to reflect on your instruction assessments Con?dential and Proprietary. Copyright 0 2015 Eduutlmal Twang Service. All reserved. 43 5 him. Assessment of Student Moan-ace and #109193: ELA Performance Task #2 Grade 11?Research Task (continued) Research Item?2 points Claim #4 can engage in research/inquiry to investigate topics, and to analyze, integrate, and present information. 3: Evaluate Information/Sources tie-appropriate source material to use as ?eft task using the sources. (Scoring Guide/Rubric) Con?dential and Proprietary, Copyright 0 2015 Educational Tsung Service. All righls reserved. 7/31/2015 22 ?i I cum: Assessment or Student Performume and Frogreu ELA Performance Task #2 Grade 11?Sources, Assignment, Rubric, Exemplars - Read the sources and assignment. Review rubric. euieyysample papers (exemplars). Con?dentiai and Proprietary. Cnpy?ght 0 2015 Educational Testing Service. All rights reserved. 45 7 5 Cam Assessment at Student 981m and Progress ELA Performance Task #2 Grade 11?Research Task (continued) - Grade-Appropriate Source Material for Support Handouts: Four reading passages . -: .1 5.3.1@337. Con?dentia! and Proprietary. Copyright 2015 Eduationai Testing Service. All rights reserved. 45 7/31/2015 23 ELA Performance Task #2 Grade 11?Research Task (continued) Writing task with a clearly stated writing question that asks the students to justify their answer and cite the source. Assignment?see Handout Con?dential and Proprietary. Copyright 0 2015 Education-I Testing Service. All rights reserved. ?aming. Learning. my 47 a MW at swam Penmnce and Progress Grade 11 Research Rubric?2 pts. Rubric?see Handout C519, Con?dential and Proprietary. Copyright 2015 Eduutional Testing Service. All rights reserved. 7/31/2015 24 Samples Read samples and annotations Discuss samples and annotations Con?dential and Proprietary. Copyright 2015 Eduational Tus?ng Servicl. All rights reserved. 49 Begin Group Work on Research Item Con?umtiai and Proprietary. Copyright 0 2015 Education-I Tasting Sun/ice. All rights reserved. 50 7/31/2015 25 7/31/2015 a Ca?o?ua Assessment oi smoent Performmoe and Progress Reconvene Large Group to Discuss Practice Scores Con?dential and Proprietary. Copyright 0 2015 Educational Tating Service. All rights reserved. 51 Additional Task/Rubric: Grade 11 Explanatory - Grade 11?FuII-Write Item Sources (same as for Research short-text item) Assignment - Annotated samples :brics?each response scored for three Con?dential and Proprimry. Copyright 0 2015 Educational Testing Service. Ali rights reserved. 52 26 I Cam: Assessment of 81m Perform: and Progress Additional Task/Rubric: Grade 3 Opinion Grade 3?Full-Write Item: Sources ?'Assignment Annotated samples response scored for three a'nization/purpose ?3 1 . . I dence/elaboratlon .. (if? Con?dcntial and Proprietary. Copyright 2015 Educational Testing Service. All rights reserved. 53 Calm: Assessment of Sam and Purges; Closing Activity - Reflect on today?s session, and discuss with your table group: Strategies you learned that are relevant to classroom instruction ?Ideas you will take back to the "classroom Other thoughts about scoring training "-prOfessional development Con?dential and Proprietary. Copyright 6 2015 Edu?ticnll Testing Service. All rights reserved. 7/31/2015 27 7/3 1/2015 Questions? Comments? Con?dential and Proprietary. Copyright 0 2015 Educational Testing Servica. All rights raservod. 55 28 Full Write Item Sco. Sheet Shore [Stere Score? Stare}; Seore Score Score; Dev/j." tony; Org/ Conv Org/ Devl- Conv gElab-f?: Purp 'Elah Purp Elab . - voufriiz- .vour Group Group-4: group. SBAC SBAC Practice-1 Pra?tice 2 '5 Practice 3 fPractim 4. .. Pra??ti??S - Practice 6 __Pra?tice 7 Prattice 8 9 Practice '10? Research Item Score Sheet Group Your Consensus SBAC Score Score Score Comments Practice 1 Practice 2 Practice 3 Practice 4 Practice 5 Practice 6 Practice 7 Practice 8 Practice 9 Practice 10