Multiple Measures Assessment Project (MMAP) April, 2016 http://www.rpgroup.org/projects/multiple-measures-assessment-project MMAP Project Overview • Collaborative effort of CCCCO Common Assessment Initiative (CAI) designed to develop, pilot, and assess implementation of placement tool using multiple measures through joint efforts of Cal-PASS Plus, RP Group and now 45 CCCs • Develop multiple measures models for English, Mathematics, ESL and Reading – Eventually Chemistry and Biology • Identify, analyze and validate multiple measures data, including high school transcript data, non cognitive variable data, and selfreported HS transcript data • Engage pilot colleges to conduct local replications, test models and pilot their use in placement, and provide feedback • bit.ly/MMAP2015 Why Multiple Measures? • Historically, tests alone have had relatively low predictive validity • Multiple measures • provide a more complete picture of student ability • provide a way to increase the accuracy of placement, particularly reducing underplacement • http://bit.ly/CCRCPlacementAccuracy • are required by law • MM is supported by statewide senate Data Set for Models • CCC students enrolled in an English, Math, Reading or ESL class with matching high school data in CalPASS • ~1 M cases for Math & English; ~200k for Reading & ESL • Bulk of first CCC enrollments from 2008 through 2014 • Data files include: • High school course grades, unweighted GPA, course type • Assessment data, where avail. (ACCUPLACER, CST, EAP) • CCC data (course grades, course level) • Other derived info. (e.g., delay, CCC math class type) •Rules were developed with the subset of students who had four years of high school data (about 25% of total sample) Variables Explored in the Models • High School Cumulative GPA (primary predictor) • Grades in high school courses • CST scores • Advanced Placement course taking • Taking higher level courses (math) • Delay between HS and CCC (math) • HS English types (expository, remedial, ESL) • HS Math level (El Alg, Int Alg, Pre-Calc) Transfer Level Rule Sets Transfer Level Course Direct Matriculant Non-Direct Matriculant College Algebra (STEM) HS 11 GPA >=3.2 OR HS 12 GPA >=3.2 OR N=216,420 HS 11 GPA >=2.9 AND Pre- HS 12 GPA >=3.0 AND PreCalculus C (or better) Calculus or Statistics (C or better) Statistics (Non STEM) HS 11 GPA >=3.0 OR HS 12 GPA >=3.0 OR N=216,420 HS 11 GPA >=2.3 AND Pre- HS 12 GPA >=2.6 AND PreCalculus C (or better) Calculus C (or better) English N=347,332 HS 11 GPA >=2.6 HS 12 GPA >=2.6 theRPgI?oup the Research Planning Group A em 0f 3133 for California Community Colleges Potential Statewide Transfer Level Placement 100% - lCurrent lDisjunctive 61% 60% 50% 409English Math {n=103,510) {n=143,253) theRPgr?oup the Research Planning Group A em 0f ata for California Community Colleges Projected impact on course success rates (completion of course with or better) 100% - 909' Historic success rate I Projected success rate 30Tra nsfer-Ievel Math Transfer-Level English Successful completion of transfer level course Transfer Level English Placement Current Disjunctive MM 100% 90% 80% 73% 74% 70% 60% 50% 40% 30% 53% 51% 41% 40% 24% 30% 20% 10% 0% Afr Am Latino Asian White Transfer Level Math Placement Current Disjunctive MM 100% 90% 80% 70% 65% 60% 53% 50% 41% 40% 32% 30% 20% 51% 22% 15% 21% 10% 0% Afr Am Latino Asian White One Level Below Rule Sets One Level Below Course Reading N=21,450 Direct Matriculant HS 11 GPA >=2.2 Non-Direct Matriculant HS 12 GPA >=2.4 AND 12th Grade English C (or better) HS 12 GPA >= 2.4 AND CST English >= 322 HS 12 Grade GPA >=1.7 AND 12th Grade English C+ (or better) ESL N=4,901 ● ● HS 11 GPA >=2.7 HS 12 GPA >=2.6 The vast majority of ELL/ELD HS students (~85%) who enter CC begin directly in mainstream English coursework. Other major populations of ESL students (e.g., international students, migrants, older immigrants) will not have US high school transcripts and so other multiple measures, such as essays, must be used with those groups. Common Concerns about MMAP • • • • • • • High school grades are inflated Students placed via MMs will not be successful Our courses will have lower pass rates Our test is different Students would be better off in remedial coursework Students will only get a “C” in transfer-level work Students who get a “C” in transfer-level won’t be able to transfer • High school GPA is only good for recent graduates Evidence for grade inflation low at best • Little evidence for grade inflation over last decade • Earlier observations of grade inflation may have been partly artifactual –adjustments to GPA for AP/IB/Honors • Zhang & Sanchez, 2014: http://bit.ly/ACTGradeInfl ation 'l the Research Planning Group for California Community Colleges Sierra College: Higher success rates for students placed via MMs T533 T396- TEE-E3- ?49-6- TIE-E3- T096- 53%- 55%- 54%- 52%- 50% A System of Data Fall 1-311 Fall E?ll Fall 2?13 F14 H5 Data Accuplacer .r/f Fl theR h&Pianning Group for California Community Colleges A System of Data Our tests are different IVC Math Test CTEP Grammar .10 .16 CTEP Reading .06 .09 CTEP Syntax .03 .12 Math Assessment Test .21 .29 11":h Grade GPA - English .37 .38 111th Grade GPA Math .38 .39 Our tests are different - Compass Course Compass Test Compass HSGPA HSGPA + Compass English 1 Writing Skills .31 .57 .62 Arithmetic Pre-Algebra .57 .34 .66 Algebra Pre-Algebra .36 .65 .80 Intermediate Algebra Algebra .47 .66 .84 College Algebra Algebra .41 .76 .88 College Algebra College Algebra .51 .76 .94 http://bit.ly/COMPASSValidation (Table 4 - Median Logistic R) Our tests are different - Accuplacer 11th Grade GPA Accuplacer Transfer .27 .10 1 level below .24 .12 2 levels below .25 .12 3 levels below .18 .12 4 levels below .21 .07 English Community college grade of C or better 11th Grade GPA Accuplacer Transfer - STEM .24 .19 Transfer – Stats .31 .16 Transfer – GEM .26 .09 1 level below .28 .21 2 levels below .26 .11 3 levels below .23 .11 4 levels below .19 .05 Math Remedial courses are better for students source: Hayward and Willett (2014) Belfield & Crosta (2012): Given the frequency of underplacement, the poor predictive validity of assessment tests and the lack of positive outcomes for student placed into remediation, it would be statistically defensible and really quite reasonable to just put all students into transfer-level work. Will only get a “C” in transfer course Internal Node/ Split Success rate Percent of sample in leaf Leaf Cal-PAsspli?? the Research Planning Group A SyStem Of Data for California Community Colleges Distribution of Statistics Node 8 (Circled) 35% 309Students who get a “C” in transfer-level won’t be able to transfer Irvine Valley College, first course enrolled in, Spring 2000 to Fall 2011 who took an English course. N= 28,279, transfer within 4 years. Correlation between Predictor and 1st CC English Grade High school GPA is only good for recent graduates Semesters of Delay (approx. 6 months each) Correlation between Predictor and 1st CC Math Grade High school GPA is only good for recent graduates Semesters of Delay (approx. 6 months each) Multiple Measures Assessment Project (MMAP) Cañada Pilot Study April 2016 4/15/2016 CAÑADA COLLEGE 24 Progression from Placement to Enrollment English: 2014 vs. 2015 300 66% 60% 250 # of Students 70% 54% 50% 200 40% 150 259 100 30% 20% 170 123 50 10% 66 0 0% 2014 College Level English 2015 College Level English 2014: COMPASS Placed 4/15/2016 2015: MMAP + COMPASS Enrolled CAÑADA COLLEGE Progression 27 # of Students Progression from Placement to Enrollment Math: 2014 vs. 2015 300 70% 250 60% 47% 200 50% 37% 150 40% 30% 245 100 191 50 20% 116 70 10% 0 0% 2014 College Level Math 2015 College Level Math 2014: COMPASS Placed 4/15/2016 2015: MMAP + COMPASS Enrolled CAÑADA COLLEGE Progression 28 English 100 Course Outcomes by Grade Distribution Students placed by MAP GRADES :3 if; NUMBER 100% Dvcra" Course Success 4/15/2016 Students placed by Compass GRADES NUHHEH at :31- SI 56% m' B- 9 16% {211TUTAL 55 Dvcrall Cc: ursc Success CANADA COLLEGE Math Course Outcomes by Course Number 4/15/2016 CANADA COLLEGE 133 Wplamdby?umpass Cm: Numb-er FALL 1015 FAIL 3115 FALL 1015 FALL EDIE Hunt-er Sum '55- Numb: Slim 125 3 I an 1' SHIE- 16 31 9'6 EUU 3] ?ight 59 56% 112 I 4 100% 22 5 2 9 ETQE. 2-1 I 5 ?ll'?i 509-2. 2 3 34 75-91:. 2 52 I 10067% MMAP Students Feedback (Grade D, F, or W) What was your experience? Do you think you were placed into the right level? Did something else interfere with your success in the College-level class? •Right level, people are easy to get along with, materials are good •It is the right level. She didn't like the teacher though. She thinks the teacher is disorganized •Right level •It is the right level. The balance between classes can be a problem. Need more time because of the amount of class materials •Right level for him. Nothing really interfere •Yes. everything is good •Yes. nothing is intervening •Right level, not enough time for the assignment 4/15/2016 CAÑADA COLLEGE 31 Contacts Loris Fagioli The RP Group lfagioli@ivc.edu John Hetts Educational Results Partnership jhetts@edresults.org Mallory Newell The RP Group newellmallory@deanza.edu Ken Sorey Educational Results Partnership ken@edresults.org Terrence Willett The RP Group twillett@rpgroup.org Daniel Lamoree Educational Results Partnership dlamoree@edresults.org Craig Hayward The RP Group chayward@rpgroup.org Peter Bahr University of Michigan prbahr@umich.edu