TRANSCRIPT OF PROCEEDINGSTESTING INTEGRITY SYMPOSIUM02 . 28 . 2012EUNICE GREER: GOOD MORNING. MY NAME IS EUNICE GREER. ON BEHALF OF THEDEPARTMENT OF EDUCATION, I HAVE THE PRIVILEGE OF WELCOMING JOHNEASTON, THE DIRECTOR OF IES, AND INVITING HIM TO KICK US OFF THISMORNING. JOHN?JOHN EASTON: THANKS VERY MUCH, EUNICE. AS EUNICE JUST SAID, I AM JOHNEASTON, AND I'M THE DIRECTOR OF THE INSTITUTE OF EDUCATIONSCIENCES, AND I WANT TO WELCOME YOU ALL AND ESPECIALLY THANK YOUFOR COMING TO PARTICIPATE IN TODAY'S DISCUSSION. I KNOW MANY OF YOUHAVE TRAVELED TO BE WITH US, AND WE APPRECIATE THAT.THE NATIONAL CENTER FOR EDUCATION STATISTICS IS SPONSORING TODAY'SSERIES OF PANEL DISCUSSIONS THAT FOCUS ON MAINTAINING THEINTEGRITY OF ASSESSING STUDENT ACHIEVEMENT. WE AT THE UNITEDSTATES DEPARTMENT OF EDUCATION ARE SEEKING TO COLLECT AND SHAREINFORMATION ABOUT BEST PRACTICES THAT HAVE BEEN USED TO PREVENT,DETECT, AND RESPOND TO IRREGULARITIES IN ACADEMIC TESTING.EDUCATORS, PARENTS, AND THE PUBLIC IN GENERAL RELY ON ACCURATE,RELIABLE, AND TIMELY INFORMATION ON STUDENT ACADEMIC PERFORMANCETO IMPROVE INSTRUCTION AND HELP ALL STUDENTS REACH AND MAINTAINHIGH LEVELS OF ACHIEVEMENT. INDEED, THE AVAILABILITY OF VALID,RELIABLE, AND TIMELY DATA ON STUDENT PERFORMANCE IS ESSENTIAL FORINFORMING INSTRUCTION, IDENTIFYING PROFESSIONAL DEVELOPMENT NEEDS,2HELPING ENSURE MEANINGFUL ACCOUNTABILITY, AND IMPLEMENTINGEFFECTIVE EDUCATION REFORMS.AS A FEDERAL STATISTICAL AGENCY, THE NATIONAL CENTER FOR EDUCATIONSTATISTICS IS ESPECIALLY CONCERNED WITH ISSUES OF DATA QUALITY ANDACCURACY, AS IS THE ENTIRE INSTITUTE OF EDUCATION SCIENCES AND THEDEPARTMENT OF EDUCATION. SO THIS SYMPOSIUM IS ONE OF SEVERAL STEPSTHAT'S BEING TAKEN BY THE DEPARTMENT TO COLLECT INFORMATION ANDGATHER SUGGESTIONS TO ASSIST STATE EDUCATION AGENCIES, LOCALEDUCATION AGENCIES, AND TESTING INTEGRITY FOCUS ORGANIZATIONS THATSERVICE THEM. THOUGH THE DEPARTMENT IS NOT DEVELOPING TESTINGINTEGRITY REGULATIONS, IT ANTICIPATES MAKING USE OF THISINFORMATION TO FACILITATE FURTHER DIALOGUE AND TO HELP SEAS ANDLEAS IDENTIFY, SHARE, AND IMPLEMENT BEST PRACTICES FOR PREVENTING,DETECTING, AND INVESTIGATING IRREGULARITIES IN ACADEMIC TESTING.MANY STATES AND ORGANIZATIONS, SUCH AS THE COUNCIL OF CHIEF STATESCHOOL OFFICERS, HAVE DONE WORK TO IDENTIFY AND SHARE BESTPRACTICES AMONG STATES. THIS SYMPOSIUM IS DESIGNED TO BRINGTOGETHER EXPERTS WHO CAN ALSO SHARE INFORMATION ABOUT THE LATESTTHINKING ON ASSURING THE INTEGRITY OF OUR TESTING PROGRAMS. LASTMONTH, THE DEPARTMENT ISSUED A REQUEST FOR INFORMATION TO COLLECTINPUT FROM THE GENERAL PUBLIC ABOUT THE INTEGRITY OF ACADEMICTESTING.3THE DEPARTMENT POSED A SERIES OF QUESTIONS, TO WHICH WE INVITEDINTERESTED MEMBERS OF THE PUBLIC TO RESPOND. THIS SYMPOSIUM ISDESIGNED TO SHOWCASE EXTERNAL EXPERTS WHO CAN ENGAGE IN FURTHERDISCUSSION AND PROBE THESE ISSUES IN GREATER DEPTH. FOLLOWING THESYMPOSIUM, THE DEPARTMENT WILL PUBLISH A DOCUMENT SUMMARIZING THERECOMMENDATIONS THAT WERE DEVELOPED AS A RESULT OF THE REQUEST FORINFORMATION AND FROM THE SYMPOSIUM, AS WELL AS OTHER RESOURCESIDENTIFIED BY PANELISTS TODAY. I'D LIKE TO NOW TURN THIS OVER TOJACK BUCKLEY.JACK IS THE COMMISSIONER OF THE NATIONAL CENTER FOR EDUCATIONSTATISTICS. JACK IS ON LEAVE FROM HIS POSITION AS PROFESSOR OFAPPLIED STATISTICS AT NEW YORK UNIVERSITY. HE'S WELL KNOWN FOR HISRESEARCH ON SCHOOL CHOICE AND ON STATISTICAL METHODS FOR PUBLICPOLICY.THANK YOU, AND WELCOME AGAIN.JACK BUCKLEY: THANK YOU VERY MUCH, JOHN, AND THANK YOU ALL FORCOMING.WHETHER YOU'RE HERE IN THE ROOM OR PARTICIPATING VIA WEBCAST, WEAPPRECIATE YOUR ATTENTION TODAY, AND WE APPRECIATE THAT YOU'RE4JOINING US TO DISCUSS TESTING INTEGRITY AND TESTINGIRREGULARITIES.I WANT TO START BY DEFINING WHAT I MEAN BY THAT, ALTHOUGH I HAVE AFEELING THAT THE DEFINITION WILL BE A MOVING TARGET AS THE PANELSSPEAK AND AS THE DISCUSSION COMMENCES.FOR STARTERS, A TESTING IRREGULARITY INCLUDES ANY OCCURRENCE THATMAY INAPPROPRIATELY INFLUENCE A STUDENT'S PERFORMANCE ON ANACADEMIC ASSESSMENT, PROVIDE THE APPEARANCE OF IMPROPRIETY, OROTHERWISE CONSTITUTE A BREACH IN TEST SECURITY OR AN IMPROPERADMINISTRATION OF ACADEMIC TESTING.TO HELP FOCUS THE DISCUSSION TODAY, WE'VE TRIED TO ORGANIZE THEPANELS INTO 4 TOPICS--PREVENTING AND REDUCING TESTINGIRREGULARITIES IN ACADEMIC ASSESSMENTS, DETECTING AND ANALYZINGTESTING IRREGULARITIES, REVIEWING AND INVESTIGATING ALLEGEDTESTING IRREGULARITIES, AND, LOOKING TO THE FUTURE, HOW THESE 3ISSUES MIGHT CHANGE FOR ASSESSMENTS DELIVERED ONLINE AND BYCOMPUTER. FROM OUR OUTSTANDING SET OF PANELISTS TODAY, YOU CANEXPECT TO HEAR ABOUT BEST PRACTICES, SOLUTIONS, ADVICE, TECHNICALINFORMATION, LEGAL, REGULATORY, AND POLICY APPROACHES TO THEPROBLEM, AND OTHER LESSONS LEARNED RELATED TO THE PREVENTION,5DETECTION, AND INVESTIGATION OF ALLEGED OR ACTUAL TESTINGIRREGULARITIES.AS WE ALL KNOW, STATES EMPLOY A VARIETY OF SAFEGUARD PROCESSES ANDPROCEDURES TO ENSURE THAT ASSESSMENTS ARE ADMINISTERED UNIFORMLYAND SECURELY. THERE ARE SOME IMPORTANT COMMONALITIES THAT WE'VESEEN.ACROSS STATES, MANUALS USED BY TESTING ADMINISTRATORS AREDEVELOPED FOR THE ADMINISTRATION OF STATE ASSESSMENTS, AND STATESTRAIN ANNUALLY ON THEIR TEST SECURITY REQUIREMENTS, COVERING ARANGE OF TOPICS, INCLUDING SECURING TEST MATERIALS, HANDLING TESTMATERIALS THROUGHOUT THE ADMINISTRATION, AND SECURING MATERIALSPOST ADMINISTRATION. ANOTHER MAJOR COMPONENT OF THESE TRAININGSESSIONS INCLUDES HOW TO HANDLE ANY DEVIATION FROM SPECIFIEDSECURITY PROCEDURES.WE FOUND A MAJORITY OF STATES INCLUDE DETAILED INSTRUCTIONS TOADMINISTRATORS ON HOW TO REPORT ANY CONCERNS THAT MAY IMPACT THEVALIDITY OF STUDENT PERFORMANCE. IN A NUMBER OF STATES, STAFFINVOLVED IN THE ASSESSMENT ARE REQUIRED TO SIGN AN AFFIRMATION OFSECURITY, ENSURING THEIR UNDERSTANDING OF PROCESSES ANDPROCEDURES.6AT THE CONCLUSION OF THE ASSESSMENT, THERE ARE ALSO COMMONSAFEGUARDS EMPLOYED ACROSS THE STATES. THE MAJORITY OF THESE, ASMENTIONED ABOVE, ARE DIRECTED TOWARDS MAINTAINING SECURITY OF THETESTING ITEMS AND ENSURING THAT TEST SECURITY HAS NOT BEENCOMPROMISED. INCLUDED IN MANY OF THE TRAINING SESSIONS AND MANUALSARE PROCEDURES FOR SECURING AND SHIPPING MATERIALS TO TEST-SCORINGORGANIZATIONS THAT CONTRACT TO STATE DEPARTMENTS OF EDUCATION.AS JOHN SAID, EDUCATORS, PARENTS, AND THE PUBLIC IN GENERAL RELY ONACCURATE, RELIABLE, AND TIMELY INFORMATION ON STUDENT ACADEMICPERFORMANCE TO IMPROVE INSTRUCTION AND HELP ALL STUDENTS MAINTAINHIGH LEVELS OF ACHIEVEMENT. STATE EDUCATION AGENCIES AND LOCALEDUCATION AGENCIES WORK TO ENSURE THE INTEGRITY OF THE DATA THATTHEY USE TO MEASURE STUDENT ACHIEVEMENT AND ENSURE MEANINGFULEDUCATIONAL ACCOUNTABILITY.IN ORDER TO OBTAIN THESE DATA, STATES MUST ESTABLISH AND MAINTAINASSESSMENT SYSTEMS THAT ARE VALID, RELIABLE, AND CONSISTENT WITHNATIONALLY RECOGNIZED PROFESSIONAL AND TECHNICAL STANDARDS. EVENTHE SLIGHTEST APPEARANCE OF IMPROPRIETY IN THE TEST ADMINISTRATIONPROCESS CAN UNDERMINE STATE ACCOUNTABILITY SYSTEMS.AS WE ALL KNOW, TESTING IRREGULARITIES HAVE OCCURRED THROUGHOUTTHE UNITED STATES, IN URBAN, SUBURBAN, AND RURAL SETTINGS, LOW7INCOME AND AFFLUENT AREAS, AND IN CHARTER AND NON-CHARTER SCHOOLS.TESTING IRREGULARITIES ARE NOT NEW PHENOMENA AND CAN BE TRACED TOTHE BEGINNING OF STATE ASSESSMENT PROGRAMS IN THE MID 1800S.SOMETIMES, TESTING IRREGULARITIES HAVE INVOLVED LARGE-SCALE ANDCOORDINATED EFFORTS AFFECTING MULTIPLE SCHOOLS AND/OR DISTRICTSWITHIN A STATE, BUT OTHER TIMES, IRREGULARITIES HAVE BEEN ISOLATEDINCIDENTS IN SINGLE SCHOOLS. SOME OF THE ALLEGED TESTINGIMPROPRIETIES COULD BE DESCRIBED AS TOP-DOWN, INITIATED ORCONDONED BY A SCHOOL OR DISTRICT. IN OTHER CONTEXTS, CHEATING WASCLEARLY CONFINED TO CASES INVOLVING INDIVIDUAL TEACHERS ACTINGINDEPENDENTLY.IT'S IMPORTANT TO REMEMBER THAT USING ACCURATE METHODS TO DETECTAND VERIFY IRREGULARITIES IS CRITICAL TO PROTECTING THE INTEGRITYOF ASSESSMENTS. AS JOHN MENTIONED, THE PURPOSE OF THIS SYMPOSIUM ISTO SHARE BEST PRACTICES AND PROCEDURES, AS RECOMMENDED BY TESTINGEXPERTS AND EDUCATION PRACTITIONERS, TO IMPROVE THE ABILITY OFSTATE AND LOCAL EDUCATION AGENCIES TO PREVENT, DETECT, AND RESPONDTO TESTING IRREGULARITIES.WE HAVE ASSEMBLED 4 PANELS TODAY, MADE UP OF SOME OF THE LEADINGEXPERTS IN TESTING INTEGRITY, TO DISCUSS THESE ISSUES. EACH PANELWILL BEGIN WITH PRESENTATIONS FROM THE PANELISTS, FOLLOWED BY A8MODERATED DISCUSSION AMONG PANELISTS, AND WILL CONCLUDE WITHQUESTION AND ANSWER FROM OTHER PANELISTS, AS WELL AS THE AUDIENCEAND WEBCAST VIEWERS.AND BEFORE ANY OF OUR DISTINGUISHED PANELISTS SPEAK, I SHOULD SAYTHAT THE COMMENTS TODAY REFLECT THE EXPERTISE AND OPINIONS OF THESEPANELISTS, AND NOT THE OFFICIAL POSITION OF THE U.S. DEPARTMENT OFEDUCATION.THE FIRST 2 PANELS WILL BE FOLLOWED BY A ONE-HOUR BREAK FOR LUNCH.THERE WILL ALSO BE 2 PANELS AFTER LUNCH, WITH A 15-MINUTE BREAKBETWEEN THEM.NOW I'D LIKE TO INTRODUCE OUR FIRST SPEAKER OF THE DAY.KATHI KING IS THE MATHEMATICS DEPARTMENT CHAIR AND 12TH GRADEMATHEMATICS TEACHER AT MESSALONSKEE HIGH SCHOOL IN OAKLAND, MAINE.SHE HAS OVER 34 YEARS' TEACHING EXPERIENCE IN CALIFORNIA, NEW YORK,AND MAINE.MS. KING'S TEACHING RESPONSIBILITIES INCLUDE ADVANCED PLACEMENTCALCULUS AT THE AB AND BC LEVEL, AND AS DEPARTMENT CHAIR, SHE'SFOSTERED THE CREATION OF COURSES IN ROBOTICS AND PRE-ENGINEERING.SHE ALSO TEACHES STATISTICS AT THE COLLEGE LEVEL.9MS. KING IS A FORMER MEMBER OF THE NATIONAL ASSESSMENT GOVERNINGBOARD, WHICH SETS POLICY FOR THE NATIONAL ASSESSMENT OFEDUCATIONAL PROGRESS, OR NAEP. SHE RECEIVED A MASTER'S DEGREE INCOMPUTER TECHNOLOGY EDUCATION FROM THOMAS COLLEGE AND A MASTER'SDEGREE AND BACHELOR'S DEGREE IN MATHEMATICS FROM CALIFORNIA STATEUNIVERSITY, NORTHRIDGE.MS. KING.KATHI KING: GOOD MORNING. THE LENS I WOULD LIKE TO USE THIS MORNINGFOR MY COMMENTS IS THROUGH THE EYES OF A CLASSROOM TEACHER. I'VEBEEN TEACHING FOR CLOSE TO 40 YEARS, IN SCHOOLS OF EVERY TYPEIMAGINABLE, INCLUDING JUNGLE SCHOOLS IN THE U.S. PEACE CORPS,SMALL PRIVATE SCHOOLS, LARGE URBAN SCHOOLS IN THE STATES OFCALIFORNIA, NEW YORK, AND MAINE.OVER THESE 4 DECADES, I'VE WATCHED TESTING INTEGRITY TRANSFORMFROM HEAVY RELIANCE ON PERSONAL HONOR CODES TO STAYING ONE STEPAHEAD OF STUDENTS WHO WANT TO COMPROMISE THE PROCESS FOR PERSONALGAIN.CHEATING IS NOTHING NEW. IT'S ALWAYS BEEN WITH US. IN ORDER TOMAINTAIN TESTING INTEGRITY, I BELIEVE FOCUS SHOULD BE DIRECTED TO 310MAIN AREAS: DESIGN, VIGILANCE, AND CONSEQUENCES. THESE 3 AREAS OFFOCUS APPLY TO STUDENTS, EDUCATORS, ADMINISTRATORS, ANDEDUCATIONAL LEADERSHIP. THE VERY FACT THAT THIS SYMPOSIUM HAS BEENCALLED IS A REFLECTION OF A RESPONSIBLE COMMUNITY EAGER TOREEVALUATE AND TAKE APPROPRIATE ACTION TO IMPROVE THE TESTINGENVIRONMENT. DESIGN.YEARS AGO, AS I WAS PASSING BACK THE FIRST TEST IN AN AP CALCULUSCLASS, ONE OF THE STUDENTS RAISED HIS HAND AND HE COMPLAINED THATHE HAD THE EXACT SAME RESPONSES AS HIS NEIGHBOR, AND SHE GOT A 90AND HE GOT A 20. I THINK WE'VE ALL MAYBE BEEN THERE. IT SEEMS HE'DNEVER BEEN IN A MATH CLASS WHICH USED MULTIPLE VERSIONS OF A TEST.GOOD DESIGN DOESN'T COME CHEAP, WHETHER IT'S THE CLASSROOM, THESTATE, OR THE NATION. THE TIME TO CREATE MULTIPLE VERSIONS AND THETIME TAKEN IN CLASS TO ADDRESS TESTING INTEGRITY ARE VALUABLEMINUTES TAKEN AWAY FROM TEACHING AND LEARNING, BUT THEY'REABSOLUTELY NECESSARY. A STUDENT WHO PLANS ON CHEATING LEARNS VERYDIFFERENTLY THAN A STUDENT WHO PLANS ON TAKING OWNERSHIP FOR HIS ORHER LEARNING.EACH YEAR IN MY SCHOOL, IN MATH AND ENGLISH, WE PRE- AND POST-TESTWITH THE NORTHWEST EVALUATION ASSOCIATION ASSESSMENTS, OR AS WELIKE TO CALL IT, NWEA. STUDENTS UNDERSTAND THE FACT THAT THEY WILL11BE ANSWERING QUESTIONS TAILORED TO THEIR INDIVIDUAL LEVELS OFACHIEVEMENT. THERE IS A CERTAIN RESOLVE WHICH EXISTS IN THISCULTURE OF COMPUTER-ADAPTIVE TESTING WHICH RESULTS IN ABSOLUTELYNO ATTEMPT IN ANY WAY TO CHEAT. ONE ADDITIONAL AND DESIRABLEFEATURE IS THAT SCORES ARE IMMEDIATELY AND ELECTRONICALLY REPORTEDAND RECORDED, AND ONLY IF THE STUDENT HAS MET WHAT I CALL THE"HONEST ATTEMPT CRITERIA." SO DESIGN IS KEY.MY NEXT FOCUS IS VIGILANCE. A SIMILAR STUDENT RESOLVE EXISTS TOWARDCHEATING ON S.A.T. TESTS AT MY SCHOOL. I'VE PROCTORED S.A.T. TESTSOVER THE DECADES, AND STUDENTS ARE WELL AWARE OF THE FACT THATTHERE ARE MULTIPLE VERSIONS. WHEN I PROCTOR S.A.T.S, IT'S IN MYSCHOOL, OFTEN IN MY OWN CLASSROOM, WITH STUDENTS WHO ATTEND MY HIGHSCHOOL. ANONYMITY ISN'T SOMETHING THAT CAN BE USED TO BEAT THETEST.THE PROBLEM, AS WE'VE SEEN RECENTLY IN NEW YORK, IS NOT A DESIGNPROBLEM, BUT A VIGILANCE PROBLEM. STUDENTS PAYING AS MUCH AS $3,500RELIED ON ANONYMITY TO HAVE SOMEONE ELSE TAKE THE TEST FOR THEM.HIGH STAKES AND THE IMMENSE PRESSURE TO GET INTO SOME OF THE MOREPRESTIGIOUS UNIVERSITIES AND COLLEGES WAS ONE OF THE REPORTEDMOTIVATIONS FOR THIS BREACH. AND JUST A QUICK AND GREAT EXAMPLE OFVIGILANCE ARE THE WONDERFUL WEBSITES AVAILABLE TO COMBAT12PLAGIARISM. MY SCHOOL LIBRARIAN IS LIKE A REGULAR SHERLOCK HOLMESWHEN IT COMES TO THIS, AND WE REALLY RELY ON HER.RECENTLY, VIGILANCE HAS HIT AN EVEN GREATER CHALLENGE, HOWEVER,THAN MONITORING INDIVIDUAL STUDENTS. WITH RELIANCE AT DISTRICT ANDSTATE LEVELS ON HIGH-STAKES TESTING, EDUCATORS ARE FEELING INTENSEPRESSURES THAT HAVE NEVER EXISTED BEFORE. LET'S NOT FOOLOURSELVES. THERE ARE PRIZES TO BE HAD, PRIZES FOR WHICH TEACHERS,PRINCIPALS, AND EVEN UNIVERSITY ADMINISTRATORS ARE WILLING TO PUTTHEIR INTEGRITY ON THE LINE FOR. HOW DID IT EVER GET TO THIS? HOWDID TESTING EVER GAIN PRIORITY OVER TEACHING AND LEARNING?I THINK ABOUT AN OLYMPIC ATHLETE TRYING TO SHAVE SECONDS OFF HISTIME. HE MIGHT USE TESTING, EVALUATION, AND ASSESSMENT TO GET ATRUE PICTURE IN AN EFFORT TO IMPROVE. THE DIFFERENCE IS THE DESIRETO USE THE INFORMATION TO GET BETTER. WHY CAN'T ACADEMIC TESTING BELIKE THAT? WHY CAN'T AN ACADEMIC TEST BE A TOOL WHICH WE CAN USE TOIMPROVE?WE NEED TO PUT LEARNING BACK INTO THE TESTING PROCESS BECAUSELEARNING IS THE BUSINESS OF EDUCATION, NOT TESTING. YEARS AGO, IRECALL READING "FREAKONOMICS" BY THE ECONOMIST STEVEN LEVITT, ANDMAYBE I'M JUST NAIVE, BUT READING THE DETAILS OF TEACHERS ERASINGAND REPLACING ANSWERS WAS BEYOND BELIEF.13I FIND GREAT CONSOLATION IN THE FACT THAT SOFTWARE EXISTS TODAY TOIDENTIFY THIS TYPE OF CHEATING. WE DON'T ALWAYS GET LUCKY TO CATCHOFFENDERS RED-HANDED, AND SO SAFEGUARDS MUST BE BUILT INTO THETESTING PROCESS. AS A FORMER MEMBER OF THE NATIONAL ASSESSMENTGOVERNING BOARD AND CHAIR OF THE ASSESSMENT DEVELOPMENT COMMITTEE,I BECAME FAMILIAR WITH NAEP TESTING FOR THE NATION'S REPORT CARD.THE STRATEGY OF BRINGING IN INDEPENDENT TEST ADMINISTRATORS ISANOTHER WAY TO ELIMINATE THE TEMPTATION TO ALTER TEST RESULTS ANDTHUS PROMOTE TESTING INTEGRITY. VIGILANCE IS KEY.MY FINAL COMMENTS ARE ON CONSEQUENCES. EACH YEAR WHEN I BRIEF MY APSTUDENTS BEFORE THE TEST, I EXPLAIN TO THEM THAT IF THEY CHEAT,THEY PUT THE SCHOOL AT RISK FOR NOT BEING ABLE TO OFFER AP EXAMS.THE CONSEQUENCES ARE STRINGENT ENOUGH TO BE A DETERRENT, ANDSTUDENTS PAY ATTENTION. IF THE STAKES ARE HIGH, THEN THEPUNISHMENTS MUST BE COMMENSURATE.PEOPLE MAKE MISTAKES, AND WE ALL LEARN AND GROW FROM THEM, BUTWE'RE TALKING ABOUT COMPROMISING TESTING INTEGRITY FOR PERSONALGAIN. WE NEED TO WORK HARD TO GET AN ACCURATE PICTURE OF JUST WHATWE'RE DEALING WITH AND THEN DEVISE STRATEGIES TO ADDRESS IT.14THE GOOD NEWS IS THAT INCIDENCES ARE COMING TO LIGHT AND BEINGDEALT WITH. I WOULD LIKE TO COMMEND THE CITY OF ATLANTA FORCONFRONTING THE TEST FRAUD WHICH OCCURRED THERE. IT WAS A DIFFICULTAND PAINFUL PROCESS WHICH MADE IT CLEAR THAT THIS TYPE OF BEHAVIORCANNOT BE TOLERATED. DEPENDING ON THE CONSEQUENCES, THIS INCIDENTCAN SERVE AS A DETERRENT AND CONTRIBUTE TO A CHANGE OF ATTITUDE.I'M NOT HERE TO POINT FINGERS, BUT ACCORDING TO MOST OF THE SURVEYDATA I'VE SEEN, THE CASUAL ATTITUDE TOWARD CHEATING IS PERVASIVEAMONGST STUDENTS, EDUCATORS, PARENTS, AND COMMUNITY MEMBERS.I WOULD LIKE TO CONCLUDE BY SAYING THAT THE TIME SPENT TODAY HEREIS ABSOLUTELY ESSENTIAL TO MY JOB AS A CLASSROOM TEACHER. GOODSTUDENTS MAKE GOOD CHOICES EVERY DAY, AND THEY ARE DEPENDING ON ALLOF US TO SAFEGUARD TESTING INTEGRITY SO THAT THE REWARDS FORCHEATING AND THE REWARDS FOR WORKING HARD CAN NEVER BE THE SAME.THE PRESENCE OF SMARTER BALANCE AND PARCC HERE TODAY GIVE ME HOPETHAT THE NEW ERA OF TESTING WHICH IS JUST AROUND THE CORNER WILLPROVIDE CLASSROOM TEACHERS WITH TOOLS WHICH WILL HELP US GET BACKTO THE BUSINESS OF TEACHING AND LEARNING.THANK YOU SO MUCH, AND I HOPE MY COMMENTS WILL BE HELPFUL IN OURWORK TODAY.[APPLAUSE]15JACK BUCKLEY: THANK YOU VERY MUCH, KATHI.NOW IT'S MY DISTINCT PLEASURE TO WELCOME KAYA HENDERSON, CURRENTLYTHE CHANCELLOR OF THE DISTRICT OF COLUMBIA PUBLIC SCHOOLS. HEREDUCATION CAREER BEGAN AS A MIDDLE SCHOOL SPANISH TEACHER IN THESOUTH BRONX, AND SHE SPENT HER SUMMERS OVERSEEING THE PROFESSIONALDEVELOPMENT OF NEW TEACHERS AT SUMMER INSTITUTES WITH TEACH FORAMERICA. SHE ALSO SERVED TEACH FOR AMERICA AS A RECRUITER ANDNATIONAL DIRECTOR OF ADMISSIONS. IN 1997, SHE BECAME THE EXECUTIVEDIRECTOR OF TFA-DC.SHE CAME TO DCPS AS DEPUTY CHANCELLOR IN 2007, WHERE SHE LED THEDISTRICT'S HUMAN CAPITAL EFFORTS AND SERVED AS CHIEF NEGOTIATORFOR THE 2010 CONTRACT BETWEEN DCPS AND THE WASHINGTON TEACHERS'UNION.HENDERSON'S TEAM ALSO LED THE DEVELOPMENT OF IMPACT, THEDISTRICT'S NEW TEACHER ASSESSMENT SYSTEM.SHE RECEIVED HER BACHELOR'S DEGREE IN INTERNATIONAL RELATIONS FROMGEORGETOWN SCHOOL OF FOREIGN SERVICE AND HER MASTER OF ARTS INLEADERSHIP ALSO FROM GEORGETOWN UNIVERSITY. I'LL TURN IT OVER TOKAYA.16KAYA HENDERSON: GOOD MORNING.IN MARCH OF 2011, LESS THAN 6 MONTHS AFTER I WAS NAMED INTERIMCHANCELLOR FOR DC PUBLIC SCHOOLS, "USA TODAY" PRINTED A STORY WHICHALLEGED THAT THERE WERE SIGNIFICANT TESTING IMPROPRIETIES AT DCPUBLIC SCHOOLS.UPON LEARNING OF THESE ALLEGATIONS, MY SOLE INTEREST WAS INDETERMINING WHAT ACTUALLY HAPPENED AT OUR SCHOOLS AND HOW COULD WEENSURE THAT PARENTS, TEACHERS, STUDENTS, AND POLICYMAKERS WOULDHAVE CONFIDENCE IN OUR RESULTS? THIS SEEMINGLY SIMPLE BUT CRITICALDESIRE TURNED OUT TO BE MUCH MORE DIFFICULT THAN I ANTICIPATED.OVER THE PAST DECADE, WITH THE PASSAGE OF NO CHILD LEFT BEHIND,TEST SCORES HAVE PLAYED A CRITICAL ROLE FOR BOTH SUPPORTERS ANDCRITICS OF OUTCOME-BASED EDUCATION REFORM. WE USE TESTS TODETERMINE OUR STUDENTS' PROGRESS, WE USE TESTS TO CHARTIMPROVEMENTS IN SCHOOLS AND DISTRICTS, WE USE TESTS AS ONE ELEMENTOF OUR EVALUATION OF TEACHER EFFECTIVENESS, AND PARENTS USE TESTSCORES TO MAKE DECISIONS ABOUT WHERE THEY WANT TO SEND THEIRSTUDENTS TO SCHOOL.17BECAUSE EDUCATION QUALITY IS MULTIFACETED AND DIFFICULT TOEXPLAIN, WE FREQUENTLY--PROBABLY TOO FREQUENTLY--USE TEST SCORESAS OUR ONE PROXY FOR DISCUSSING REAL EDUCATION QUALITY. OUR USE OFTEST SCORES AND DATA HAVE HELPED US ADVANCE A MEANINGFULCONVERSATION ABOUT WHAT IT MEANS FOR SCHOOLS TO DO WELL. WE NOWFOCUS ON ISSUES RELATED TO THE BLACK/WHITE ACHIEVEMENT GAP WITHGREATER FOCUS, WE CAN ZERO IN ON THE PERFORMANCE OF STUDENTS WITHSPECIAL NEEDS AS WE WORK TO ENSURE THAT ALL OF OUR STUDENTS RECEIVEA QUALITY EDUCATION, AND WE CAN TARGET DIFFERENTIATEDINTERVENTIONS TO STUDENTS WHO ARE WELL BEHIND IN THEIR PROGRESSWHEN COMPARED TO THEIR PEERS.WE'VE CREATED AN EDUCATION ENVIRONMENT IN WHICH IT IS NOW THE NORMTO TALK ABOUT STUDENT ACHIEVEMENT DATA. EVEN DC CITYCOUNCILMEMBERS WHO ONCE SPOKE ONLY OF PAROCHIAL ISSUES RELATED TOTHEIR SCHOOLS NOW QUIZ ME REPEATEDLY ON SCHOOL PERFORMANCE DATA.BUT WE FAIL DEEPLY IN EXPLAINING TO PARENTS AND THE PUBLIC WHATTEST SCORES MEAN, HOW THEY SHOULD BE USED, AND HOW THEY COULD BEINTERPRETED.PRINCIPALS OFTEN VIEW TEST SCORES AS THE MEASURE--NOT ONE MEASURE- IN HOW THEIR SCHOOL IS DOING. WE FAIL TO EXPLAIN WHAT PROGRESS,AS OPPOSED TO ABSOLUTE SCORES, MEANS FOR A SCHOOL, AND WE'VECREATED A CULTURE IN WHICH STUDENTS, TEACHERS, AND PRINCIPALS FEAR18THEIR END-OF-YEAR TESTS, RATHER THAN LOOKING AT THEM AS ANOPPORTUNITY TO DEMONSTRATE THEIR PROGRESS AND IDENTIFY AREAS FORIMPROVEMENT.WHILE I WAS FAMILIAR WITH THESE CHALLENGES FROM MY YEARS OF WORKINGIN URBAN EDUCATION, I DIDN'T REALIZE THAT WE HAD ALSO FAILED TOESTABLISH CLEAR MEANS OF DETERMINING WHAT RELIABLE TEST SCORESARE, WE FAILED TO CREATED ANY NATIONALLY ACCEPTED STATISTICALANALYSIS TO DETERMINE WHEN SCORES ARE SUSPECT, AND WE FAILED TOPROPOSE A RELIABLE INVESTIGATIVE PLAN WHEN RESULTS ARE CALLED INTOQUESTION. WE'VE DONE THE HARD WORK TO ENSURE THAT WE COULD HAVE ADISCUSSION OF EDUCATION OUTCOMES THAT WOULD BE INFORMED BY TESTSCORES, BUT WE'VE NOT APPLIED EQUAL EFFORTS TO HELPINGPOLICYMAKERS OR OUR COMMUNITY UNDERSTAND WHAT TEST SCORES ARE FOR,WHAT THEY MEAN, AND HOW WE CAN MAKE SURE THAT THEY'RE RELIABLE.UPON LEARNING OF THE IMPROPRIETIES ALLEGED BY "USA TODAY," IQUICKLY DOVE INTO THE WORLD OF TEST INTEGRITY. I FOUND A DIZZYINGWORLD OF DATA ANALYSIS, NON-STANDARDIZED APPROACHES, ANDUNRELIABLE OUTCOMES, NONE OF WHICH SEEMED DIRECTED AT HELPINGDISTRICTS OR COMMUNITIES GAIN CONFIDENCE IN THEIR OUTCOMES--MOSTOF WHICH, IN FACT, PUT DISTRICTS IN A DEFENSIVE POSTURE.19BY ANY REASONABLE MEASURE, DCPS HAD DONE EVERYTHING THAT WE WERESUPPOSED TO DO TO ADDRESS ALLEGED TESTING IMPROPRIETY.WE HAD A RIGOROUS OVERSIGHT SYSTEM FOR ADMINISTERING TESTS THATENSURED THAT CENTRAL OFFICE STAFF MEMBERS AND OTHER THIRD PARTIESWERE PRESENT TO OBSERVE TESTING.WE HAD A CONFIDENTIAL TIP LINE WHICH ALLOWED CONCERNED PARTIES TOREPORT WRONGDOING.WE INVESTIGATED EVERY SINGLE ALLEGATION OF IMPROPRIETY THAT WERECEIVED, INCLUDING BOTH THOSE REPORTED BY INDEPENDENT SOURCES INSCHOOLS AND IN THE COMMUNITY AND THOSE IDENTIFIED FOR US BY OUROFFICE OF THE STATE SUPERINTENDENT OF EDUCATION.WE USED AN OUTSIDE CONTRACTOR TO CONDUCT INVESTIGATIONS INCLASSROOMS WHERE THE STATE IDENTIFIED CONCERNS, AND WHEN WEIDENTIFIED THE FEW STAFF WHO HAD COMPROMISED TEST INTEGRITY, WEACTED SWIFTLY TO PUNISH AND, WHEN APPROPRIATE, TO TERMINATE THESTAFF WHO WERE INVOLVED.AT THE SAME TIME, IT WAS EASY SPORT FOR THE PRESS TO PLAY THE "WHATMORE COULD BE DONE" GAME. COULDN'T WE HAVE LOOKED AT RIGHT-TO-WRONGERASURES IN MORE SCHOOLS? WASN'T THERE MORE THAT OUR VENDOR COULD20HAVE DONE TO IDENTIFY INCONSISTENCIES? COULDN'T INVESTIGATIONS BEMORE THOROUGH? CAN'T WE RELEASE THE INFORMATION TO THE PRESS SOTHAT THE PRESS AND THE PUBLIC COULD DECIDE FOR THEMSELVES?AND THE ANSWER TO ALL OF THOSE QUESTIONS, OF COURSE, IS YES. THEREARE INNUMERABLE ADDITIONAL THINGS THAT WE COULD HAVE DONE, BUTTHERE WAS NO REASON TO BELIEVE THAT ANY OF THESE ACTIONS WOULD HAVEYIELDED MORE RELIABLE RESULTS OR MORE ACCURATE RESULTS. IT WASCLEAR TO ME THAT IT WOULD BE VERY EASY FOR A DISTRICT LIKE OURS TOFALL DOWN A RABBIT HOLE OF TESTING INVESTIGATIONS ONLY TO FIND OUTTHAT BECAUSE THERE ARE NO WIDELY ACCEPTED STANDARDS, THERE'S NOAGREED-UPON RESULT THAT WOULD HAVE SATISFIED THE PRESS OR THEPUBLIC.WE HAVE A HUGE NUMBER OF TOOLS THAT WE CAN USE TO DETERMINE IF WETHINK THERE HAS BEEN IMPROPRIETY IN A TEST ADMINISTRATION. WE CANLOOK AT RIGHT-TO-WRONG ERASURES, THE MOST POPULAR AND EASIEST TOUNDERSTAND. WE LOOK AT ITEM PATTERN ANALYSIS, STUDENT GROWTHANALYSIS, CHANGE FROM THE PREDICTED OUTCOMES OF THE INTERIMASSESSMENTS, STUDENT PERFORMANCE COMPARED TO RESULTS FROM OUROBSERVATIONS OF TEACHERS, AND AN EVALUATION OF THE STUDENTS WHOCROSS SIGNIFICANT THRESHOLDS-- FOR EXAMPLE, FROM BELOW BASIC TOBASIC OR FROM BASIC TO PROFICIENT.21WHILE THESE TOOLS ARE ALL USEFUL AND MAKE INTUITIVE SENSE, THERE'SNO CLEAR STANDARD FOR USING THESE MEASURES. SHOULD WE INVESTIGATEEVERY CLASSROOM WITH AN ABOVE-AVERAGE LEVEL OF ERASURES, KNOWINGTHAT THIS WOULD FORCE US TO INVESTIGATE HALF OF ALL OF OURCLASSROOMS AND WOULD ONLY CATCH ONE FORM OF CHEATING?SHOULD WE INVESTIGATE CLASSROOMS WHERE THERE'S A HIGH LEVEL OFGROWTH, KNOWING THAT THIS WOULD PLACE A HUGE BURDEN ON OURABSOLUTELY HIGHEST PERFORMING TEACHERS, THE VERY ONES WE WANT TOREWARD AND KEEP IN OUR CLASSROOMS?IF WE'RE GOING TO USE ANY OF THESE MEASURES, HOW MANY STANDARDDEVIATIONS ABOVE THE MEAN DOES ONE NEED TO BE TO WARRANT ANINVESTIGATION?AND SHOULD WE USE SOME COMBINATION OF FACTORS, KNOWING THAT THISWILL INVOLVE DOZENS, IF NOT HUNDREDS, OF DECISIONS, EACH OF WHICHMAY LIMIT THE INVESTIGATION AND EACH OF WHICH MAKES EASY FODDER FORTHOSE WHO WOULD LIKE TO TEAR DOWN OUTCOME-BASED EDUCATION REFORM?AND WHAT SHOULD INVESTIGATIONS LOOK LIKE? SHOULD WE INTERVIEWSTUDENTS? AT WHAT AGE? HOW MANY? GIVEN THAT TEST RESULTS ARE NOTAVAILABLE UNTIL MONTHS AFTER ADMINISTRATION AND THAT INFORMATIONREGARDING POTENTIAL IMPROPRIETY MIGHT NOT BE AVAILABLE FOR MANY22MORE MONTHS, CAN WE RELY ONLY ON EYEWITNESSES? CAN WE INVESTIGATEWITHOUT ASKING LEADING QUESTIONS?BECAUSE THERE IS NO STANDARD, EITHER FOR IDENTIFYING POTENTIALWRONGDOING OR FOR INVESTIGATING ONCE CHEATING IS ALLEGED, WE ARELEFT WITH A FUZZY PICTURE OF WHAT RELIABLE OUTCOMES ARE. AS ARESULT, WE DO 3 THINGS THAT ARE VERY BAD FOR CHILDREN.WE TURN TESTS, WHICH ARE IN PLACE SO THAT WE CAN CELEBRATE STUDENTSUCCESSES AND ADDRESS OUR AREAS OF CONCERN, INTO GAMES FOR ADULTS.WE UNDERMINE OUR VERY BEST TEACHERS BY CALLING INTO QUESTION THEIRGOOD WORK AND GIVING THEM NO OPPORTUNITY TO DEFEND THEMSELVES ANDCLEAR THEIR NAME. AND WE ALLOW CRITICS OF OUTCOME-BASED REFORM TOUNDERMINE OUR EFFORTS TO ENSURE THAT WE IDENTIFY THE NEEDS OF EACHAND EVERY STUDENT AND TO WORK TO ENSURE THAT EACH STUDENT ACHIEVESPROFICIENCY NOT BECAUSE IT'S A GOAL, BUT BECAUSE IT'S A BASELINE.TOGETHER, WE MUST ESTABLISH A CLEAR SET OF FACTORS THAT CAN BEAPPLIED NATIONALLY TO DETERMINE WHEN TESTING IMPROPRIETIES OCCUR,HOW TO CONDUCT CONSISTENT STATISTICAL ANALYSIS TO HIGHLIGHTPOTENTIAL WRONGDOING, AND HOW TO INVESTIGATE CLASSROOMS WHEREANALYSIS RAISES QUESTIONS.23IN YESTERDAY'S "NEW YORK TIMES," MICHAEL WINERIP SAID HE WASDISAPPOINTED THAT I WAS INVITED TO SPEAK AT THIS CONFERENCE. MOREGENERALLY, THE PRESS HAS FRAMED THE CHALLENGE OF TEST INTEGRITY ASTHEIR STRUGGLE TO OUT DECEITFUL SCHOOL DISTRICTS. THIS APPROACH ISEXACTLY WRONG.SCHOOL DISTRICTS LIKE OURS STRUGGLE WITH THESE ISSUES CONSTANTLY.WE DON'T STRUGGLE TO HIDE CHALLENGES OR TO CONCEAL WRONGDOING. WESTRUGGLE TO ENSURE THAT OUR TEST RESULTS ARE RELIABLE ANDTRUSTWORTHY. WE STRUGGLE TO MAKE SURE WE KNOW HOW OUR STUDENTS AREDOING SO WE CAN INFORM PARENTS, PROVIDE INTERVENTIONS, AND MAKECORRECTIONS.TOGETHER, DCPS AND OUR STATE SUPERINTENDENT HAVE DEVELOPED ASTRONG APPROACH TO IDENTIFYING POTENTIAL INCONSISTENCIES ANDINVESTIGATING THESE CASES, AND WE ARE HAPPY TO USE THIS AS ASTARTING POINT FOR OUR NATIONAL CONVERSATION.HOWEVER, WE KNOW THAT ABSENT A NATIONAL STANDARD FOR THIS WORK, OURANALYSIS WILL ALWAYS BE SUBJECT TO SECOND-GUESSING. NO SCHOOLDISTRICT, NO TESTING COMPANY HAS BEEN ABLE TO PROVIDE A PROTOCOLFOR ENSURING THAT THERE IS INTEGRITY TO OUR TEST RESULTS, AND SO ICHOSE TO STEP FORWARD AND ASK THE U.S. DEPARTMENT OF EDUCATION FORGUIDANCE.24THE DEPARTMENT HAS BROUGHT YOU TOGETHER, A GROUP OF EXPERTS, TOHELP TACKLE THIS CHALLENGING ISSUE. ALONE, MY SCHOOL DISTRICTCANNOT HIRE A TEAM OF STATISTICIANS TO DEVELOP A RIGOROUS,NATIONALLY ACCEPTED INVESTIGATIVE STRATEGY. BY ITSELF, OUR STATECANNOT DEVELOP A PROTOCOL FOR INVESTIGATIONS THAT WILL WITHSTANDSCRUTINY.ALTHOUGH WE STRUGGLE WITH THE CHALLENGE OF VALIDATING OUR TESTRESULTS, NO ONE SCHOOL DISTRICT OR STATE CAN ARRIVE AT A VIABLESOLUTION BY ITSELF. AND SO TOGETHER, WE FACE A CHALLENGE AND ANOPPORTUNITY. OUR CHALLENGE IS THAT WE MUST COME TOGETHER TORECOGNIZE THAT THIS IS NOT A BATTLE OF DISTRICTS VERSUS THE PRESS.IT SHOULD BE OUR SHARED CHALLENGE TO VALIDATE TEST RESULTS SO WECAN FOCUS ON THE REAL ISSUE OF IMPROVING EDUCATIONALOPPORTUNITIES.WE ALSO HAVE AN OPPORTUNITY FOR YOU TO PROVIDE A UNIVERSALLYACCEPTABLE NATIONAL STANDARD FOR TEST INTEGRITY. TOGETHER, WE HAVETO ESTABLISH A NATIONAL CONSORTIUM THAT WILL BRING TOGETHER THEEXPERTISE OF STATISTICIANS, THE KNOWLEDGE OF TESTING EXPERTS, ANDTHE REAL-WORLD EXPERIENCES OF DISTRICT AND STATE LEADERS.25IF WE ARE GOING TO MOVE FORWARD WITH EDUCATION REFORM THATPRIORITIZES STUDENT OUTCOMES, THAT PRIZES CLEAR, UNDERSTANDABLE,AND RELIABLE RESULTS, AND THAT INFORMS PARENTS SO THEY CAN MAKEWISE DECISIONS ABOUT THEIR STUDENTS' EDUCATION, WE MUST MAKE SURETHAT OUR TESTING OUTCOMES ARE TRUSTED, BY PARENTS, BYPOLICYMAKERS, AND BY THE PRESS.AND SO TODAY I CALL ON YOU TO RECOGNIZE THE IMPORTANT ROLE THAT YOUPLAY AND TO ACCEPT THAT YOUR SUCCESS OR FAILURE WILL NOT BEMEASURED BY STATISTICIANS OR YOUR COLLEAGUES IN THIS ROOM. WE WILLONLY BE SUCCESSFUL WHEN WE CAN MEANINGFULLY EXPLAIN TO THE PARENTSOF OUR STUDENTS THAT THE TEST SCORES THEY RECEIVE PROVIDE CRITICALAND RELIABLE INFORMATION ABOUT THEIR STUDENTS' ACHIEVEMENT ANDTHAT THESE RESULTS ARE ONE IMPORTANT ELEMENT IN GAUGING STUDENTSUCCESS. THANK YOU.[APPLAUSE]JACK BUCKLEY: THANK YOU VERY MUCH, KAYA.NOW, IN THE FIRST OF WHAT'S GOING TO BECOME A FAMILIAR DRILL, WE'REGOING TO TAKE A COUPLE MINUTES WHILE WE SHUFFLE NAME PLACARDS UP TOTHE FRONT FOR OUR FIRST PANEL. SO IF OUR PANEL CAN JOIN US AT THEFRONT OF THE ROOM, WE'LL MOVE YOUR NAMETAGS.2627PANEL I: PREVENTION OF IRREGULARITIES IN ACADEMIC TESTINGJACK BUCKLEY: AS I MENTIONED IN THE INTRODUCTION, OUR FIRST PANELADDRESSES THE QUESTION OF HOW TO PREVENT TESTING IRREGULARITIES,WITH A FOCUS ON WHAT SEAS AND LEAS CURRENTLY DO, BARRIERS TOIMPLEMENTING BEST PRACTICES, AND LESSONS LEARNED FROM THE FIELD.OUR FIRST PANELIST, AUDREY AMREIN-BEARDSLEY, IS CURRENTLYASSOCIATE PROFESSOR IN THE MARY LOU FULTON TEACHERS COLLEGE ATARIZONA STATE UNIVERSITY. HER RESEARCH INTERESTS INCLUDEEDUCATIONAL POLICY, EDUCATIONAL MEASUREMENT, AND RESEARCH METHODS,AND SHE HAS PUBLISHED EXTENSIVELY ON ACADEMIC TESTING.GREGORY J. CIZEK IS PROFESSOR OF EDUCATIONAL MEASUREMENT ANDEVALUATION AT THE UNIVERSITY OF NORTH CAROLINA-CHAPEL HILL, WHEREHE TEACHES COURSES IN PSYCHOMETRICS, ASSESSMENT, STATISTICS,RESEARCH METHODS, AND PROGRAM EVALUATION. HIS INTERESTS INCLUDESTANDARD SETTING, VALIDITY, TEST SECURITY, AND TESTING POLICY, ANDHE'S THE AUTHOR OF "FILLING IN THE BLANKS" IN 1999, "CHEATING ONTESTS: HOW TO DO IT, DETECT IT, AND PREVENT IT" IN 1999, "DETECTINGAND PREVENTING CLASSROOM CHEATING," 2003, AMONG OTHERPUBLICATIONS.28SCOTT NORTON IS THE ASSISTANT SUPERINTENDENT OF THE OFFICE OFSTANDARDS, ASSESSMENTS, AND ACCOUNTABILITY AT THE LOUISIANADEPARTMENT OF EDUCATION. HE HOLDS A PH.D. IN EDUCATIONALADMINISTRATION AND SUPERVISION FROM LOUISIANA STATE UNIVERSITY.HIS OFFICE IS RESPONSIBLE FOR IMPLEMENTATION OF CONTENT STANDARDSAND DEVELOPMENT OF THE LOUISIANA COMPREHENSIVE CURRICULUM, ALLPARTS OF THE STATEWIDE ASSESSMENT PROGRAM, AND IMPLEMENTATION OFTHE STATE ACCOUNTABILITY SYSTEM. HE'S ALSO THE STATE'S K-12 LEAD INTHE PARTNERSHIP FOR ASSESSMENT OF READINESS FOR COLLEGE ANDCAREERS, OR PARCC, ONE OF THE TWO RACE TO THE TOPASSESSMENTCONSORTIA.FINALLY, JAMES S. LIEBMAN IS THE SIMON H. RIFKIND PROFESSOR OF LAWAT COLUMBIA LAW SCHOOL, WHERE HE HAS BEEN A MEMBER OF THE FACULTYSINCE 1985. FROM 2006 T0 2009, HE TOOK A PARTIAL LEAVE FROM LAWSCHOOL TO SERVE AS CHIEF ACCOUNTABILITY OFFICER OF THE NEW YORKCITY DEPARTMENT OF EDUCATION. WHILE THERE, HE CREATED AND LED THEDIVISION OF ACCOUNTABILITY AND ACHIEVEMENT RESOURCES, DIRECTED THECITY'S ASSESSMENT PROGRAMS, LED THE DESIGN AND IMPLEMENTATION OFTHE CITY'S COMPREHENSIVE NEW ACCOUNTABILITY SYSTEM, FACILITATEDTHE ESTABLISHMENT IN EACH OF THE CITY'S 1,500 SCHOOLS OF INQUIRYTEAMS FOR USING DATA TO IMPROVE STUDENT LEARNING, AND DESIGNED ANDDEPLOYED AN AWARD-WINNING CITY-WIDE INSTRUCTIONAL DATA SYSTEM.29WE'LL BEGIN WITH AUDREY.AUDREY AMREIN-BEARDSLEY: GOOD MORNING. THANK YOU FOR HAVING US TODAY.I'M GOING TO START WITH DEGREES OF CHEATING AND THE PREVENTION OFTESTING IRREGULARITIES. BACKGROUND -- WE GOT SOME BACKGROUND FROMBEFORE, THE PRESENTERS BEFORE. FROM THE RESEARCH, HERE ARE SOMETHINGS THAT I PULLED FROM THE LAST TWO DECADES OF RESEARCH.FOR STARTERS, PHELPS, 2005, WROTE, "IF A TEACHER'S PERFORMANCE ISJUDGED, IN WHOLE OR IN PART, ON THE BASIS OF THEIR STUDENTS' TESTRESULTS, CERTAINLY THEY ARE GIVEN AN INCENTIVE TO CHEAT." NICHOLSAND BERLINER, 2007, WROTE, "HIGH-STAKES TESTING ALMOST ALWAYSCORRUPTS THE INDICATORS USED AND THE EDUCATORS JUDGED BY SUCHTESTS." CAMPBELL'S LAW, 1976--"THE MORE ANY QUANTITATIVE SOCIALINDICATOR IS USED FOR SOCIAL DECISION-MAKING, THE MORE SUBJECT ITWILL BE TO CORRUPTION PRESSURES AND THE MORE APT IT WILL BE TODISTORT AND CORRUPT THE SOCIAL PROCESSES IT IS INTENDED TOMONITOR."BUT BACK IN 1999, BEFORE NO CHILD LEFT BEHIND WAS PASSED, SACKSQUESTIONED THE PREVALENCE OF CHEATING, ARGUING THAT IT IS "MORELIKELY RARE THAN COMMON."30LORRIE SHEPHARD, PROFESSOR AT BOULDER, COLORADO, 1990, ARGUED ADECADE EARLIER THAT CHEATING "IS GENERALLY BELIEVED TO OCCUR IN AVERY TINY PERCENTAGE OF SCHOOLS," APPROXIMATELY 1.3%.AGAIN, THIS IS PRE-NO CHILD LEFT BEHIND. WHAT WE DO KNOW, IT'S VERYDIFFICULT TO QUANTIFY THE LEVEL AT WHICH CHEATING OCCURS, AND IT'SALSO DIFFICULT TO QUANTIFY HOW IT OCCURS, AND THAT'S WHAT WE'VEENGAGED IN THIS RESEARCH STUDY THAT I'M GOING TO TALK ABOUT NEXT.RECENTLY, AS WE ALL ARE AWARE AND ARE HERE FOR, ANALYSTS FOUND THATONE IN 5 ELEMENTARY AND MIDDLE SCHOOLS ACROSS THE STATE OF GEORGIASUBMITTED "HIGHLY ABNORMAL" ANSWER SHEETS, WITH ALMOST 90% OF ONESCHOOL'S SCORES SUSPECT. AND NEWSPAPER ARTICLES CONTINUE TO BEPUBLISHED ON PRETTY MUCH A DAILY BASIS NOW ABOUT TEACHERS ANDADMINISTRATORS ACCUSED OF CHEATING ON HIGH-STAKES TESTS, YET STILLAN ACCURATE NUMBER OF INCIDENCES STILL ELUDES US.THEORETICAL ASSERTIONS. WE CAN ASSUME WITH CONFIDENCE THAT THEREARE MANY MORE INCIDENTS OF CHEATING ON HIGH-STAKES TESTS THAN AREREPORTED IN THE MEDIA. SO THE PROBLEM IS PROBABLY WORSE THAN WEEVEN EXPECT IT TO BE GIVEN WHAT WE'VE READ. WE CAN ALSO ASSUME WITHCONFIDENCE THAT THIS HAS BECOME INCREASINGLY APPARENT POST-NOCHILD LEFT BEHIND, WITH THE HIGHER-STAKES CONSEQUENCES ATTACHED TOTEST OUTCOMES.31BUT WHAT IS CHEATING? HOW DO WE DEFINE CHEATING? AND MIGHT WEUNDERSTAND CHEATING BETTER TO PREVENT SUCH INCIDENCES FROM, FOREXAMPLE, CAUSING DRAMATIC IRREGULARITIES AND DISTORTING VALIDINTERPRETATIONS? WE HAVE A RELIABILITY ISSUE IN TERMS OFCONSISTENCY OVER TIME ON TEST SCORES, BUT I THINK MORE IMPORTANTLY,WE HAVE AN ISSUE WITH THE ABILITY TO WHICH WE CAN MAKE VALIDINTERPRETATIONS ABOUT STUDENT LEARNING, STUDENT ACHIEVEMENT,TEACHER QUALITY, TEACHER EFFECTIVENESS, SCHOOL QUALITY, AND SOFORTH. IT'S THE VALID INTERPRETATIONS WITH WHICH I THINK WE'RE MOSTCONCERNED.THE EMPIRICAL APPROACH THAT WE TOOK--THERE'S A GROUP OF COLLEAGUESOF MINE AT ARIZONA STATE UNIVERSITY. WE INVESTIGATED THE TYPES OFAND DEGREES TO WHICH TEACHERS IN ARIZONA--THE SAMPLE WAS ABOUT3,000 FROM THE POPULATION--ENGAGED IN TEST-RELATED CHEATINGPRACTICES ON STATE-MANDATED, HIGH-STAKES, LARGE-SCALE TESTS. THISIS THE NO CHILD LEFT BEHIND TEST IN ARIZONA, WHICH IS CALLED THEAIMS.KNOWING THAT THESE BEHAVIORS DO CREATE IRREGULARITIES IN ACADEMICTESTING, PARTICULARLY WHEN CONSEQUENCES ARE ATTACHED, THE GOALHERE WAS TO UNDERSTAND WHAT CHEATING LOOKS LIKE IN THE CLASSROOM ATTHE APPLIED LEVEL, AND BY UNDERSTANDING, TO INFORM POLICIES--STATE32LEVEL, DISTRICT LEVEL, CLASSROOM LEVEL, EVEN--POLICIES TO PREVENTFURTHER CHEATING.ONCE WE STARTED GATHERING THE DATA, WE DETERMINED THAT WE COULD NOTCAPTURE--ONE DEGREE OF CHEATING WAS DIFFERENT THAN ANOTHERINCIDENCE OF CHEATING, SO WE KNEW THAT WE HAD TO COME UP WITH SOMEKIND OF DEFINITION. WE CAME UP WITH A TAXONOMY OF CHEATING.WE BASED THIS ON THE FIRST-, SECOND-, AND THIRD-DEGREE OFFENSES INTHE FIELD OF LAW, SO WE BASICALLY CALLED THIS THE FIRST DEGREE OFCHEATING, SECOND DEGREE OF CHEATING, AND THIRD DEGREE OF CHEATINGTO TALK ABOUT THIS MORE ELOQUENTLY, WE HOPE, AND TO ALSO UNDERSTANDIT TO PREVENT IT.WE HOPED THE TAXONOMY WAS USEFUL FOR UNDERSTANDING AND PREVENTINGFURTHER IRREGULARITIES. THE TAXONOMY CAME OUT TO BE LIKE THIS.THIS EMERGED AFTER THE DATA WERE COLLECTED FROM THESE 3,000TEACHERS IN THE STATE OF ARIZONA. CHEATING IN THE FIRST DEGREE,JUST AS IN THE FIELD OF LAW--WILLFUL AND PREMEDITATED, THE MOSTSERIOUS AND THE MOST WORTHY OF SANCTIONS.FOR EXAMPLE, THINGS THAT CAME OUT THROUGH THE DATA—ERASING ANDCHANGING STUDENTS' TEST ANSWERS, FILLING IN BUBBLES LEFT BLANK,33OVERTLY AND COVERTLY PROVIDING CORRECT ANSWERS ON TESTS,FALSIFYING STUDENT I.D. NUMBERS, BECAUSE IF THE STUDENT I.D.NUMBERS ARE FALSE, THEN THE STUDENT'S TESTS USUALLY GET REJECTEDFROM THE SYSTEM, EXCLUDING/SUSPENDING STUDENTS WITH POOR ACADEMICPERFORMANCE--THE ONES THAT ARE LESS DESIRABLE WHEN TESTING TIMECOMES.CHEATING IN THE SECOND DEGREE WE CLASSIFIED AS OFTEN MORE SUBTLE,DEFINED MORE CASUALLY, NOT NECESSARILY PREMEDITATED OR WITHMALINTENT. FOR EXAMPLE, ENCOURAGING STUDENTS TO REDO PROBLEMS ORDOUBLE-CHECK THEIR WORK, ACCOMMODATING FOR "STUPID MISTAKES"--WETALKED TO A LOT OF TEACHERS. A LOT OF TEACHERS REPORTED THEY'D WALKBY AND THEY KNEW THAT THEIR STUDENTS KNEW THE RIGHT ANSWER, ANDTHEY'D GIVE THEM A NUDGE AND SAY, YOU KNOW, "DOUBLE-CHECK THATONE." SO IT WAS SOMETHING THAT WASN'T FIRST DEGREE--IT WASN'TOUTRIGHT PREMEDITATED--BUT IT WAS SOMETHING THAT A LOT OF TIMES,THE TEACHERS THOUGHT THAT THEY WERE HELPING THEIR STUDENTS, WHEN ITCOULD BE DEFINED AND CLASSIFIED AS CHEATING. DISTRIBUTING "CHEATSHEETS," SOMETIMES THAT INCLUDE VOCABULARY WORDS. THE VOCABULARYMIGHT BE INCLUDED IN THE TEST STEMS OR THE QUESTIONS. TALKINGSTUDENTS THROUGH PROCESSES AND DEFINITIONS, GIVING STUDENTS EXTRATIME ON TESTS OR TIME DURING LUNCH, RECESS, AND BEFORE OR AFTERSCHOOL.34CHEATING IN THE THIRD DEGREE WE DEFINED AS CAUSED BY INDIFFERENCE,RECKLESSNESS, OR NEGLIGENCE, AND ALSO REFERRED TO AS INVOLUNTARYCHEATING. IN MANY OF THESE INCIDENCES, TEACHERS DIDN'T THINK WHATTHEY WERE DOING WAS CHEATING. A LOT OF TIMES, THEY THOUGHT IT WASREALLY GOOD TEST PREPARATION PRACTICES. THINGS LIKE, WHICH WE'VEALL HEARD BEFORE, "TEACHING TO THE TEST," WHICH IS VERYCONTROVERSIAL BECAUSE THERE'S A DIFFERENCE BETWEEN TEACHING TO THETEST, WHICH IS ACTUALLY THE TEST ITEMS, AND THEN TEACHING TO THESTANDARDS. SO THERE'S A LOT OF CONTROVERSY. JAMES POPHAM FROM UCLATALKS ABOUT THIS QUITE A BIT.ACCESS TO TEST BLUEPRINTS, WHICH I BELIEVE IS A STATE POLICY.BLUEPRINTS ARE PUT ONLINE, BUT A LOT OF TIMES, TEACHERS TAKE THOSEAND TEACH TO THE BLUEPRINT, SO THERE'S ACTUALLY ANOTHER CAVEATTHERE, WHERE IT'S NOT JUST TEACHING TO THE TEST; IT COULD BETEACHING TO THE BLUEPRINT, WHICH STILL MARGINALIZES SOME THINGSTHAT ARE NOT TESTABLE.NARROWING OF THE CURRICULUM. I THINK WE ALL UNDERSTAND WHAT THATIS, BUT MARGINALIZING THINGS LIKE PHYSICAL EDUCATION, RECESS, EVENSOCIAL STUDIES AND SCIENCE, ESPECIALLY RIGHT BEFORE THE TESTS AREADMINISTERED. HYPER-UTILIZING "CLONE ITEMS," WHERE YOU TAKE THEITEMS IN THE WORD PROBLEMS AND YOU CHANGE THE NAMES OF THE PEOPLE35IN THE WORD PROBLEMS AND YOU CHANGE THE NUMBERS, BUT YOU BASICALLYUSE THE SAME TYPES OF TESTS AND DEVELOP CLONE ITEMS FOR TESTPREPARATION. IS THAT PROFESSIONAL? IS IT ETHICAL? IS THATAPPROPRIATE TEST PREPARATION PRACTICE?THE CHEATING IN THE THIRD DEGREE IS WHERE IT GETS REALLY INTO THEGRAY AREA, AND A LOT OF PEOPLE DIFFER ON WHETHER THEY AGREE THATTHAT'S CHEATING OR NOT. AND ALSO INORDINATELY FOCUSING ON TESTTAKING STRATEGIES. DO WE HAVE TO CLICK TO GET THAT BACK?WE LOST OUR...BUCKLEY: TIME'S UP, TIME'S UP.[LAUGHTER]AMREIN-BEARDSLEY: OK. ALL RIGHT. I'LL JUST TURN AROUND. THERE WE GO.OK. BUT I FAST-FORWARDED. NOW I GOT TO GO BACK.OK. THAT WAS OUR TAXONOMY, AND THIS IS THE CHART THAT'S PUTTOGETHER IN THE ACTUAL PAPER. THE PAPER IS AVAILABLE. IT'S FREEONLINE. IT'S CALLED "CHEATING IN THE FIRST, SECOND, AND THIRDDEGREE," IN CASE ANYONE IS INTERESTED IN READING FURTHER. IT'S36ABOUT 60 SINGLE-SPACE PAGES, SO IT'S QUITE THE READ, BUT THESE ARETHE MAIN PIECES THAT I PULLED OUT OF IT.YOU'LL SEE IN THIS CHART HERE, WE'VE SEPARATED INTO FIRST-DEGREECHEATING, SECOND-DEGREE CHEATING, AND THIRD-DEGREE CHEATING. IFYOU LOOK AT THE FAR LEFT, FOR EXAMPLE, IT'S CATEGORIZED BY, A-CHEATING BEHAVIOR IS ERASED AND CHANGED TEST ANSWERS. THE WHITECOLUMN IS HEARSAY--"I'VE HEARD ABOUT OTHER TEACHERS DOING THIS,BUT I'VE NEVER DONE THIS MYSELF"--AND THEN THE DARKER COLUMN ISSELF-ADMITTED--"YEAH, I HAVE DONE THIS BEFORE IN THE PAST, AT SOMELEVEL."THEN YOU CAN GO ACROSS, AND IT'S THE SAME AS THE HEARSAY AND THENTHE SELF-REPORTED ACROSS, AND YOU CAN SEE JUST BY LOOKING AT THISTHAT THE FIRST-DEGREE CHEATING, THE MOST OUTRIGHT PREMEDITATED, ISWITH THE LOWEST FREQUENCY, AND PARTICULARLY WITH THE SELF-REPORT,ALTHOUGH THERE'S ALSO A SOCIAL DESIRABILITY THREAT THERE, WHETHERTEACHERS ARE REALLY GOING TO ADMIT THEY ACTUALLY DID IT, BUT IT'S ALOOSE INDICATOR OF WHAT THIS MIGHT LOOK LIKE.YOU CAN SEE AS YOU GET TO THE SECOND-DEGREE LEVELS AND THE THIRDDEGREE LEVELS, THESE THINGS ARE MORE COMMON. FOR EXAMPLE, LEVEL"H," WHICH IS ENCOURAGED STUDENTS TO REDO PROBLEMS, WAS PRETTYCOMMON AMONG THESE PARTICIPANTS. OK?37SO WHAT TO DO? WE FRAMED THESE RECOMMENDATIONS ALSO WITHIN THE SAMETAXONOMY. FIRST-DEGREE RECOMMENDATIONS--KEEP TESTS SECURE BEFOREAND AFTER ADMINISTRATION, WHICH SEEMS TO BE PRETTY COMMON PRACTICETHESE DAYS; PREVENT EXPOSURE, PHOTOCOPYING, WHICH WAS VERY COMMON,ITEM TRANSFORMATION AND CLONING, WHICH WAS VERY COMMON, AND THEYEAR-TO-YEAR USE OF OLD FORMS. A LOT OF TEACHERS REPORTED THAT IFTHEY EVEN HAD ACCESS, THEY WOULD EITHER MEMORIZE THE ITEMS ANDWRITE THEM DOWN, OR IF THEY HAD A CHANCE, THEY WOULD MAKEPHOTOCOPIES AND USE THEM AS THE TEST-PREPARATION MATERIALS FOR THEFOLLOWING YEAR 1, YEAR 2, YEAR 3. THEY'D USE THEM CONTINUOUSLY OVERTIME.HAVE THE LEAST LIKELY TO DISTORT AND ARTIFICIALLY INFLATE IN CHARGEOF TESTING PROCEDURES. SO DECREASE THE LIKELIHOOD OF THINGS LIKEORAL EMPHASIS OF CORRECT ANSWERS, REWORDING PROBLEMS, DEFINING KEYTERMS, EXTRA TIME ON TESTS, TEST MANIPULATION, WHAT WE CALL"CLEANING UP THE TEST."ADMINISTER TESTS IN ARTIFICIAL ENVIRONMENTS. ADMINISTRATION SHOULDOCCUR WHERE ACCESS TO CURRICULAR MATERIALS, RESOURCES, AND VISUALSIS MOST LIMITED. ONE OF THE THINGS THAT WAS ASKED ON THE SURVEY ISWHETHER TEACHERS KEPT UP CLASSROOM RESOURCES AROUND THE CLASSROOM,INCLUDING POSTERS AND NUMBER CHARTS, AND A LOT OF THEM WERE38ACTUALLY VERY ANGRY BY THAT ITEM BECAUSE THEY THOUGHT THAT IT WASNOT FAIR TO MAKE THE ENVIRONMENT ARTIFICIAL, THAT THE STUDENTSSHOULD HAVE ACCESS TO THOSE RESOURCES, AND SO THAT'S SOMETHING THATWE'D HAVE TO DEBATE, AS WELL--WHETHER IT SHOULD BE AN ENTIRELYARTIFICIAL ENVIRONMENT OR IT SHOULD JUST BE WHAT THEY'RE AROUND ONA DAILY BASIS ANYWAY.PUT IN PLACE POLICIES TO ENSURE THAT THE MOST "UNDESIRABLE" ORLOWEST-SCORING STUDENTS CANNOT BE EXEMPTED OR EXCLUDED ORCONVINCED TO STAY HOME UNTIL THE TEST IS OVER. THAT WAS QUITECOMMON, ACTUALLY—CONVINCING SOME OF THESE STUDENTS THAT, "DON'TBOTHER TO COME TO SCHOOL TODAY," PARTICULARLY ENGLISH-LANGUAGELEARNERS AND SPECIAL EDUCATION STUDENTS WHO ARE TYPICALLY SOME OFOUR LOWEST-SCORING STUDENTS ANYHOW. SECOND- AND THIRD-DEGREERECOMMENDATIONS. THIS IS WHERE THE GRAY AREA COMES BACK IN. WHETHERWE DEFINE THESE ARE CHEATINGS OR APPROPRIATE TEST-PREPARATIONPRACTICES HAS TO BE DETERMINED, PROBABLY MORE ON A LOCAL BASIS.FIRST THING HERE IS DO NOT BECOME OVERLY DEPENDENT ON TECHNICALSOLUTIONS. FOR EXAMPLE, ERASURE ANALYSES AND OTHER TECHNICALAPPROACHES CAN ONLY DO SO MUCH. IF I WAS A TEACHER, I THINK ITWOULD BE VERY LIKELY IF I KNEW THAT THE ERASURE ANALYSES WERE THEKEY INDICATORS THAT WERE GOING TO CATCH ME AS A CHEATER, AS ATEACHER CHEATER, I PROBABLY WOULD JUST CONVINCE THE STUDENTS TO39LEAVE THE ANSWERS BLANK AND JUST GO BACK IN AND FILL THEM IN MYSELFOR WHATEVER IT MIGHT BE.WE CAN'T OVER-RELY ON TECHNICAL SOLUTIONS TO THIS. I THINK WE HAVETO BE VERY HOLISTIC IN THE APPROACHES THAT WE TAKE AND UNDERSTANDTHE COMPLEXITY AND THE DEGREES TO WHICH THESE THINGS EXIST ANDOCCUR. A HEALTHY TESTING CULTURE IS MOST IMPORTANT. MANY DO NOTCONSIDER, AGAIN, WHAT THEY ARE DOING IS CHEATING, AND WHILE ONEEDUCATOR MIGHT CONSIDER SOME OF THESE PRACTICES SMART, OTHERSMIGHT CONSIDER THEM UNPROFESSIONAL OR UNETHICAL.SOME IRREGULARITIES OCCUR AS PART OF DISTRICT OR SCHOOL TESTPREPARATION AND ADMINISTRATION POLICIES, TOO. THERE WERE A LOT OFTEACHERS THAT SAID, "I DON'T AGREE WITH WHAT MY PRINCIPAL IS SAYINGTHAT WE SHOULD DO WITH TEST PREPARATION OR USING THESE MATERIALSFOR TEST PREPARATION OR BRINGING IN TEST CONSULTANTS SAYING, YOUKNOW, 3 IN A ROW, NO, NO, NO, BECAUSE NO SELF-RESPECTING TESTAUTHOR WOULD EVER PUT THE ANSWERS C, C, C." REALLY MANIPULATIVETYPES OF TEST PREPARATION EXERCISES THAT SOME OF THESE STUDENTS AREGOING THROUGH, AND THE TEACHERS REJECTED THAT, BUT THEY WOULD SAY,"WELL, THE ADMINISTRATOR...THIS IS THE ADMINISTRATOR. IT'S ONE OFTHOSE TOP-DOWN INITIATIVES, AND HOW DO WE COMBAT THIS? WE AGREETHAT WE'RE CHEATING, BUT IT'S ALSO THE ADMINISTRATOR SAYING THATWE'RE GOING TO DO THIS FOR PROFESSIONAL DEVELOPMENT DAY."40PARENT/TEACHER/ADMINISTRATOR ANONYMOUS WHISTLE-BLOWING SYSTEM WASA SUGGESTION, BUT MANY BELIEVED THAT WHEN CHEATING WAS EVENREPORTED ANONYMOUSLY, IT WENT IGNORED OR CONSEQUENCES WERENEGLIGIBLE--A SLAP-ON-THE-HAND TYPE OF CIRCUMSTANCE ORCONSEQUENCE.OOH! I'M ALMOST OUT OF TIME. COME TO COLLECTIVE UNDERSTANDINGS ANDLOCAL POLICIES ABOUT PROFESSIONAL AND ETHICAL TEST PREPARATIONPRACTICES. I THINK THE BIG KEY HERE IS COLLECTIVELY DETERMINEWHETHER THINGS LIKE TEACHING TO THE TEST, WHETHER THESE THINGS AREAPPROPRIATE, AND I THINK THAT THE LOCAL OR THE DISTRICT LEVEL, YOUCAN COME TO DO THAT. IT'S A MORE CULTURALLY APPROPRIATE SOLUTION TOTHIS.AND KEEP DECISIONS ALWAYS, AND POLICIES, IN THE BEST EDUCATIONALINTERESTS OF THE CHILDREN. AND I THINK I CAN SKIP THAT, SINCE I'MOUT OF TIME.THANK YOU.JACK BUCKLEY: THANK YOU VERY MUCH. GREG.GREG CIZEK: THANK YOU, JACK, EUNICE.41IT'S AN HONOR, AND I REALLY APPRECIATE BEING INVITED TO SPEAK HERE.MY COMMENTS ARE ENTITLED "SHARED RESPONSIBILITIES." I WANT TO TALKABOUT THE ROLE THAT STUDENTS, EDUCATORS, TEST DEVELOPERS, ANDPROFESSIONAL ASSOCIATIONS HAVE IN PREVENTING.FIRST, I'LL JUST QUICKLY GLOSS OVER SOME THINGS RELATED TO WHAT--ITHINK STUDENTS PLAY A ROLE IN THIS, AS WELL. THAT IS--3 QUICKTHINGS. STUDENTS NEED TO BE A PART OF AN ACADEMIC COMMUNITY THATEMBRACES INTEGRITY. STUDENTS NEED TO ADOPT MORE LEARNING VERSUSPERFORMANCE GOALS, AND EDUCATORS HAVE A ROLE, OBVIOUSLY, INFACILITATING THAT. AND STUDENTS HAVE SOME OBLIGATIONS, ALSO, TOREPORT THEIR CONCERNS. WE KNOW THAT HONOR CODES ARE EFFECTIVE.THERE'S REASONS WHY THEY'RE EFFECTIVE, AND IT'S NOT BECAUSE PEOPLESIGN SOMETHING. IT'S BECAUSE THE CULTURE, THE SCHOOL, EVERYONE INTHE COMMUNITY SEES THEMSELVES AS ALL EMBRACING ACADEMIC INTEGRITY.EDUCATORS HAVE A ROLE—TRAINING IN ASSESSMENT AND QUALIFICATION.THE FIRST THING--LET'S SEE. DID I SKIP ONE? YEAH. THE FIRST THING ISHOULD SAY ABOUT EDUCATORS IS THAT THEY CAN DO THE SAME KINDS OFTHINGS THAT STUDENTS DO IN TERMS OF BEING RESPONSIBLE FORIMPLEMENTING A CULTURE WHERE LEARNING IS THE GOAL, AS OPPOSED TOPERFORMANCE, REPORTING CONCERNS, AND EMBRACING THE CULTURE OFACADEMIC INTEGRITY.42BUT BEYOND THAT, I LABELED THIS "EDUCATORS" BECAUSE I THINK TOOOFTEN WHEN WE SPEAK ABOUT THIS ISSUE, WE SAY "TEACHERS." IF WE KNOWANYTHING NOW, IT'S SCHOOL COUNSELORS, PRINCIPALS, SUPERINTENDENTS,CHANCELLORS-- EVERYBODY HAS HAD THEIR FINGER IN THE PIE AT ONE TIMEOR ANOTHER, SO IT'S UNFAIR, I THINK, TO PIN IT EXPRESSLY ON JUSTTEACHERS.THE FIRST THING I'LL TALK ABOUT IS TRAINING IN ASSESSMENT. IT'SWOEFULLY INADEQUATE. IN MANY STATES, YOU DON'T HAVE TO BE QUALIFIEDIN ASSESSMENT TO BECOME A TEACHER, YOU DON'T HAVE TO BE QUALIFIEDIN ASSESSMENT TO BECOME A PRINCIPAL. THAT'S A SHAME. STATE POLICIESSHOULD CHANGE IN THAT MATTER. WHAT HAPPENS IS IT LEADS TO ADISTRUST OF TESTING.I HEAR THE MOST OUTLANDISH THINGS IN MY STATE. WE GIVE DIFFERENTFORMS OF A TEST--THE RED, THE BLUE, OR THE PURPLE, OR SOMETHINGLIKE THAT--AND EVERYONE KNOWS THE RED ONE IS HARDER. THAT'SRIDICULOUS. THEY'RE PARALLEL FORMS. IF YOU KNOW ANYTHING ABOUTTESTING, YOU WOULD KNOW THAT IT WOULD BE IRRESPONSIBLE TO GIVE AHARDER TEST TO SOME KIDS AND EASIER TEST TO OTHERS. IT LEADS TOMISTRUST OF PEOPLE WHO TEST. IT LEADS TO OUTLANDISH THINGS LIKEPEOPLE SAYING, "WELL, THIS ISN'T RELIABLE. THE TEST THAT THE STATEGIVES, IT'S BIASED. IT'S UNRELIABLE. IT'S A SINGLE SNAPSHOT OF WHATA KID CAN DO. WHO CAN COUNT ON ONE DAY'S PERFORMANCE?"43THAT'S ABSURD. IF YOU UNDERSTOOD HOW LARGE-SCALE TESTS ARE PUTTOGETHER--I'LL JUST SINGLE OUT WES BRUCE. THE SINGLE LARGE-SCALETEST THAT HE GIVES TO STUDENTS IN THE STATE OF INDIANA EVERY YEARIS BY FAR AND AWAY THE MOST OBJECTIVE, FAIREST, LEAST BIASED, MOSTDEPENDABLE MEASURE THAT THAT KID WILL SEE THE ENTIRE YEAR,ESPECIALLY COMPARED TO THE STUFF THAT'S PRODUCED, LARGELY, BYCLASSROOM TEACHERS WHO DON'T HAVE TRAINING IN ASSESSMENT OR WHO USEWHAT HAPPENS TO BE PASTED IN THE BACK OF SOME INSTRUCTOR'S MANUAL,AND THEY RELY ON THAT KIND OF STUFF.LARGE-SCALE TESTS ARE REALLY PRETTY GOOD, AND THERE'S JUST GROSSIGNORANCE BECAUSE--I'M NOT FINGERING TEACHERS OR PRINCIPALSBECAUSE OF THIS; I'M FINGERING STATES WHO DON'T REQUIRE EDUCATORSTO HAVE A FIRM GROUNDING IN ASSESSMENT.I TALK ABOUT QUALIFICATION. YOU NEED TO BE QUALIFIED TO TEACH. YOUALSO NEED TO BE QUALIFIED TO GIVE AND INTERPRET ASSESSMENTS, AND WEDON'T HIGHLIGHT THAT ENOUGH. PEOPLE NEED TO BE FAMILIAR WITH HOW TOGIVE A TEST AND FAMILIAR WITH THE DOS AND DON'TS OF QUALITY DATAINFORMATION GATHERING. THAT ALSO INCLUDES EXAMINATION PROCTORINGOR TEST PROCTORING. IT'S NOT JUST A DAY OFF OR A PERIOD OFF TOWATCH OR READ A BOOK OR CATCH UP ON SOME GRADING. THERE'S A44CONSCIENTIOUS ROLE THAT EDUCATORS HAVE TO PLAY WHEN A TEST IS GOINGON, AND THERE'S ALSO A NEED TO ASSIST STUDENTS.I'M TALKING ABOUT ASSISTING STUDENTS IN AN APPROPRIATE WAY.IF A STUDENT DOESN'T KNOW HOW TO FILL IN A GRIDDED RESPONSE ITEM,YOU'RE GOING TO GET AN INVALID TEST SCORE FOR THAT KID, SO ATEACHER SHOULD BE THERE OR AN EDUCATOR SHOULD BE THERE TO HELPPEOPLE UNDERSTAND THE TESTING PROCESS. THERE SHOULD BE SPECIFICRESPONSIBILITIES ASSIGNED FOR PEOPLE IN TERMS OF REVIEWING ORMONITORING TEST RESULTS.IN TOO MANY JURISDICTIONS, THERE'S NOT SOMEBODY WHOSE SPECIFIC JOBIT IS, AS PART OF THEIR JOB RESPONSIBILITY, TO MONITOR TESTRESULTS. THERE SHOULD BE DISSEMINATION OF THE PROCEDURES THAT AREIN PLACE TO PREVENT AND FOLLOW-UP DISSEMINATION OF THECONSEQUENCES THAT HAVE TAKEN PLACE. WHEN YOU HEAR THAT YOURNEIGHBOR GOT AUDITED, IT'S A STRONG DISINCENTIVE FOR YOU TO CHEATON YOUR TAXES, SO EVEN THE CONSEQUENCES SHOULD BE DISSEMINATED, ASWELL.AND WE SHOULD CONSIDER PROCTORING ASSIGNMENTS FOR EDUCATORS. FORLOWER GRADES, IT'S REALLY IMPORTANT TO HAVE THE CLASSROOM TEACHERTHERE, BUT THERE IS ABSOLUTELY NO REASON WHERE AN END-OF-GRADE TEST45IN ALGEBRA AT THE HIGH-SCHOOL LEVEL HAS TO BE NURTURED BY THATHIGH-SCHOOL KID'S CLASSROOM TEACHER. THERE CAN BE INDEPENDENTASSIGNMENT OF PROCTORING.TEST PROVIDERS, CONTRACTORS, TEST DEVELOPERS HAVE A ROLE, TOO.SOME BROADER ONES--I THINK WE NEED A CLEAR DEFINITION OF CHEATING,AND I'M DIFFERENTIATING CHEATING FROM AN IRREGULARITY. ANIRREGULARITY IS WHERE SOMEBODY JUST INNOCENTLY DOES SOMETHINGINAPPROPRIATE. IT'S NOT AN INTENTIONAL ATTEMPT TO ALTER THEMEANING OF A TEST SCORE, BUT CHEATING IS, AND WE NEED A CLEARDEFINITION OF WHAT THAT IS.WE NEED CLEARER, I PROBABLY SHOULD HAVE SAID, EDUCATOR REFERENCEMATERIALS. I'VE SEEN A LOT OF THE TEST ADMINISTRATION MANUALS THATARE DISTRIBUTED TO FIELD PERSONNEL TO USE, AND THEY'RE JUST INTYPICALLY TESTING JARGON AND NOT WELL SUITED TO WHAT EDUCATORS NEEDTO UNDERSTAND HOW TO APPROPRIATELY GIVE A TEST. SO THINGS LIKEEXAMPLE/NON-EXAMPLE. IF YOU DO THIS WITH A STUDENT, THAT'S REALLYGOOD. THAT HELPED ENSURE A VALID TEST SCORE. IF YOU DO THIS WITH ASTUDENT, THAT WAS INAPPROPRIATE. THAT WOULD BE CHEATING. THOSEKINDS OF EXAMPLES/NON-EXAMPLES NEED TO BE DISSEMINATED.AND PERHAPS SOMETHING LIKE--I'M GOING OUT A BIT ON A LIMB HERE--BUTI SAID PEOPLE NEED TO BE QUALIFIED TO GIVE TESTS JUST LIKE THEY'RE46QUALIFIED TO TEACH. WE DON'T ACTUALLY HAVE PEOPLE QUALIFIED TO BEABLE TO GIVE TESTS. I'M AT A UNIVERSITY WHERE IF I'M GOING TO DOANY KIND OF RESEARCH, WHICH IS REQUIRED IN MY JOB, I HAVE TOQUALIFY TO PROTECT THE RIGHTS OF HUMAN PARTICIPANTS IN RESEARCH.OTHERWISE, I CAN'T EVEN DIRECT A DISSERTATION, MUCH LESS CONDUCTRESEARCH MYSELF.SOME CONTRACTOR, I THINK, OUGHT TO TAKE THE LEAD IN DEVELOPING AWEB-BASED QUALIFICATION TOOL THAT PEOPLE COULD GO IN AND READSCENARIOS ABOUT WHAT'S APPROPRIATE OR INAPPROPRIATE PRACTICE ANDBE QUALIFIED TO GIVE ASSESSMENTS IN THEIR SCHOOLS.WE CAN RELY ON LESS-CORRUPTIBLE FORMATS. MULTIPLE CHOICE IS PRETTYEASY TO GAME, BUT IT'S MUCH MORE DIFFICULT TO ERASE A STUDENT'SESSAY AND WRITE A NEW ONE FOR THEM, SO THE MORE WE RELY ONCONSTRUCTED-RESPONSE FORMATS, I THINK...PEOPLE SAY THERE'S A COSTTHERE. YES, THERE IS A COST THERE, AND I'M SORRY, BUCK UP. IT'S ACOST, OK?WE SPEND A TINY AMOUNT—THIS IS ANOTHER FALLACY. WE SPEND A TINYAMOUNT IN EDUCATION ON ASSESSMENT OF STUDENT OUTCOMES. IT'S ALWAYSBLOWN OUT OF PROPORTION, ALL THIS TIME AND EXPENSE WE DO ONTESTING. IT'S TINY COMPARED TO THE EDUCATION BUDGET. THE AMOUNT OFMONEY WE SPEND TO ENSURE THE INTEGRITY OF THOSE DATA IS EVEN47TINIER, SO IT'S ABOUT TIME WE PONY UP SOME MONEY TO DO A BETTER JOBON ASSESSMENT. LAST THING THERE SAID COMPUTER-BASED DELIVERY OFASSESSMENTS IS LESS SUSCEPTIBLE TO CORRUPTION, AS WELL.FINALLY, SOME NARROWER, MORE TECHNICAL THINGS, I THINK, TESTPROVIDERS OR CONTRACTORS CAN DO--SIMPLE STUFF, LIKE REQUIRINGSEATING CHARTS. I RECENTLY LEARNED THIS REALLY SEXY TERM--"BATCHHEADER." APPARENTLY, BATCH HEADER IS WHAT IDENTIFIES WHO MONITOREDTHE ANSWER SHEETS THAT ARE ABOUT TO BE PROCESSED BY THE SCANNINGAND SCORING SYSTEMS. IT'S IMPORTANT TO KNOW WHO WAS RESPONSIBLE FORADMINISTERING A TEST, SO A SIMPLE ADDITION TO A TEST DEVELOPER'SAND STATE'S PROCEDURES OF INCLUDING BATCH HEADERS IS REALLY ANIMPORTANT STEP.CHAIN OF CUSTODY--KNOW WHO HAD TEST MATERIALS AT ALL TIMES. DELIVERMATERIALS JUST IN TIME AT THE DISTRICT AND GET THEM OUT OF THESCHOOLS JUST IN TIME. REQUIRE NONDISCLOSURE AGREEMENTS ANDSIGNED STATEMENTS THAT NOT ONLY SAY, "I'M WILLING TO UPHOLD THERULES," BUT, "I UNDERSTAND WHAT THE PENALTIES ARE." TOO OFTEN,WE LEAVE OFF THAT THERE'S SOMETHING SERIOUS THAT CAN HAPPEN IF YOUDON'T DO THISRIGHT. TEST DEVELOPERS CAN ALSO DO MORE PURPOSEFUL TEST DESIGN INTERMS OF WHERE THE PLACEMENT OF FIELD TEST BLOCKS ARE AND48SCRAMBLING OF FORMS AND THINGS LIKE THAT, AND TEST DEVELOPERS CANALSO DO A BETTER JOB OF CONSULTING WITH STATES AND DISTRICTS ONMETHODS OF DETECTION.WHEN I WAS INVOLVED IN A RECENT ISSUE, WE ACTUALLY WENT AND LOOKEDAT ANSWER SHEETS, AND THE SCANNERS ARE SET PRETTY CONSERVATIVELY.WHEN YOU LOOK AT ANSWER SHEETS BY HUMAN EYE, YOU SEE A LOT MOREERASURES. ON AVERAGE, WHAT WE SAW WAS ABOUT 10% MORE ERASURES THANTHE SCANNERS DETECTED, AND IT'S EASY TO SEE. SO THOSE RESPONSIBLEFOR TESTING NEED TO CONSULT WITH FOLKS ABOUT THEIR DETECTIONMETHODS.FINALLY, I'LL WRAP UP, NEARLY, WITH POLICY DEVELOPMENT.PROFESSIONAL ASSOCIATIONS HAVE A ROLE HERE, AS WELL. I'M UNDER THEIMPRESSION--WAYNE, MAYBE YOU CAN CONFIRM THIS--BUT OURPROFESSIONAL ASSOCIATION, THE NATIONAL COUNCIL ON MEASUREMENT INEDUCATION, WORKED AS FAST AS A PROFESSIONAL ASSOCIATION COULDPOSSIBLY WORK TO DEVELOP A DRAFT MODEL POLICY ON TEST INTEGRITY TODELIVER TO THE U.S. DEPARTMENT OF EDUCATION.YOU SHOULD HAVE THAT NOW. IF NOT, IT IS NOW READY FOR YOU, AND WEWILL DELIVER IT TO YOU. WE'RE PLEASED TO PROVIDE THAT, AND I THINKIT SHOULD, WE HOPE, BECOME SOMETHING THAT DISTRICTS AND STATES CANRELY ON, AND THE U.S. DEPARTMENT OF ED, AS SOME POLICY GUIDANCE.49FINALLY, JUST SOME CONCLUDING THOUGHTS. SHOULD WE BE SURPRISEDTHAT CHEATING IS OCCURRING? NO. IT HAPPENS EVERYWHERE. EVERYBODYCHEATS ON EVERYTHING. SOME SMALL PERCENTAGE OF PEOPLE WILL CUTCORNERS WHEREVER THERE ARE INCENTIVES TO DO SO.SHOULD WE BE SHOCKED? ABSOLUTELY. YOU KNOW, WHERE BARRY BONDS CANSAY, "I TAKE STEROIDS, BUT YOU DON'T PAY ME TO BE A ROLE MODEL FORKIDS. YOU PAY ME TO HIT HOME RUNS, FILL SEATS, AND MAKE A LOT OFMONEY FOR MY TEAM AND ENTERTAIN PEOPLE," HE'S ABSOLUTELY RIGHT.EDUCATORS, HOWEVER, CAN'T SAY THAT. WE, BY AND LARGE, ARE CHARGEDWITH BEING ROLE MODELS, SO WE SHOULD BE SHOCKED AT THAT.WHAT'S A WRONG REACTION? "DO AWAY WITH TESTS." TOO MANY PEOPLE ARESAYING, "OH, TESTS HAVE CONSEQUENCES," AND "OH, THAT'S TERRIBLE.WE SHOULD DIMINISH THE CONSEQUENCES. WE SHOULD LESSEN OUR RELIANCEON TESTING."I THINK THAT'S ABSURD. I THINK THAT'S LIKE SAYING IF THERE'S VOTERFRAUD, WE SHOULD DO AWAY WITH FREE ELECTIONS. NOBODY WOULD CONSIDERTHAT, AND DOING AWAY WITH THE SOUND INFORMATION WE GET FROM TESTSAND THE CONSEQUENCES ASSOCIATED WITH THEM IS PROBABLY THE MOSTWRONG RESPONSE. THE BEST RESPONSE IS PREVENTION, AND THE GOOD NEWS50IS THERE IS PLENTY THAT CAN BE DONE. THERE ARE A LOT OF METHODS TOPREVENT CHEATING AND TEST IRREGULARITIES.AND FINALLY, I'LL JUST END WITH, IT'S A SHARED RESPONSIBILITYAMONGST STUDENTS, EDUCATORS, PROFESSIONAL ASSOCIATIONS, AND ALLINVOLVED IN THE ENTERPRISE OF TESTING.JACK BUCKLEY: ALL RIGHT. THANKS VERY MUCH.SCOTT.SCOTT NORTON: I THINK I'M UP. I ALSO WANT TO SAY THANK YOU TO THEDEPARTMENT FOR INVITING ME TO PARTICIPATE. I APPRECIATE THAT VERYMUCH.I'M FROM LOUISIANA. I'M HERE REALLY, I THINK, TO REPRESENT THE SEAPERSPECTIVE. I THINK LOUISIANA IS PROBABLY PRETTY TYPICAL OFSTATES THAT ARE FAIRLY INVOLVED IN TEST SECURITY, AND I'LL TAKE ASTAND ON THAT AND SAY STATES SHOULD BE INVOLVED IN TEST SECURITY.IF WE'RE RESPONSIBLE FOR THE TESTS, WE'RE RESPONSIBLE TO MAKE SURETHAT THE RESULTS ARE VALID AND RELIABLE. WHAT THAT MEANS FOR USMOSTLY IS DEVELOPMENT OF STATE POLICY THROUGH THE STATE BOARD, ANDI'LL TALK ABOUT THAT JUST A LITTLE BIT, AND THEN FOLLOW-UP ON THAT51POLICY, WORKING WITH THE DISTRICTS AND SCHOOLS. AS ALREADY STATED,IT IS A SHARED RESPONSIBILITY AMONG ALL THOSE PARTIES.SO WHAT DOES THAT MEAN? WHAT DOES A POLICY LOOK LIKE? I'LL JUSTGIVE YOU A FEW OF OUR HIGHLIGHTS.WE DO REQUIRE THAT SCHOOL DISTRICTS DEVELOP THEIR OWN DISTRICT TESTSECURITY POLICY. IT HAS TO INCLUDE EVERYTHING THAT WE REQUIRE.THERE ARE SOME LOCAL THINGS THAT WE ASK THEM TO PUT IN, AND THEYSUBMIT THOSE TO THE STATE. WE REQUIRE OATH OF SECURITY, AS WASMENTIONED EARLIER. WE HAVE DONE THAT FOR SOME TIME FOR THE ADULTSINVOLVED IN TEST ADMINISTRATION. IN RECENT YEARS, WE'VE ALSO ASKEDTHE STUDENTS TO SIGN AN OATH OF SECURITY. WHETHER OR NOT THAT'SHELPFUL, WE'LL PROBABLY DISCUSS THAT MORE THROUGHOUT THE DAY.WE REQUIRE CERTAIN PROCEDURES FOR THE HANDLING OF TEST MATERIALS.THAT'S ALSO BEEN DISCUSSED A LITTLE BIT ALREADY. OUR STATE POLICYALLOWS THAT THE STATE SUPERINTENDENT CAN CANCEL OR VOID SCORES IFTHERE'S INAPPROPRIATE FINDINGS, EVEN IF THE DISTRICT DISAGREES, SOTHAT'S WRITTEN INTO OUR POLICY.THE POLICY REQUIRES SEVERAL SPECIFIC TYPES OF ANALYSES. IT DOES NOTREQUIRE SOME OTHERS THAT I'LL MENTION IN JUST A MINUTE. WE REQUIREERASURE ANALYSIS AND PLAGIARISM, AND I'LL MENTION THOSE IN THE NEXT52SLIDE OR TWO. AND THEN THERE ARE PROCEDURES THAT TELL DISTRICTSWHAT TO DO--WHAT TO DO IF YOU FIND A STUDENT THAT MAY BE INVOLVEDIN SOME TEST SECURITY BREACH, WHAT TO DO IF THERE'S AN ADULTINVOLVED, AND SO FORTH. SOME OF THAT IS SPELLED OUT IN THE POLICY.REFERRED TO IN THE POLICY BUT A LITTLE MORE HANDS-ON, WE DO QUITE ABIT OF STATE TEST MONITORING. LITERALLY, THAT'S JUST WHAT IT SOUNDSLIKE. WE GET IN OUR CARS AND DRIVE AROUND ACROSS THE STATE. WEPROVIDE TRAINING TO THE STATE DEPARTMENT STAFF WHO DO THEMONITORING, GIVE THEM A LITTLE BACKGROUND ON WHAT THEY SHOULD BELOOKING FOR. THERE'S A FORM THEY FILL OUT. THOSE ARE REVIEWED ONCETHEY RETURN AND SORTED INTO VARIOUS CATEGORIES OF OFFENSES, FROMMINOR TO MORE SEVERE.WHO DO WE GO SEE? WELL, WE GO SEE ANYBODY WHO HAS A PRIOR PROBLEM.WE SINGLE THOSE OUT. YOU CAN EXPECT A VISIT FROM THE STATE IF YOUHAD AN ISSUE THE YEAR BEFORE. THE REST OF THE VISITS ARE RANDOM. WETRY TO MAKE SURE WE ARE IN EVERY SCHOOL DISTRICT, JUST SO THEY KNOWWE'RE COMING. THE VISITS ARE ANNOUNCED IN THAT WE REMIND THEM WE'RECOMING, BUT UNANNOUNCED IN THAT WE DON'T SAY SPECIFICALLY WHICHSCHOOLS WE'RE COMING TO.SO HOW MANY SCHOOLS CAN THE STATE DEPARTMENT COVER? IT'S A BIGSTATE. ABOUT 1,400 PUBLIC SCHOOLS. WE GET TO ABOUT 250-400 SCHOOLS53EVERY YEAR. I THINK OUR LINEUP FOR THIS SPRING, WE'RE GOING TO BEIN ABOUT 300, 325 SCHOOLS. IT TAKES ABOUT 125 PEOPLE TO DO THAT, SOTHAT IS QUITE AN EFFORT, BUT THAT HAS BEEN A GOOD THING FOR US.JUST A COUPLE OF WORDS ABOUT PLAGIARISM, AND THEN I'LL MENTIONERASURE ANALYSIS ALSO.PLAGIARISM, WE DO ASK THE VENDOR TO LOOK FOR CERTAIN TYPES OF JUSTWHAT YOU WOULD EXPECT--REPETITIVE INFORMATION ACROSS THE STUDENTRESPONSES. BACK IN THE DAYS OF PAPER, WHEN THEY WERE LOOKING AT THEACTUAL PAPERS, WE HAD TO MAKE SURE THEY WERE BATCH HEADERS TOGETHERSO THAT IF THERE WAS A PATTERN, IT COULD BE SEEN. THAT'S DONE VIASCANNER NOW, BUT IT'S THE SAME THING. WE NEED TO TRY TO MAKE SURETHAT THE REPONSES ARE LOOKED AT BY SOME OF THE SAME INDIVIDUALS.WE FOLLOW THAT UP AT THE STATE LEVEL WITH A COUPLE OF LAYERS OFREVIEW BEFORE SCORES ARE CANCELED, BUT AFTER THOSE REVIEWS, WE DOIN FACT CANCEL SCORES OR VOID SCORES BASED ON PLAGIARISM. I'LLPAUSE FROM THE SLIDE JUST A SECOND AND SAY, I THINK IF YOU ONLYREAD THE ARTICLES ABOUT TEST SECURITY, YOU MIGHT BE LED TO BELIEVETHAT ERASURE ANALYSIS IS TEST SECURITY. AS HAS BEEN MENTIONED HEREA FEW TIMES ALREADY, IT IS JUST ONE TOOL. IT'S A GOOD ONE, AND WEUSE IT IN LOUISIANA, BUT IT REALLY IS JUST ONE THING THAT WE CANDO.54WE DO TYPICALLY LOOK FOR WRONG-TO-RIGHT ERASURES AND PROVIDE ATHRESHOLD AND LOOK FOR CLASSROOMS WHERE THAT THRESHOLD ISEXCEEDED. THERE'S A COUPLE MORE WORDS HERE, BUT I THINK THE POINT IREALLY WANT TO MAKE IS THAT WHILE WE DO ERASURE ANALYSIS, AND OTHERSTATES DO THE SAME, THE SPECIFIC WAY THAT WE DO IT IS VERYHOMEGROWN. IT'S LOUISIANA-SPECIFIC. WE CAME UP WITH THOSE RULESOURSELVES, AND SHOULD THERE BE MORE STANDARDIZATION THERE?SOMETHING I THINK WE SHOULD PROBABLY TALK ABOUT.WE ALSO ASK THE DISTRICTS TO DO QUITE A BIT. THEY ARE THERE WHERETHE TESTS ARE GIVEN. THEY ARE CLOSEST TO THE SOURCE OF THE PROBLEMIF THERE IS ONE, AND WHAT THEY ARE ABLE TO DO IS VOID THE SCORESTHEMSELVES. WHILE THE STATE IS IN CHARGE OF ERASURE ANALYSES ANDPLAGIARISM, THE DISTRICTS CAN FIND MANY OR LOTS OF OTHER THINGS,AND IF THEY DO, THEY ARE ABLE TO REPORT UP TO THE STATE AND CANCELOR VOID THOSE SCORES, AS WELL. I TALKED FAST SO I COULD LEAVE ACOUPLE OF MINUTES AT THE END TO SAY, WHILE WE DO A LOT, I THINKLOUISIANA IS PROBABLY, AS I SAID, TYPICAL, AND MAYBE A LITTLE MOREINVOLVED THAN SOME STATES.I THINK THERE'S A COUPLE OF AREAS THAT WE WOULD LIKE TO IMPROVE ON,AND SOME OF THESE HAVE ACTUALLY BEEN MENTIONED ALREADY. CHANCELLORHENDERSON--I WROTE IT DOWN. SHE SAID SHE SPENT A LOT OF TIMEDETERMINING WHAT ACTUALLY HAPPENED. WELL, TRUER WORDS, RIGHT? WE55SPEND A LOT OF TIME BETWEEN THE STATE AND DISTRICTS TRYING TODETERMINE WHAT ACTUALLY HAPPENED, AND WE REALLY RELY ON THEDISTRICTS HEAVILY TO HELP US DETERMINE WHAT ACTUALLY HAPPENED. ANDTHAT'S A GOOD THING, BECAUSE THEY'RE THERE WHERE THE SOURCE OF THEPROBLEM IS, BUT IN CASES WHERE THE DISTRICT IS NOT AS INTERESTED ORNOT AS CAPABLE OF DOING THAT, I THINK, AT LEAST IN OUR STATE, OURPOLICY IS A LITTLE BIT LIGHT ON THEN WHAT THE STATE SHOULD DO TOINTERVENE AND PROVIDE MORE OVERSIGHT IN THAT INVESTIGATION, ORPERHAPS EVEN BE INVOLVED IN THOSE INVESTIGATIONS.FOR VERY ESTABLISHED PROCEDURE, SUCH AS ERASURE ANALYSIS—AND I'VEALLUDED TO THIS ALREADY--I THINK THERE PROBABLY COULD BE MORECONSISTENCY IN HOW THAT'S DONE. LIKE I SAID, WE HAVE RULES. WE HAVEVERY SPECIFIC RULES BETWEEN OUR STATE AND THE VENDOR, BUT THOSE AREREALLY HOMEGROWN RULES, AND THERE PROBABLY, PERHAPS, COULD BE ABETTER WAY OR MORE STANDARDIZATION.THIRDLY, IT'S REALLY THE SAME POINT, JUST APPLIED TO THE OTHERTYPES OF ANALYSES, OTHER THAN ERASURE ANALYSIS. WE'VE WORKED WITHCAVEON, WE'VE TALKED TO OTHER STATES, WE'VE DONE LOTS OF THINGS,AND WHAT I'VE FOUND IS, IF YOU LOOK AT AREAS OUTSIDE OF ERASUREANALYSIS--FOR EXAMPLE, UNUSUAL GAINS OR LOSSES--THERE MAY BE TOOLSTO DO THAT ANALYSIS, BUT EVEN ONCE YOU HAVE THE ANALYSIS, IT'S VERYDIFFICULT TO KNOW WHAT TO DO NEXT.56AND AGAIN, CHANCELLOR HENDERSON SAID IT VERY WELL--THERE ARE NOWIDELY ACCEPTED PRACTICES ABOUT THOSE KINDS OF THINGS, SO EVEN IFYOU DO THOSE THINGS, IT'S A LITTLE BIT UNCLEAR AT THE STATE LEVELWHAT YOU SHOULD DO WITH THAT INFORMATION AND WHAT RISES TO THELEVEL OF INTERVENTION. PERHAPS THE NCME STANDARDS WILL BE A GOODSTEP IN THIS DIRECTION, BUT I DO SUPPORT THE DEVELOPMENT OF SOMEWIDELY ACCEPTED TEST SECURITY PRACTICES. THAT WOULD BE VERYHELPFUL TO THE STATE. THANK YOU.JACK BUCKLEY: THANKS VERY MUCH. JIM.JIM LIEBMAN: THANK YOU, JACK.I'M GOING TO TALK FASTER THAN SCOTT, I PROMISE YOU, TO TRY TO GETWHAT I NEED TO SAY OR WANT TO SAY HERE.FIRST OF ALL, I THINK IT'S REALLY IMPORTANT TO--LET'S SEE IF I CANGET THIS ADVANCED HERE; THERE--TO AVOID A COUPLE OF FALLACIES THATPEOPLE HAVE ALREADY TALKED ABOUT. ONE FALLACY IS THAT BECAUSECHEATING CAN OCCUR, IT WILL OCCUR, AND IT NEUTRALIZES THE VALUE,THE MANY VALUES OF TESTING THAT PEOPLE HAVE TALKED ABOUT. THAT'SNOT TRUE, AND AS WE ARE HEARING, THERE ARE MANY THINGS THAT YOU CANDO ABOUT IT.57BUT THERE'S THE REVERSE FALLACY, WHICH IS TO THINK THAT BECAUSE ALOT OF PEOPLE WHO RAISE THE CHEATING ISSUE REALLY DON'T CARE ABOUTCHEATING, THEY JUST DON'T LIKE TESTING OR THEY DON'T LIKEACCOUNTABILITY MUCH, CAN LEAD YOU TO THINK THAT THERE REALLY ISN'TAN ISSUE THERE, AND THERE IS AN ISSUE THERE, AND YOU NEED TO TAKEIT SERIOUSLY. WHEN YOU PUT STAKES ON OUTCOMES, PEOPLE WILL REACT TOTHOSE STAKES, AND ONE OF THOSE REACTIONS CAN BE CHEATING. I THINKTHE TWO BASIC ANTIDOTES TO THOSE, BEYOND ALL OF THE SPECIFICS THATWE'RE GETTING INTO, IS, INFORM YOUR EDUCATORS AND DEMONSTRATETHROUGH YOUR ACTIONS THAT YOU REALLY DO CARE ABOUT CHEATING, AND IFYOU HEAR ABOUT IT, YOU WILL RESPOND TO IT, AND YOU'RE CONSTANTLYTAKING STEPS TO AVOID IT, AND IT'S REALLY IMPORTANT TO SAY THE SAMETHING TO THE PUBLIC SO THAT THE PUBLIC UNDERSTANDS.SO NOW, JUST TO GET INTO A COUPLE OF THE SPECIFIC THINGS WE DO,I'VE USED, IN THE HEADERS HERE, PROFESSOR AMREIN-BEARDSLEY'STAXONOMY, WHICH I THINK IS VERY HELPFUL, BUT TO GET AT THE REALLYSYSTEMATIC CHEATING, A COUPLE OF THINGS TO DO.FIRST, TEST ADMINISTRATION. HERE, KEEP THE TEST WINDOW SHORT. INNEW YORK STATE, THE TEST WINDOW IS LIKE TWO WEEKS. IN NEW YORKCITY, WE KEPT IT TO A DAY OR TWO TO MAKE SURE THAT PEOPLE AREN'TPASSING THE BOOKS AROUND OR TALKING TO EACH OTHER OR WHATEVER. AND58THEN JUST MAKE SURE THAT THE MATERIALS ARE IN THE SCHOOLS FOR THELEAST AMOUNT OF TIME NECESSARY.WE SHRINK-WRAP EVERYTHING FOR A SCHOOL INTO ONE PACKAGE. INSIDETHAT PACKAGE IS A SHRINK-WRAPPED PACKAGE FOR EACH CLASSROOM, AND INTHAT SHRINK-WRAPPED PACKAGE FOR EACH CLASSROOM, WE DO SOMETHINGWITH THE ELEGANT NAME OF PRE-SLUGGED, WHICH MEANS THAT STUDENTS'NUMBERS ARE ALREADY PUT ONTO THE SHEET. THOSE BIG PACKAGES FOR THESCHOOL ARE OPENED WITHIN AN HOUR OF THE START TIME, AND THEPACKAGES FOR EACH CLASSROOM ARE OPENED WITHIN 15 MINUTES OF STARTTIME, AND WE DO MONITORING FROM THE DISTRICT, IN ADDITION TO SOMEMONITORING DONE BY THE STATE, TO BE THERE WHEN ALL OF THAT ISSUPPOSED TO HAPPEN AND MAKE SURE THAT IT DOES HAPPEN ON TIME-AGAIN, UNANNOUNCED MONITORING. WE ONLY GET TO ABOUT 10% OF THESCHOOLS, BUT MAKE IT VERY, VERY VISIBLE THAT THAT'S ALL HAPPENING,AND THEN OF COURSE GET THOSE MATERIALS OUT OF THE SCHOOL AS QUICKLYAS POSSIBLE. EVERYBODY WHO HANDLES THE MATERIAL HAS TO SIGNSOMETHING SAYING "I GOT X NUMBER, AND I RETURNED X NUMBER, AND INBETWEEN I DID NOTHING ELSE WITH THEM. I DIDN'T COPY THEM. I DIDN'TDISTRIBUTE THEM IN ANY WAY." THAT'S ALL SIGNED AND CERTIFIED, ANDIF THINGS ARRIVE LATE, THE PRINCIPALS HAVE TO EXPLAIN WHY THATHAPPENED.59ON THE SCORING SIDE OF THIS, I THINK ALSO VERY IMPORTANT TO SCOREOFF-SITE. IF YOU DO USE JUST THE MULTIPLE CHOICE, THAT'S EASY. YOUCAN SCAN THEM OUT OF THE SCHOOL REALLY QUICKLY AND GET THEM SCOREDELECTRONICALLY ELSEWHERE. AS WAS SUGGESTED, WE IN NEW YORK USE ALOT OF CONSTRUCTED RESPONSE, ESSAYS, AND ALL OF THAT. ALL OF THOSEARE GRADED BY TEACHERS. REALLY IMPORTANT TO GET THAT GRADINGOUTSIDE OF THAT SCHOOL AND INTO ANOTHER SCHOOL, SOMETHING WE'VEBEEN DOING WITH THE ELEMENTARY AND MIDDLE BUT JUST HAVE STARTEDDOING IN HIGH SCHOOL, AND IT'S REALLY WAY LATE FOR THAT TO HAVEHAPPENED, BUT FINALLY IT IS HAPPENING. EVERYTHING THAT IS READ ISREAD BY 2 OR 3 TEACHERS AT LEAST, AND THERE'S A COMPARISON BY ATABLE LEADER OF THE EXTENT TO WHICH THERE'S CORRESPONDENCE BETWEENWHAT THEY'RE GETTING.WHERE THERE ISN'T CORRESPONDENCE, THERE'S STEPS THAT ARE TAKEN.THE CITY OF NEW YORK HAS JUST DISTRIBUTED A DISTRIBUTED SCORING RFPTHAT YOU MIGHT WANT TO TAKE A LOOK AT, BECAUSE I THINK IT'S PRETTYMUCH STATE OF THE ART, WHAT'S BEING ASKED FOR. WE DON'T KNOW YETWHAT THEY'RE GOING TO GET BACK, BUT ESSENTIALLY NOT TO HAVESOMEBODY DO THE SCORING, BUT TO HAVE SOMEBODY COME IN AND DO ALL OFTHE DISTRIBUTION, PICKING UP, EVERYTHING SCANNED, INCLUDING THECONSTRUCTED RESPONSES, SO THEY GO IMMEDIATELY INTO THE MACHINESAND CAN'T BE TAMPERED WITH AND CAN BE DISTRIBUTED WHEREVER YOU WANTTHEM TO BE GRADED AND CAN HAVE INSTANT, AUTOMATIC, CITY-WIDE60COMPARISON OF HOW EACH SCORE COMPARES TO SCORES ELSEWHERE IN THECITY SO THAT YOU CAN BE SURE YOU'RE GETTING INTER-RATER RELIABILITYAND THE LIKE.FOR BOTH FIRST- AND SECOND-DEGREE CHEATING, WHERE IT'S GETTING ALITTLE BIT MORE HAPHAZARD AND MAYBE NOT EVEN INTENTIONAL, WE DOHAVE A TEST COORDINATOR ASSIGNED IN EACH SCHOOL. THAT'S SOMEBODYWHO'S REALLY RESPONSIBLE FOR THIS AND CAN SORT OF WORK WITH THEPRINCIPAL, BUT ALSO MAKE SURE THAT NOT TOO MUCH PRESSURE ANDRELIANCE IS PUT ON THE PRINCIPAL.WE HAVE A TERRIFIC TEST ADMINISTRATION HANDBOOK. I SEE, I THINK,HERE IN THE AUDIENCE THE ORIGINAL AUTHOR OF THAT, WHO'S NOW INBALTIMORE. IT'S A REALLY VERY CLEAR BOOK THAT EVERYBODY IS SUPPOSEDTO READ. ANYONE WHO'S GOING TO PROCTOR OR TEACH, BE INVOLVED IN THETEST, HAS TO SIGN THAT THEY'VE READ IT, THEY UNDERSTAND IT, THEYKNOW WHAT THE PENALTIES ARE FOR VIOLATING.ONE POINT ON HERE THAT'S REALLY IMPORTANT--TEACHERS ARE ALLIES. MYSENSE IS THAT TEACHERS REALLY DON'T LIKE CHEATING, FOR THE MOSTPART, AND WHEN THEY SEE IT, THEY'RE OFFENDED BY IT. IF YOU MAKE AWAY TO ENABLE THEM TO PROVIDE INFORMATION ABOUT THAT, THEY WILL.CHANCELLOR WALCOTT RECENTLY JUST SAID EMAIL HIM. WE'VE GOT LOTS OF61WAYS YOU CAN DO THAT IN NEW YORK, BUT YOU CAN ALSO EMAIL THECHANCELLOR.SO I THINK YOU PROVIDE ANONYMITY WHERE YOU NEED TO. OFFERCONFIDENTIALITY WHERE THAT IS HELPFUL. PROCTORING IS A REALLYIMPORTANT POINT. AS WAS POINTED OUT ALREADY, WE HAVE A LOT OFMATERIALS ABOUT THAT, FAQS AND THINGS LIKE THAT. BASICALLY, THETEACHER SHOULD TALK FROM THE FRONT OF THE ROOM ABOUT WHAT THEPROCEDURES ARE, SHOULD CIRCULATE, BUT WHEN CIRCULATING, SHOULD NOTHAVE PENS, PENCILS, ANY MATERIALS AT ALL IN HER HAND, SHOULD TRY TOKEEP THE CONVERSATION WITH STUDENTS TO A MINIMUM AND FROM THE FRONTOF THE ROOM, AS OPPOSED TO WITH INDIVIDUALS. THIS IS ANOTHER THINGTHAT WE MONITOR WHEN WE GO AROUND WITH THIS UNANNOUNCED MONITORING.EVERY ROOM WHERE A TEST IS TAKING PLACE IS EITHER SUPPOSED TO HAVEWINDOW IN THE DOOR THAT IS NOT BLOCKED, OR THE DOOR IS AJAR ENOUGHSO THAT THE MONITOR CAN LOOK IN AND SEE WHAT'S TAKING PLACE TOMONITOR THE PROCTORING.I'M NOT GOING TO TALK SO MUCH ABOUT--THESE ARE SOME OPEN QUESTIONS.A LOT OF PEOPLE HAVE TALKED ABOUT THE EXTENT TO WHICH ERASUREANALYSIS IS VALID AND USEFUL, BUT THE EXTENT TO WHICH IT IS NOT ANDTHE INTER-RATER RELIABILITY ANALYSES, ISSUES ABOUT WHETHER62PROCTORS SHOULD BE FROM THE CLASSROOM OR NOT--IN OTHER WORDS, THETEACHER IN THAT CLASSROOM OR NOT, AND AT WHAT AGE YOU CAN CHANGETHAT.I WANTED TO TALK JUST A MOMENT ABOUT THIRD-DEGREE CHEATING BECAUSEI THINK IT GETS LESS ATTENTION THAN IT DESERVES, AND IT ISSOMETHING THAT CAN BE DEALT WITH. IF YOU GIVE TESTS OVER TIME, YOUCAN EXPECT THAT THERE WILL BE SLIPPAGE AND THAT THE TESTS WILL GETEASIER OR THAT THE SCORES WILL GET HIGHER OVER TIME. THIS IS ACOMBINATION OF ALL OF THE OTHER KINDS OF CHEATING, AS WELL AS THESLIPPAGE IN TERMS OF TEST ITEMS AND TEST PREP AND THINGS LIKE THAT.THERE'S JUST WAYS TO LOOK OUT FOR THIS AND RESPOND TO THIS. I'VECALLED THEM THE KORETZ TESTS BECAUSE DAN KORETZ AT HARVARD HASDEVELOPED SOME OF THESE. ESSENTIALLY WHAT YOU DO IS YOU ANCHOR YOUREXPECTED OUTCOMES TO SOMETHING THAT IS NOT SUBJECT TO CHEATING,LIKE HERE THE A.C.T. YOU CAN USE NAEP. YOU CAN USE GRADUATIONRATES. THEN YOU TRY TO SEE IF IMPROVEMENTS IN TEST SCORESARE MATCHED BY IMPROVEMENTS IN OTHER THINGS THAT SHOULD IMPROVESOMEWHAT SIMILARLY.THIS IS FROM KENTUCKY WAY BACK IN THE NINETIES. IT'S NOT TO VILIFYANYBODY, BUT YOU CAN SEE THAT THERE WAS A TON OF SLIPPAGE HERE INTERMS OF WHAT THE STATE TEST WAS SHOWING--THE DOTTED LINE AT THE63TOP--AND THE A.C.T. TEST WAS PRETTY STABLE OR GOING DOWN EVEN, ANDTHE DIFFERENCE BETWEEN THE TWO IS SUGGESTIVE OF A LOT OF SLIPPAGETHAT SHOULD RAISE SOME PROBLEMS. WE ASKED DAN KORETZ TO COME INTONEW YORK CITY TO SEE AND ESSENTIALLY TO ANCHOR--WE LOOKED AT OUREIGHTH-GRADE TEST SCORES AND LOOKED TO SEE IF THEY PREDICTEDGRADUATION RATES 4 YEARS LATER SIMILARLY FROM ONE YEAR TO THE NEXT.IF YOU START TO SEE THAT THE SAME GRADE PREDICTS A LOWER GRADUATIONRATE, THEN YOU'VE GOT SLIPPAGE. I'M HAPPY TO SAY THAT THERE WASN'TMUCH FOUND. THIS WAS THE WORST OF IT, EIGHTH-GRADE MATH. YOU CANSEE AT THE TOP, FOR THE HIGHEST-PERFORMING STUDENTS, IN ONE YEAR,THE LAST OF THE 3 YEARS, THERE WAS A LITTLE BIT LOWER GRADUATIONRATE THAN YOU WOULD HAVE EXPECTED. THIS IS NOT A SERIOUS PROBLEMHERE, BUT IT CERTAINLY WAS ONE THAT STARTED US THINKING ABOUT WHATWE COULD DO ABOUT IT, SO THERE ARE WAYS TO MONITOR THIS, AND IENCOURAGE PEOPLE TO THINK ABOUT THOSE.JACK BUCKLEY: THANK YOU VERY MUCH.SO I'D LIKE TO SHIFT TO A MODERATED DISCUSSION JUST WITH THE PANELBEFORE I OPEN IT UP TO THE BROADER AUDIENCE. IN CASE YOU DIDN'TNOTICE, THERE'S AN AWFUL LOT THAT JUST GOT COVERED. IF I COULDTHINK OF A PLACE TO START, I GUESS IT WOULD BE WHETHER OR NOT WE CANREACH SOME KIND OF CONSENSUS ON--I THINK WE STARTED TO PARTITION64THE WORLD INTO DIFFERENT TYPES OF CHEATING, OR AS GREG POINTED OUT,CHEATING DOES NOT EQUAL IRREGULARITIES, WHICH I THINK IS ALSOANOTHER IMPORTANT POINT.FROM THE STANDPOINT OF THE DEPARTMENT OR NCES, WE WERE THINKINGREALLY MORE ABOUT, AUDREY, WHAT YOU CALL FIRST- AND SECOND-DEGREECHEATING OR IRREGULARITIES, ALTHOUGH THE THIRD DEGREE IS VERYINTERESTING FROM A POLICY PERSPECTIVE. IF WE CAN STAY--WHAT AREYOUR THOUGHTS AS A PANEL ON WHETHER OR NOT IT'S FAIR TO SAY THATSOMETHING LIKE THE FIRST DEGREE, WHAT WE'RE REALLY TALKING ABOUTWITH PREVENTION, ARE OPERATIONAL MEASURES--THINGS LIKE CHAIN OFCUSTODY, SHRINK-WRAPPING. I WOULD PUT PROCTORING IN THAT CATEGORY.THE SECOND DEGREE REALLY IS MORE, I THINK, WHAT A COUPLE OF FOLKSTOUCHED ON AS CULTURE, AS SOMETHING WHERE YOU'VE GOT TO PUT OUTLIKE SCOTT'S PROCEDURES OR POLICIES AND WHERE WE NEED TO THINKABOUT THE ENTIRE SYSTEM OF ASSESSMENT AND NOT JUST SORT OF, "WHATDID I DO FOR THIS OPERATIONAL STEP?"DOES THAT SEEM LIKE A FAIR TYPOLOGY, OR WOULD SOMEBODY SUGGESTSOMETHING DIFFERENT OR MORE USEFUL?GREG CIZEK: I MIGHT SAY THAT THE CULTURE ISSUE WOULD CUT ACROSS ALL 3OF THOSE LEVELS. I THINK THERE'S A SENSE THAT SOMEHOW--IT'S65SOMEWHAT OF AN UNFAIRNESS. 179 DAYS OF THE YEAR, YOU ASK A TEACHERTO BE AN ADVOCATE FOR A KID, YOU ASK PRINCIPALS TO BE FOR THE KID'SSUCCESS AND ALL THAT KIND OF THING, AND THEN ON A DAY WHEN THERE'SA HIGH-STAKES TEST GIVEN, YOU ASK THEM TO BE AN INVIGILATOR, APROCTOR, SOMEONE WHO IS NOT THERE ADVOCATING FOR THE GREATESTSUCCESS FOR THE STUDENT. THAT CULTURE CUTS ACROSS AND ENGENDERS ALL3 OF THOSE LEVELS OF INAPPROPRIATE PRACTICE.JIM LIEBMAN: I WOULD AGREE WITH THAT. I THINK THERE'S A PROPORTION OFBOTH. I THINK YOUR PROPORTIONS ARE RIGHT, THAT PROCEDURES ANDOPERATIONS ARE MORE IMPORTANT FOR THE FIRST DEGREE, BUT I THINKCULTURE CUTS ACROSS THERE AS WELL, AND I THINK, EVEN AT THECLASSROOM LEVEL, ONE OF THE THINGS THAT OUR HANDBOOKS DO THAT AREREALLY GOOD IS JUST AN FAQ THAT JUST RAISES ALL THE THINGS THAT CANHAPPEN AND JUST GOES THROUGH IT, SO A TEACHER HAS READ THAT ANDSORT OF IS READY FOR, YOU KNOW, THE KID RAISES HIS HAND, HE WANTSTO GO TO THE BATHROOM. WELL, WHAT DO YOU DO ABOUT THAT? I THINKJUST TO HAVE SOME PROCEDURES THAT YOU'VE THOUGHT ABOUT BEFOREHANDEVEN FOR THAT KIND OF THING REALLY MAKES IT EASIER FOR PEOPLE TOJUST FEEL LIKE THEY'RE DOING THE RIGHT THING.SCOTT NORTON: I'LL JUST BE BRIEF. I LIKE THE TAXONOMY. I THINK THAT'SHELPFUL AND A GOOD WAY OF THINKING ABOUT IT. FOR THE SECOND LEVELIN OUR STATE, WE DO SOME OF THAT. WE IMBED THAT IN THAT NOT-SO66EASY-TO-READ TEST ADMINISTRATION MANUAL THAT--I THINK HE READ OURMANUAL—THAT HE TALKED ABOUT, BUT I THINK WE COULD PROBABLY DO ABETTER JOB OF SHOWING FOLKS HOW THAT WORKS, WEBINARS OR HANDS-ONTRAINING IN WHAT IS CROSSING THE LINE, IF YOU WILL.AUDREY AMREIN-BEARDSLEY: I'D LIKE TO ECHO WHAT GREG HAD SAID BEFOREABOUT THE ROLES OF THE TEACHER EDUCATION COLLEGES IN THIS. I THINKTHAT'S REALLY IMPORTANT. WE CAN TRAIN OUR FUTURE TEACHERS TO DO AMUCH BETTER JOB. MANY OF OUR COLLEGES OF EDUCATION HAVE DROPPED THEASSESSMENT COURSES THAT USED TO EXIST, AND SO I THINK ALSO TURNINGTHAT INTO A PROFESSIONAL RESPONSIBILITY--NOT WHERE YOU'RE A POLICEOFFICER, BUT THIS REALLY IS YOUR PROFESSIONAL RESPONSIBILITY TOLET YOUR STUDENT SHINE, AND YOU WANT TO EMBRACE THAT, VERSUS TRYINGTO CASUALLY HELP THEM AND SO FORTH. THIS IS SOMETHING THAT'S MOREOF A PROFESSIONAL NEED ON THEIR PARTS.JACK BUCKLEY: JUST TO GET BACK TO SOMETHING YOU SAID, GREG, YOUMENTIONED OR BRIEFLY TOUCHED ON AN IDEA ABOUT SOME KIND OF A THIRDPARTY QUALIFICATION TOOL OR CERTIFICATION SYSTEM, FOR LACK OF ABETTER WORD. WE'VE HEARD JIM'S EXAMPLE OF A VERY DETAILEDADMINISTRATION GUIDE, SCOTT'S EXAMPLE OF ANOTHER VERY DETAILEDSORT OF STATE LEVEL GUIDE, AND YOU ALSO BRIEFLY TOUCHED ON THE NCMEGUIDANCE, WHICH WE'D LIKE TO HEAR MORE ABOUT, I'M SURE. SO WHERE INTHOSE DIFFERENT LEVELS WOULD SUCH A QUALIFICATION OR CERTIFICATION67SYSTEM FALL? WHAT SHOULD IT COVER? WHAT WOULD YOU LIKE TO SEE INSOMETHING LIKE THAT?GREG CIZEK: I WISH IT WERE UNNECESSARY BECAUSE, LIKE AUDREY, I WISHSCHOOLS OF EDUCATION WERE DOING A BETTER JOB PREPARING PEOPLEIN THEIR PROFESSIONAL RESPONSIBILITIES. WHERE THEY'RE NOT, I THINKIT WOULD BE AN APPROPRIATE THING FOR A STATE TO WORK WITH A VENDORTO HELP WITH SOME SORT OF PROFESSIONAL DEVELOPMENT TOOL THAT WOULDACTUALLY BE REQUIRED FOR PEOPLE TO PARTICIPATE IN, IN ORDER TO BE AQUALIFIED TEST ADMINISTRATOR. IT COULD COVER EVERYTHING LIKEWHAT'S PROPER FACILITATION OF STUDENT PERFORMANCE, WHAT'SINAPPROPRIATE INTERVENTION IN STUDENT PERFORMANCE--THE ENTIREGAMUT OF THINGS COVERED IN, FOR EXAMPLE, THE NCME DOCUMENT THAT YOUHAVE, AS WELL AS OTHER GUIDANCE. I'M INTRIGUED TO SEE THE FAQ THATYOU HAVE PUT TOGETHER. I'VE NOT SEEN THAT, BUT I SUSPECT THERE'S ALOT OF GOOD THINGS IN THERE THAT COULD BE COVERED, AS WELL.AUDREY AMREIN-BEARDSLEY: IT WOULD POSSIBLY EVEN BE INCLUDED IN THECERTIFICATION TYPES OF EXAMS, AS WELL, SO WHEN THE FUTURE TEACHERSARE TESTED ON CONTENT AND PEDAGOGICAL SKILLS, IT WOULD POSSIBLY FITSOMEWHERE IN THAT TYPE OF ADMINISTRATION, AS WELL.GREG CIZEK: I ACTUALLY WANT THIS TO BE REAL FUZZY AND WARM ANDFRIENDLY, AND I WANT PEOPLE TO DO THIS IN THEIR PJS AT HOME ON THEIR68HOME COMPUTER, QUALIFYING IN THEIR OWN TIME, AND NOT HAVE IT BE ANONEROUS KIND OF THING THAT WOULD BE OPPRESSIVE BECAUSE I THINKTESTING FOLKS HAVE A HARD ROAD TO GO IN TERMS OF HELPING PEOPLE TOSEE THAT IT'S NOT AN OPPRESSIVE, BIASED, DISCRIMINATING,MARGINALIZING, CONTROLLING ACCESS TO WEALTH AND POVERTY INAMERICA, BLAH, BLAH, BLAH--YOU KNOW, ALL THAT STUFF THAT TESTS AREACCUSED OF. I THINK WE HAVE A LONG WAY TO GO TO HELP PEOPLE SEE THATTHERE ARE REALLY BENEFITS OF GETTING QUALITY INFORMATION ABOUTTESTS AND THAT EDUCATORS HAVE A ROLE IN HELPING TO ENSURE THATQUALITY.JACK BUCKLEY: SO IF I WERE WORKING ON AN SEA AND WANTED TO TAKE THENEXT STEPS, LET'S SAY, IN TERMS OF PREVENTION--AGAIN FOCUSING ONPREVENTION--WHAT ARE SOME OTHER EXAMPLES? WHO SHOULD I TURN TO INTERMS OF IDENTIFYING BEST PRACTICES WITH RESPECT TO THIS KIND OFMANUAL OR TRAINING, IF I WANTED TO IMPLEMENT A REQUIREMENT LIKETHAT, MAYBE WITHOUT TURNING TO A VENDOR FIRST? I THINK WE'VE HEARDA COUPLE TODAY, BUT DOES ANYONE HAVE ANY OTHER IDEAS ABOUT TOWHOM TO TURN?SCOTT NORTON: I DO WANT TO TIE THAT BACK TO THE LAST COMMENT. I THINKTHAT'S AN INTRIGUING IDEA, AND IT'S SOMETHING WE MAY TRY TO DOBEFORE IT'S COMMON PRACTICE. WE DO REQUIRE THAT TEST PROCTORS ANDTEST MONITORS BE TRAINED, BUT THAT IS A VERY LOOSE DEFINITION, AND69EVEN DESPITE OUR EFFORTS, WE KNOW THAT IT'S PROBABLY INADEQUATE.SOME TYPE OF CERTIFICATION ONLINE--YOU GUIDE YOURSELF THROUGH ATRAINING OR SOMETHING LIKE THAT--COULD BE DEVELOPED, I THINK,RELATIVELY EASY ENOUGH, AND THEN FOLKS WOULD BE REQUIRED TO DO THATJUST SO THAT THEY WOULD BE AWARE WHAT'S NEEDED. I DON'T THINK IT'SBEEN EXPLICITLY STATED YET, SO I'LL GO AHEAD AND MENTION, I THINKWHAT WE'RE DOING HAS BEEN ADEQUATE ENOUGH FOR THE PAST FEW YEARS,BUT WITH THE ADVENT OF THE HIGHER-STAKES TEACHER EVALUATIONSYSTEMS THAT ARE UPON US--SOME ALREADY HAPPENED, SOME HAPPENINGSOON--I THINK WHAT HAS BEEN ADEQUATE IN THE PAST IS SOON GOING TOBE INADEQUATE.AUDREY AMREIN-BEARDSLEY: MAYBE YOU COULD USE KIND OF WHAT GREG WASTALKING ABOUT, TOO--THE INSTITUTIONAL REVIEW BOARD PROCEDURES THATWE DO AT THE UNIVERSITY LEVEL TO DO RESEARCH. MAYBE THAT WOULD BESOMETHING THAT WOULD ALSO BE CONSIDERED. GO THROUGH AN IRB TYPE OFPROCESS, LEARN ABOUT THE HISTORY OF THIS, AND ACTUALLY EARN YOURCERTIFICATE TO ACTUALLY BE A PROCTOR OR SOMEBODY IN CHARGE OF ACLASSROOM THAT'S DELIVERING TESTS.GREG CIZEK: YOU KNOW, THERE'S A DATABASE OF THOSE, RIGHT? ONCE YOUQUALIFY, YOU QUALIFY, AND AN EDUCATOR AND A STATE COULD QUALIFY.THERE WOULD BE A DATABASE THAT WOULD SAY, NO MATTER DISTRICT YOU'RE70IN, YOUR NAME IS LOGGED AS SOMEONE WHO'S COMPLETED THEQUALIFICATION, AND YOU'RE GOOD TO GO.JACK BUCKLEY: ALL RIGHT. JUST TO SWITCH GEARS--ANOTHER THING THAT,SCOTT, BOTH YOU AND JIM DISCUSSED, AND I THINK MAKES A LOT OFSENSE, IS AN INSPECTION PROGRAM, SOME SORT OF MONITORING BEYONDJUST, SAY, THESE PROCTORS. LET'S SAY THEY'RE BETTER TRAINED ORTHEY'RE CERTAINLY AWARE OF WHAT THEY'RE DOING INSIDE THEADMINISTRATION ROOM. I THINK THERE'S AN ISSUE HERE IN TERMS OFCOST. SCOTT, I THINK YOU MENTIONED SOMETHING LIKE 125 STAFF? IASSUME THAT'S NOT THEIR FULL-TIME JOB, MONITORING, ALTHOUGH ITCOULD BECOME ONE, I'D IMAGINE.SCOTT NORTON: I'LL JUST SAY A WORD ABOUT THAT. THOSE ARE 125 PEOPLEWHO HAVE OTHER JOBS MOST OF THE YEAR. FOR A WEEK, WE GET THEMTRAINED UP AND SEND THEM OUT INTO THE FIELD TO HELP MONITOR THESETESTS. WE TRY TO RELY ON THE FOLKS WHO HAVE EDUCATIONAL BACKGROUNDSAND ARE AT LEAST USED TO BEING AT SCHOOLS FOR SOME PURPOSE. I WASTALKING FAST, BUT I'LL SAY JUST A WORD MORE ABOUT THAT. THAT'S ARELATIVELY NEWER THING THAT WE'VE STARTED DOING FROM THE STATELEVEL, AND I THINK IT'S BEEN A GOOD THING. WE TRY TO BE UNOBTRUSIVEAND, YOU KNOW, NOT HEAVY-HANDED ABOUT IT. I THINK IT DOES SEND ASTRONG SIGNAL WHEN A STATE MONITOR WALKS IN THE DOOR OF THE SCHOOLTHE MORNING OF TESTING, AND YOU MAY OR MAY NOT BE EXPECTING THEM TO71BE THERE. FRANKLY, I'D LIKE TO SEE US DO A LITTLE BIT MORE OF THAT.IT'S A PRETTY MECHANICAL, GET IN YOUR CAR, DRIVE-AROUND-THE-STATEKIND OF THING, BUT IT HAS SERVED A GOOD PURPOSE FOR US, I BELIEVE.JIM LIEBMAN: IN NEW YORK, WE DO THE SAME THING. WE USE CENTRAL STAFFWHO VOLUNTEER FOR THIS, SO IT DOESN'T COST ANYTHING FOR THOSEPEOPLE. IT COSTS THEM SOMETHING. WE ACTUALLY USE SUBWAY RATHER THANCAR, BUT WHAT WE TRY TO DO IS TO ASSIGN PEOPLE TO SCHOOLS THAT ARENEAR WHERE THEY LIVE. BECAUSE WE WANT THEM TO GET THERE VERY EARLYIN THE MORNING, WHEN THE SHRINK-WRAP IS OPENED, WE WANT THEM TOSTART OUT PRETTY CLOSE TO WHERE THEY ARE. WE'VE GOTTEN TO THE POINTWHERE THIS IS MUCH MORE REGULARIZED AND THERE'S A VERY GOODCHECKLIST NOW AND EVERYBODY IS TRAINED AND EVERYBODY DOES THE SAMETHING. IN THE EARLY YEARS, IT WAS NOT AS RIGOROUS AS IT IS NOW. IACTUALLY THINK THAT WHERE YOU COULD GET A COMBINATION OF STATE ANDLOCAL MONITORING, SO THAT IF THE STATE CAN COVER ITS QUARTER OR 30%AND THE DISTRICT IS COVERING ANOTHER 10% OR 15%, YOU CAN GET INTOAN AWFUL LOT OF SCHOOLS, AND MOST SCHOOLS CAN EXPECT TO SEESOMEBODY EVERY FEW YEARS AT LEAST, AND I THINK THAT'S REALLY THEMAIN THING, THAT ONCE YOU SEE THAT HAPPENING, IT HAS A BIG EFFECT,AND I THINK IT'S IMPORTANT TO CIRCULATE AROUND THE SCHOOL SO THATEVERYBODY IN THE SCHOOL WHO'S INVOLVED IN THE TESTING DOES SEE ITWHEN YOU'RE THERE.72JACK BUCKLEY: WELL, GIVEN THE TIME, I THINK AT THIS POINT WE'D LIKETO OPEN IT UP BOTH TO THE OTHER SETS OF PANELISTS BUT ALSO TO THEAUDIENCE AND TO ANYBODY ONLINE VIEWING THE WEBCAST, SO PLEASE, FEELFREE.[INDISTINCT][SPEAKER'S MICROPHONE IS NOT ON]WAYNE CAMARA: A POINT I WANTED TO MAKE BRIEFLY IS JUST THAT I THINKWE DO KNOW A LOT ABOUT SOME OF THE BEST PRACTICES AND THEDEVELOPMENTAL POLICIES THAT SHOULD BE ESTABLISHED. MORE CANBE DONE THERE, BUT I THINK ONE OF THE AREAS WE REALLY HAVE TO LOOKCLOSELY AT IS IMPLEMENTATION, NOT DEVELOPMENT.I WORK WITH A NUMBER OF STATES, AND THERE'S A LOT OF VARIATIONIN TERMS OF STATE AUTHORITY, BOTH LEGAL AND POLITICAL, AND WORKEDWITH ONE STATE WHERE THEY'VE HAD A NUMBER OF THESE KIND OF CHEATINGISSUES, AND ONE OF THE FIRST QUESTIONS I ASKED OF THE TAC MEMBERWAS, DO WE HAVE SOME GOOD DATA ON THE TRAINING OF PROCTORS AND TESTADMINISTRATORS IN THAT STATE WHO ACTUALLY WENT THROUGH THETRAINING? AND I WAS TOLD THAT THEY CAN'T ENFORCE TRAINING. YOUCAN'T MAKE AN LEA TEST COORDINATOR DO ANYTHING. WE CAN ENCOURAGETHEM TO GO TO TRAINING, WE CAN ENCOURAGE PROCTORS TO TAKE THE73WEBEX, BUT WE CAN'T MAKE THEM DO THIS. AND I THOUGHT, OF ALL THETHINGS TO NOT BE ABLE TO MANDATE, HOW CAN YOU MANDATE THE TEST BUTNOT THE ADMINISTRATIVE PROCESSES? SO THE WHOLE POINT--I THINK SOMEOF THE EXAMPLES AUDREY HAD IN TERMS OF THE MATERIALS IN THECLASSROOM AND WHAT YOU CAN DO AND WHAT YOU CANNOT DO, THAT CUTSACROSS LOCAL AUTHORITY AND THE DESIRE FOR LOCAL FLEXIBILITY, ANDTHE WHOLE IDEA WITH ASSESSMENTS IS FOUNDED IN STANDARDIZATION.FAIRNESS IS FOUNDED IN EQUAL OPPORTUNITY, EQUAL ACCESS, ANDPLAYING BY THE SAME RULES, AND IT'S NOT FAIR TO HAVE A LOCALDISTRICT, FOR WHATEVER REASON—LOCAL CONTRACT, LOCAL NEGOTIATIONWITH THE TEACHERS UNION, LOCAL POLITICAL POSITIONS—HAVE AUTHORITYTO DICTATE WHAT IS ON THE WALLS IN ONE SCHOOL, WHEN ANOTHER SCHOOLIS INTERPRETING IT DIFFERENTLY. I'M NOT SURE TO WHAT EXTENT WE'RETALKING ABOUT THE LACK OF GOOD ADMINISTRATIVE PROCEDURES VERSUSTHE POLITICAL IMPERATIVE IN NOT ALL STATES, BUT SOME STATES, TOHAVE AN AWFUL LOT OF LOCAL CONTROL WHERE THE STATE HAS NOAUTHORITY, AND I THINK AS LONG AS THOSE ISSUES ARE NOT RESOLVED,WE'RE NOT GOING TO GET TO THE BOTTOM OF THIS.THIS IS ALL ABOUT PREVENTION, NOT ABOUT DETECTION. IF WE'RE NOTABLE TO MANDATE GREATER STANDARDIZATION, WILL WE EVER BE ABLE TOREALLY PREVENT SOME OF THE THINGS WE'RE SEEING?74JIM LIEBMAN: AND THAT'S WHERE...I THINK IT'S TRUE THAT DIFFERENTSTATES HAVE THE DIFFERENT LAWS, BUT WHERE THERE'S A WILL, THERE'S AWAY, AND I THINK A LOT OF IT COMES DOWN TO THE STATES REALLYACCEPTING RESPONSIBILITY. I THINK THE FEDERAL GOVERNMENT HAS SHOWNTHAT YOU CAN USE THE DISTRIBUTION OF MONEY AND GRANTS AS A WAY TOGET PEOPLE TO DO THINGS THAT YOU CAN'T REQUIRE THEM TO DO. YOU CANALSO JUST USE TRANSPARENCY--YOU KNOW, WHO'S DOING IT AND WHO'S NOT,AND THESE ARE BEST PRACTICES, AND YOU CAN GO LOOK AND SEE WHO'SDOING IT. I'VE JUST WATCHED IN THE STATE OF NEW YORK, OVER THE LAST5 OR 6 YEARS, THAT THERE'S JUST BEEN A TREMENDOUS CHANGE IN THEIRSENSE OF RESPONSIBILITY FOR THIS, AND AS THEY'VE INCREASED THEIRSENSE OF RESPONSIBILITY, THEY'VE FOUND MANY WAYS TO INCREASE THEABILITY TO ACHIEVE TEST SECURITY WITHOUT REALLY CHANGING THE LAW ATALL.JACK BUCKLEY: ANYONE ELSE ON THE PANEL COMMENT? REALLY WHAT YOU'VEINTRODUCED, FOR THE FIRST TIME, THAT WE HAVEN'T TALKED ABOUT YET,IS THE QUESTION OF BARRIERS, WHAT'S OUT THERE THAT'S IN THE WAY OFPREVENTION.WE'RE THINKING SORT OF FROM AN OPERATIONAL AND CULTURALSTANDPOINT, BUT YOU RAISE THE POLITICAL ECONOMY ASSESSMENT. WHATELSE IS OUT THERE THAT WE'VE SEEN, THAT STATES AND DISTRICTS HAVEEXPERIENCED, IN TERMS OF BARRIERS TO PREVENTION, BUT THEN ALSO WAYS75THAT THEY'VE OVERCOME? CERTAINLY NEW YORK IS NOT A STRANGER TOPOLITICAL PRESSURE, AND CLEARLY, YOU WERE ABLE TO OVERCOME, IIMAGINE, JIM, A BROAD SET OF OPPOSITION.JIM LIEBMAN: YEAH. I MEAN, I THINK THAT THERE PROBABLY ARE, IN ANYGIVEN STATE, REALLY GOOD THINGS HAPPENING IN CERTAIN DISTRICTS,AND I THINK TO ESTABLISH A SORT OF RACE TO THE TOP IDEA, THAT IFIT'S HAPPENING IN SOMEPLACE AND IT'S A GOOD STANDARD, THEN IT OUGHTTO BE THE STANDARD FOR EVERYBODY ELSE, AT LEAST FOR DISTRICTS THATARE COMPARABLE IN SIZE OR WHATEVER OTHER CONSTRAINTS THEY MIGHTHAVE.BUT I THINK THE STATES CAN REALLY DO A LOT, AND I THINK A FEW YEARSAGO, SOME OF US IN NEW YORK FELT LIKE THE STATE WAS HOLDING THEDISTRICTS BACK IN CERTAIN WAYS. WE REALLY WANTED TO GET THOSE HIGHSCHOOL TESTS GRADED OUTSIDE THE HIGH SCHOOL, BUT SINCE THE STATEHAD BLESSED THEM BEING GRADED BY THE TEACHERS IN THE CLASSROOM, ITWAS VERY HARD TO MAKE THAT CHANGE AT THE LOCAL LEVEL. SO IT CANKIND OF GO BOTH WAYS.JACK BUCKLEY: JOHN?JOHN FREMER: I'VE HAD THE EXPERIENCE OF AUDITING THE SECURITYPRACTICES76OF QUITE A NUMBER OF STATES AND DISTRICTS. WHEN WE GET TO THE END,WE MAKE RECOMMENDATIONS, AND ONE OF THE THINGS THAT ALMOST ALWAYSIS ASKED OF ME, IS THERE ANYTHING THAT'S EASY TO DO AND IMPORTANT?THERE ARE A LOT OF THEM THAT AREN'T, BUT ONE THAT PRETTY MUCH ISALWAYS POSSIBLE--IT'S A GOOD ONE FOR ME TO MENTION--IT'S VERYCOMMON FOR SENIOR PEOPLE, PEOPLE LIKE THE CHANCELLOR, TO TALKABOUT, OUR GOAL IS IMPROVING TEST SCORES. SHE DIDN'T SAY THAT, BUTIT'S VERY COMMON TO READ THAT, AND YET, THAT'S NOT THE GOAL. THEGOAL IS TO HELP STUDENTS LEARN MORE AND BE BETTER EQUIPPED FOR WHATTHEY WILL DO IN LIFE. SO TO WORK ON THAT, TO NOT JUST CONTINUEWORKING WITH THOSE WHO ARE ADVISING THE PERSON IN CHARGE, TO GET USTO ALWAYS TALK ABOUT HELPING STUDENTS LEARN MORE AND NOT TO TALKABOUT IMPROVING TEST SCORES, BECAUSE SOME OF THE THINGS THATIMPROVE TEST SCORES ARE EXACTLY THE THINGS YOU'RE WORRIED ABOUT.THE THINGS THAT WILL IMPROVE LEARNING, WE CAN ALL GET BEHIND.JIM LIEBMAN: I'D SAY SOMETHING ABOUT THAT. I THINK THAT THAT'S EASYAND YOU NEED TO DO IT, BUT THERE ARE HARD PARTS ABOUT THAT THAT AREJUST AS IMPORTANT, WHICH IS TO DEVELOP A CULTURE IN THE CLASSROOMTHAT THE DATA IS REALLY USEFUL AND HELPFUL, AND ONCE THAT HAPPENS,THEN THE RELIABILITY OF THAT DATA BECOMES IMPORTANT, INCLUDING TOEVERYBODY WHO'S USING IT, AND THEN ANYTHING THE EDUCATORSUNDERSTAND TO AFFECT THE RELIABILITY CAN BECOME A PROBLEM FOR THEM,AND AT THAT POINT, I THINK IT'S EASIER TO GET PEOPLE TO UNDERSTAND77WHY IT'S IMPORTANT, ESPECIALLY FOR SOME OF THE SECOND-DEGREE KINDSOF THINGS THAT ARE EASIER TO LET SLIP.JACK BUCKLEY: ANYONE ELSE ON THE PANEL WITH A COMMENT ON THAT ONE?AUDREY AMREIN-BEARDSLEY: WELL, I'LL PANEL--"I'LL PANEL." I WILLRESPOND AS A PANELIST.JACK BUCKLEY: YOU'RE IMPANELED ALREADY.AUDREY AMREIN-BEARDSLEY: I'LL PANEL ON THIS ONE.I WOULD AGREE WITH THAT, TOO, BECAUSE I THINK THAT FITS INTO THETAXONOMY, THE LAST POINT. IF PROFESSIONALLY, AS A CULTURE, WE CANALWAYS REMEMBER THAT IN THE END, IT'S ABOUT THE STUDENT LEARNING,AS INDICATED BY THE ACHIEVEMENT TEST, THAT'S REALLY WHERE WE WANTTO GO. PROFESSIONALLY, EVEN PROFESSIONAL DEVELOPMENT WITH TEACHERSAND ADMINISTRATORS, IS ALWAYS ABOUT THE STUDENT LEARNING. SO WHENYOU FEEL TEMPTED TO MARGINALIZE THE GRAPHING OF INEQUALITIESBECAUSE YOU KNOW IT'S NOT GOING TO BE ON THE TEST, YOU'VE GOT TO GOBACK TO THAT FUNDAMENTAL LAW-- IS THIS ABOUT STUDENT LEARNING?AS INDICATED, HOPEFULLY, BY THE STUDENT TEST, THE ACHIEVEMENTTEST.78JACK BUCKLEY: I'M SORRY. JUST A REMINDER TO OTHER FOLKS AT THE TABLETHAT DON'T HAVE THESE MICROPHONES, JUST TO ASK YOU TO PLEASE PUSHTO TALK IF YOU'RE GOING TO SPEAK.DO WE HAVE A QUESTION FROM THE WEB AUDIENCE?JAMES ELIAS: WE DO. SEVERAL PEOPLE WHO ARE WATCHING THE WEBCASTWOULD LIKE TO KNOW ABOUT THE FEASIBILITY OF THIRD-PARTYCERTIFICATION, GIVEN THE FACT THAT TESTING WINDOWS ARE SO SHORT INMOST STATES.GREG CIZEK: I CAN ANSWER THAT.JACK BUCKLEY: I HAVE A FEELING THAT'S NOT--GREG CIZEK: I'M NOT SURE I UNDERSTAND THE QUESTION. I THINK THESHORT TESTING WINDOWS ARE GREAT. I LOVED HEARING JIM TALK ABOUT ONEDAY OR SOMETHING. I'LL BE INTERESTED, WHEN TONY TALKS ABOUT SMARTERBALANCED, TO HEAR HOW SECURITY WILL BE DEALT WITH IN A LONGERCOMPUTER-ADAPTIVE TESTING WINDOW. BUT YOU KNOW, SOMEQUALIFICATION, WHETHER IT'S THIRD PARTY OR DIRECTED BY THE STATE,CAN BE DONE AT ANY TIME PRIOR TO THE ADMINISTRATION OF ANASSESSMENT. SO I DON'T THINK THAT THE SIZE OF THE WINDOW IS RELATED79TO THE NEED OR THE TIMING OF THE QUALIFICATION THAT I'M TALKINGABOUT.SCOTT NORTON: THAT WAS MY UNDERSTANDING, AS WELL--THAT THECERTIFICATION WOULD BE SOMETHING THAT HAPPENED WELL BEFORE THETESTING WINDOW.JACK BUCKLEY: YES. IS THERE A MIKE?AUDIENCEMEMBER:THANK YOU. I'M FROM THE DISTRICT OF COLUMBIA. ISERVE ON THE STATE BOARD OF EDUCATION. I HAVE 2 QUESTIONS. ONE ISVERY SPECIFIC. WHAT IS THE STATE OR DISTRICT OBLIGATION, IFALLEGATIONS OF CHEATING PROVE TO BE TRUE, TO INFORM THE PARENTSWHOSE CHILD MAY HAVE SCORED PROFICIENT OR ADVANCED? AND THEN MYSECOND QUESTION IS MORE PHILOSOPHICAL. THE NATIONAL RESEARCHCOUNCIL, THE NATIONAL ACADEMY OF SCIENCES' ARM, STUDIED 10 YEARS'WORTH OF BASICALLY HIGH-STAKES TESTS AND DETERMINED THAT OURABILITY TO MONITOR FOR GAMING OF THE SYSTEM WAS TOO LOW, BUT MOREIMPORTANTLY, THE MINUTE WE USE THE TEST DESIGNED TO MEASURE STUDENTLEARNING FOR ANY OTHER PURPOSE, WE INVALIDATE IT AS A MEASURE OFSTUDENT LEARNING. SO I'D LIKE TO KNOW HOW THE PANEL CAN HELP US ASPOLICYMAKERS GET OUT OF THAT CONUNDRUM, BECAUSE IF WE'REINVALIDATING THE MEASURE, IF THE YARDSTICK IS WARPED, WE HAVE TOUSE SOMETHING ELSE TO MEASURE STUDENT LEARNING AND EVALUATE80TEACHERS OR REWARD THEM OR WHATEVER. THOSE ARE MY 2 QUESTIONS.THANKS.SCOTT NORTON: I'LL TAKE THE FIRST ONE, IF THAT'S OK, AND TALK ABOUTTHAT JUST A LITTLE BIT. IT'S A TRICKY ISSUE. WE HAVEINVESTIGATIONS. WE TALKED ABOUT THAT. THERE ARE PROCEDURES, ANDSOMETIMES SCORES ARE INVALIDATED OR GIVEN A ZERO, SO THEN YOUREALLY DON'T HAVE A TRUE MEASURE OF THE CHILD'S ABILITY AT ALL. ITGETS VERY COMPLICATED VERY FAST IN OUR STATE. SOME TESTS ARE HIGHSTAKES, FOR GRADUATION OR PROMOTION. IN THOSE CASES, WE HAVE ARETEST OPPORTUNITY THAT'S PROVIDED--AGAIN, A LOT OF DETAILS AROUNDTHAT--SO THAT A VALID SCORE CAN BE REPORTED BACK. IF THEY'RE NOTHIGH-STAKES, WE DON'T DO THAT, BECAUSE OF COST, BUT IT IS AN ISSUETO LET PARENTS KNOW WHAT IS THE TRUE MEASURE THAT WAS PROVIDED BYTHE STATE, OTHER THAN JUST A ZERO BECAUSE THERE WAS A TEST SECURITYVIOLATION. WE TRY TO WORK AROUND THAT, BUT IT HAS BEEN A PROBLEMFOR US.JIM LIEBMAN: NEW YORK CITY JUST PERFORMED A REALLY EXTENSIVE AUDITOF, LIKE, 15% OF ITS HIGH SCHOOLS WHERE THE POSSIBILITY OF PROBLEMSWAS THE GREATEST. IT WASN'T JUST ON THE REGENTS EXAM, BUT IT WASALSO ON GRADUATION AND ALL OF THAT, AND IT WAS DETERMINED THATTHERE WERE A NUMBER OF STUDENTS WHO WERE GRADUATED WHO DID NOTACTUALLY MEET THE STANDARD FOR GRADUATION, AND STEPS TAKEN THERE81WERE TO INFORM AND TO GIVE THEM THE RIGHT TO COME BACK AND GETWHATEVER IT WAS THAT THEY MISSED, BUT NOT TO PUNISH BY TAKING AWAYTHEIR HIGH-SCHOOL DIPLOMA. IT WASN'T THEIR FAULT. SO THAT'S ONE WAYTO TRY TO BALANCE THAT PROBLEM.GREG CIZEK: I'LL GO AHEAD AND WADE INTO THE MORASS OF THAT SECONDPART. AND I...FIRST OF ALL, I WOULD SAY, OF COURSE; WHEN THERE'SRELIANCE ON AN INDICATOR, PEOPLE WILL FIND WAYS TO GAME THEINDICATOR, AND SO IT BECOMES SOMEWHAT LESS PURE OF AN INDICATOR. IGUESS I WOULD QUIBBLE WITH THE TERM INVALIDATED. IT'S NOT VALID ORNONVALID. THERE'S A CONTINUUM OF VALIDITY THERE TO THE INDICATOR.BUT I ALSO COP TO SORT OF A BAD ATTITUDE HERE. IF YOU THINK BACK TOWHEN HIGH-STAKES TESTS FIRST WERE INSTITUTED, THEY WERE INSTITUTEDLARGELY TO ADDRESS A PERSONNEL EVALUATION SYSTEM THAT WAS BROKEN.THERE WAS A CONCERN THAT WHAT WAS HAPPENING IN CLASSROOMS WAS NOTWHAT SHOULD BE HAPPENING IN CLASSROOMS, AND MY FIRST CHOICE, IF IWERE TO BE THINKING ABOUT PERSONNEL EVALUATION SYSTEMS, WOULD NOTBE TO GO GIVE KIDS ACHIEVEMENT TESTS. SO WE'VE MORPHED THATINDICATOR BECAUSE IT HAPPENED TO BE THE QUICK AND EASY ANDPOLITICALLY RESPONSIVE THING TO DO, BUT LARGELY, TESTING HASSUFFERED. ACHIEVEMENT TESTING HAS SUFFERED FOR A VERY LONG TIMEBECAUSE IT'S BEEN USED AS A QUICK FIX TO ADDRESS BROKEN PERSONNELEVALUATION SYSTEMS.82JIM LIEBMAN: I WANT TO SAY SOMETHING, TOO. I DISAGREE. I THINK THATYOUR TESTING, YOUR ASSESSMENT, SHOULD BE A PACKAGE, AND YOU SHOULDBE DOING SUMMATIVE ASSESSMENT AND A LOT OF FORMATIVE ASSESSMENT ANDA VARIETY OF THINGS IN BETWEEN, AND EACH OF THOSE ASSESSMENTSPERFORMS A DIFFERENT FUNCTION. I THINK ALL OF THEM HELP STUDENTS.I THINK WHAT THE SUMMATIVE TEST TELLS YOU IS, HERE'S AN AREA WHERETHIS STUDENT MAY NOT BE DOING SO WELL; GO LOOK AND SEE WHAT YOU CANFIND OUT, AND THAT REQUIRES INQUIRY. WE HAVE THESE INQUIRY TEAMS INALL OF OUR SCHOOLS, AND THAT'S WHAT THEY INQUIRE INTO, AND YOU USETHEN THE FORMATIVE ASSESSMENTS, AND THEN YOU HAVE TO DO YOUR OWNSELF-ASSESSMENT--YOU KNOW, CLASSROOM ASSESSMENT AND ALL SORTS OFTHINGS AND LOW-INFERENCE OBSERVATION AND EVERYTHING ELSE. SOWITHIN A SCHEME LIKE THAT, I THINK THAT THE IDEA THAT PUTTINGSTAKES ON AN ASSESSMENT TAKES AWAY ALL OF ITS VALUE, INCLUDING FORPURPOSES OF PROVIDING INFORMATION ABOUT WHERE THINGS ARE GOINGWRONG, IS JUST NOT THE CASE. WHEN WE PUT A GRADE AND STARTINGGIVING SCHOOLS A GRADE, I WOULD GO AROUND AND TALK TO THE "F"SCHOOLS, AND THEY WOULD TELL ME, "WE WERE JUST SCREWING UP IN MATH.WE HAD A BAD MATH CURRICULUM." AND THAT'S SOMETHING THAT THEYPICKED UP FROM SOME STAKES PUT ON STATE TESTING, BASICALLY. WHATTHEY DID ABOUT THAT AND HOW THEY RESPONDED TO IT REQUIRED DEEPERTOOLS AND MORE SOPHISTICATED TOOLS, BUT TO KNOW THAT AND TO KNOWTHAT THE PUBLIC KNOWS IT--REALLY IMPORTANT.83AUDREY AMREIN-BEARDSLEY: I THINK SOMETHING TO REMEMBER, AS WELL, IS IN2000, THE AERA, WHICH IS THE AMERICAN EDUCATIONAL RESEARCHASSOCIATION, THE NATIONAL COUNCIL FOR MEASUREMENT IN EDUCATION,AND THE AMERICAN PSYCHOLOGICAL ASSOCIATION CAME OUT WITH STANDARDSFOR TESTING, AND THE FIRST STANDARD IS THAT NO TEST SHOULD EVER BEUSED IN ISOLATION TO MAKE ANY TYPES OF DECISIONS UNLESS A--OR HAVEA HIGH-STAKES CONSEQUENCE ATTACHED. I THINK THAT IF WE KEEP THAT INTHE FOREFRONT CONTINUOUSLY AS WE DEVELOP THESE PROGRAMS THAT YOUHAVE TO HAVE, AND THAT GOES BACK TO WHAT--WE LOOKED AT THE KORETZISSUE, WITH THE VALIDITY. WE HAVE TO MAKE THE CASE FOR CRITERIONLEVELS OF VALIDITY. RECENTLY, IN THE LAST WEEK, THERE'S BEEN 2ARTICLES PUBLISHED, ONE OUT OF TENNESSEE, ABOUT THESE VALUE-ADDEDMEASUREMENT SYSTEMS, WHICH WE'RE BASICALLY NOW LOOKING AT NOTSNAPSHOTS, BUT LOOKING AT THE GROWTH OVER TIME, AND HOW THESE ARETRUMPING THE SUPERVISOR EVALUATION SCORES. SO ADMINISTRATORSIN TENNESSEE, SPECIFICALLY, ARE GETTING IN TROUBLE IF THEIROBSERVATION SCORES DO NOT MATCH THE VALUE-ADDED SCORES OF THESETESTS, SO THEY'RE TAKING THAT DUAL APPROACH, THE MORE HOLISTICAPPROACH, AND SAYING, THESE TESTS, IT REALLY SHOULD BE A ONEINDICATOR TYPE OF SYSTEM, AND I THINK THAT'S WHERE WE'RE GOING TOGET IN TROUBLE--IF WE START LOOKING MORE NARROWLY AT THESE SINGLEINDICATORS.JACK BUCKLEY: OTHER QUESTIONS? EUNICE.84EUNICE GREER: THE ISSUE OF FORMATIVE ASSESSMENT HAS BEEN PUT ON THETABLE, AND THE WHOLE NOTION OF ASSESSMENT BEING AN ONGOINGPRACTICE, NOT JUST A ONE-TIME SHOT, AND I WOULD LOVE TO HEARGREG AND AUDREY TALK MORE ABOUT THE TEACHER PREP ROLE IN THAT,BECAUSE THE RESEARCH THAT I'VE READ SUGGESTS THAT TEACHERSREALLY STRUGGLE WITH THEIR ROLE IN FORMATIVE ASSESSMENT. AND IT'S AFABULOUS PROCESS, BUT WHAT DO YOU SEE NEEDING TO CHANGE IF IT'SGOING TO TAKE THE OPPORTUNITY IT OFFERS US IN THE WHOLE SYSTEM OFASSESSMENTS FOR TEACHER ED?AUDREY AMREIN-BEARDSLEY: PROBABLY THE BIGGEST PROBLEM I SEE IS THATTHE TEST SCORES ARE ADMINISTERED AFTER THE STUDENTS LEAVE. I THINKTHAT'S WHAT THE TEACHERS TALK ABOUT MOST, IN TERMS OF HOW ARE THEYSUPPOSED TO USE THE DATA IF THE STUDENTS ARE ALREADY GONE? THEYCAN USE IT FORMATIVELY TO TALK ABOUT GENERAL THEMES, BUT NOTACTUALLY FOCUSED ON STUDENTS UNLESS THEY FORWARD THE SCORESINTO THE FUTURE. JAMES POPHAM, WHO'S AN ASSESSMENT PROFESSOR FROMUCLA, EMERITUS, HE ALSO TALKS ABOUT THE FURTHER THE TEST GETS AWAYFROM THE CLASSROOM, FOR FORMATIVE PURPOSES, THE WORSE IT GETS. SOHE REALLY ADVOCATES FOR THE CLASSROOM LEVEL, THE DISTRICTBENCHMARK,85HOPEFULLY VALIDATED DISTRICT BENCHMARK ASSESSMENT--USING THOSE FORFORMATIVE PURPOSES, WHERE REALLY THE BIG, LARGE-SCALE, HIGH-STAKESTESTS ARE MORE APPROPRIATE FOR THE CONSEQUENTIAL SUMMATIVEPURPOSES.GREG CIZEK: THE MORE I HEAR PEOPLE TALK ABOUT TESTING, SUMMATIVE ORFORMATIVE OR WHATEVER, THE MORE CONVINCED I AM THAT WE HAVE TOINTERVENE MUCH SOONER WITH TEACHERS AND PRINCIPALS TO HELP THEMUNDERSTAND THAT ASSESSMENTS HAVE DIFFERENT PURPOSES, AND THEY'REDESIGNED TO SERVE DIFFERENT PURPOSES, AND THEY FULFILL THOSEPURPOSES DIFFERENTIALLY WELL. THE IDEA THAT EVERY TEST CAN HAVE AFORMATIVE PURPOSE IS PROBABLY NOT AN ACCURATE CONCLUSION.THE LARGE-SCALE END-OF-YEAR TEST THAT EVERY KID TAKES IN LOUISIANAOR INDIANA OR ANYWHERE IS VERY GOOD AT SYSTEM MONITORING, AGGREGATEKIND OF PERFORMANCE INFORMATION. IT'S OUTSTANDING AT THAT. IT'SNOT VERY GOOD AT PROVIDING INDIVIDUAL STUDENT REMEDIATION PLANS.FORMATIVE ASSESSMENT IS VERY USEFUL FOR HELPING TEACHERSKNOW WHAT TO DO WITH A KID THE NEXT TIME THAT TEACHER INTERVENESBUT TERRIBLE AT SYSTEM MONITORING KINDS OF FUNCTIONS. IF WECONTINUE TO INSIST THAT EVERY TEST SERVE A MULTITUDE OF PURPOSES,NONE OF THEM ARE GOING TO DO THAT EQUALLY WELL, AND SO THE IDEA OFTAILORING THE TESTING ENTITY TO THE PURPOSE WE HAVE IN MIND IS, ITHINK, ESSENTIAL TO UNDERSTANDING THE ROLE THAT LARGE-SCALE ANDCLASSROOM TESTS PLAY.86JACK BUCKLEY: THERE'S A QUESTION IN THE BACK OF THE ROOM.[WOMAN SPEAKING INDISTINCTLY]AUDIENCEMEMBER:SORRY. HI. I'M WITH THE AMERICAN INSTITUTES FORRESEARCH, AND I USED TO WORK FOR A SCHOOL DISTRICT WHERE ONE YEARWE ASKED TEACHERS TO SIGN A PIECE OF PAPER THAT SAID, "I HAVE READTHE TESTING PROCEDURES, AND I AGREE TO FOLLOW THE RULES," AND THEUNION SAID, "NO. DON'T SIGN THAT." SO I'M WONDERING, AS WE'REDEVELOPING BEST PRACTICES, HOW DO WE INVOLVE THE UNIONS, AND HOW DOWE LEVERAGE THE UNIONS?JIM LIEBMAN: I THINK, YOU KNOW, IT'S A VERY IMPORTANT ISSUE, AND ITHINK THAT I WOULD SAY, "HOW DO WE INVOLVE THE TEACHERS," ANDTHE UNIONS ARE A PART OF THAT. BUT I THINK IT'S VERY IMPORTANT, INDESIGNING EVERYTHING THAT WE'VE TALKED ABOUT, THAT THEY SHOULD BEDESIGNED IN PART BY TEACHERS AND WITH THE INVOLVEMENT OF TEACHERS,BECAUSE THEY'RE GOING TO BE IN THE CLASSROOM AND THEY'RE GOING TOTELL YOU AN AWFUL LOT ABOUT HOW IT'S GOING TO WORK OR WHY IT WON'TWORK AND THINGS LIKE THAT. I THINK THAT ONCE YOU'VE DESIGNED THESETHINGS AND ROLLED THEM OUT FOR A YEAR, THEN YOU NEED TO STARTHEARING BACK FROM TEACHERS ABOUT WHAT THEY EXPERIENCED IN THECLASSROOM WITH ALL OF THE PROCEDURES AND THEN KIND OF WORK BACK87FROM THAT. I THINK IF YOU CAN MAKE THAT KIND OF AN EFFORT WITH THETEACHERS, IT WILL HELP YOU AN AWFUL LOT WITH THE UNION, BECAUSETHEN THEY'LL UNDERSTAND THIS TO BE SOMETHING THAT'S WORKING WITHTEACHERS, NOT AIMED AT TEACHERS. BUT, YOU KNOW, THE UNIONS HAVE APOSITION ON TESTING THAT MAKES THEM, IN SOME CASES, SOMEWHATDUBIOUS ABOUT WHAT THE TESTING IS ALL ABOUT, AND THEY CAN RAISESOME QUESTIONS THAT MAKE IT DIFFICULT, AND I THINK THAT ONE OF THETHINGS THAT--WE TALKED ABOUT HOW STATES CAN BE A LITTLE MOREPASSIVE THAN THEY NEED TO BE SOMETIMES. I THINK DISTRICTS SOMETIMESARE A LITTLE MORE PASSIVE THAN THEY NEED TO BE BECAUSE THEY'REWORRIED ABOUT WHAT THE UNION WOULD DO. I THINK YOU JUST WANT TO GETOUT IN FRONT OF THAT AND WORK WITH THE UNION AS MUCH AS POSSIBLE,BUT NOT USE IT AS AN EXCUSE NOT TO DO WHAT'S NECESSARY.GREG CIZEK: ONE OF THE MAIN POINTS THAT I TRY TO MAKE IS THIS NOTIONOF SHARED RESPONSIBILITIES, AND ONE OF THE GROUPS I TALKED ABOUTWAS PROFESSIONAL ASSOCIATIONS. I HIGHLIGHTED MY OWN PROFESSIONALASSOCIATION, BUT IT IS SOMEWHAT OF AN EMBARRASSMENT THAT EDUCATORSREALLY HAVE NOT BEEN THE MOTIVATION FOR WHY WE'RE EVENHERE TODAY. IT'S LARGELY BEEN THE MEDIA. YOU HEARD EARLIER IN SOMEOF THE INTRODUCTORY REMARKS THAT AN ARTICLE IN THE "USA TODAY"PROMPTED THE CHANCELLOR TO LOOK AT WHAT ARE WE DOING, HOW ARE WEDOING IT. IT WAS AN ARTICLE IN THE "DALLAS MORNING NEWS" THATPROMPTED DALLAS AND THE STATE OF TEXAS TO START LOOKING INTO STUFF.88IT WAS AN ARTICLE IN THE "ATLANTA JOURNAL-CONSTITUTION" ANDRESEARCH DONE BY MEDIA THERE THAT PROMPTED--YOU KNOW, EDUCATORSREALLY SHOULDN'T BE--BY EDUCATORS, I MEAN ALL OF US INVOLVED IN THEEDUCATIONAL ENTERPRISE, PROFESSIONAL ASSOCIATIONS INCLUDED-SHOULDN'T BE REACTING TO OUTSIDE INFORMATION. WE SHOULD BE THEPRODUCERS AND ACTORS UPON THAT INFORMATION, THAT OUR ACTIONS GETREPORTED ON, NOT CONSTANTLY IN RESPONSE TO OUTSIDE PRESSURES.SCOTT NORTON: IN MY COMMENTS, I TALKED A LITTLE BIT ABOUT THE OATHTHAT WE HAVE IN LOUISIANA THAT EVERYBODY SIGNS OFF ON, AND ISAID I'M NOT SURE WHETHER THOSE THINGS REALLY DO A LOT OF GOODOR NOT. WE'VE OPTED TO CONTINUE THAT. I DON'T THINK IT COULD HURT,BUT I WILL SAY I'M NOT SURE WHAT THE POINT WOULD BE OF ENCOURAGINGTEACHERS TO NOT SIGN A STATEMENT THAT SAID THEY DIDN'T VIOLATE THERULES. I DON'T REALLY GET THAT. IT'S A LITTLE BEWILDERING TO ME.JACK BUCKLEY: YES.BOB WILSON: I'M SORT OF THE OUTSIDE GUY ON THIS PANEL. I'M NOT APROFESSIONAL EDUCATOR OR CONNECTED FULLY TO THE EDUCATIONALSYSTEM. I'M A SCHOOL BOARD LAWYER, BUT NOT MANY PEOPLE AROUND HAVEBEEN AS DEEP INTO THE BOWELS OF A SCHOOL SYSTEM AS I HAVE. AND THEPREVENTION, WHICH DR. CIZEK MENTIONED--AND HE WAS OUR EXPERT INTHAT CASE AND WAS TREMENDOUSLY HELPFUL TO US; I CAN'T THANK HIM89ENOUGH--STARTS WITH YOUR SUPERINTENDENT OR CHANCELLOR. IUNDERSTAND THE QUESTION ABOUT THE UNION. IT'S A VERY GOOD QUESTION.BUT YOU'RE TALKING ABOUT AN HONOR CODE, AND IF YOU GO TO WESTPOINT, YOU'RE NOT EXPECTED TO DO ANYTHING EXPECT BE HONORABLE ANDTAKE YOUR TEST FAIRLY AND REPORT IF YOU SEE SOMETHING WRONG, BUTTHEY STILL HAVE TO AGREE TO THAT HONOR CODE. AND WHOEVER THATSUPERINTENDENT OR CHANCELLOR IS FOR THAT SYSTEM, IN THE PREVENTIVEWORLD AND SHARED RESPONSIBILITY THAT DR. CIZEK MENTIONED, IS THEPERSON THAT, I THINK--AND I WANT YOUR REACTION TO IT—THATABSOLUTELY, POSITIVELY, WITHOUT DOUBT SETS THE TONE FOR THATSYSTEM, AND IT HAS TO BE THAT THEY TALK ABOUT TESTING WITH HONORAND THE RESPONSIBILITY TO EVERYBODY, STUDENTS AND PROFESSIONALS INTHE SYSTEM, WHETHER IT'S THE CAFETERIA WORKER WHO SEES SOMETHING ORA JANITOR, JUST AS MUCH AS A TEACHER OR A STUDENT. AND THEY SETTHAT TONE, AND THEY EXPECT NOTHING LESS, AND THERE WILL BE DIRECONSEQUENCES IF YOU VIOLATE IT, BUT EVERYBODY HAS A STAKE IN IT,AND WHAT I FOUND A LOT OF PLACES--WE TALK ABOUT THE TEST ALL THETIME. NEVER DO THEY TALK ABOUT TAKING THE TEST WITH HONOR. NOW, YOUCAN SIT THERE AND SAY, "OH, I CAN'T BELIEVE THAT." NOBODY INATLANTA IN THE INSTRUCTIONAL PROCESSES, OR DOUGHERTY COUNTY INSOUTH GEORGIA, WHICH WE ALSO INVESTIGATED, EVER MIXED THE TRAININGWHERE THE EMPHASIS WAS EQUALLY AS MUCH ON THE INTEGRITY OF TAKINGTHE SYSTEM--JUST THE RULES, POLICIES, AND PROCEDURES, WITHOUTREALLY UNDERSTANDING WHY. I WANT TO KNOW WHAT YOU ALL THINK OF90THAT, BECAUSE I BELIEVE IT DOES START AT THE TOP, BUT I WANT TO SEEIF YOU ALL AGREE.JIM LIEBMAN: COULDN'T AGREE MORE. I MENTIONED THAT CHANCELLORWALCOTT IN NEW YORK CITY, WHEN THERE WAS DISCUSSION OF CHEATINGAND ALL OF THAT, JUST WENT ON TV, BASICALLY, AND SAID, "CALL ME. WETAKE IT VERY SERIOUSLY." AND I THINK THAT'S REALLY IMPORTANT. NEWYORK CITY--I MENTIONED THIS AUDIT THAT IT RECENTLY DID. YOU MIGHTTHINK IT'S SUICIDE. THEY ESSENTIALLY SAID, WE'RE GOING TO GO FIGUREOUT WHAT CAN GO WRONG IN HIGH SCHOOLS, AND WE'RE GOING TO START BYLOOKING AT THE 15% WHERE THE DATA THAT WE'VE JUST LOOKED AT TELLSUS IT'S PROBABLY THE WORST, AND THEN THEY SAMPLED TEACHERS INCLASSROOMS WHERE THEY THOUGHT IT WAS PROBABLY THE WORST OF THEWORST, AND THEN THEY MADE A PUBLIC REPORT ABOUT IT, AND I THINKTHAT'S THE KIND OF THING THAT YOU NEED TO BE WILLING TO DO, EVENTHOUGH YOU KNOW YOU'RE GOING TO PUT YOURSELF OUT THERE, BECAUSE ITFORCES YOU TO CONFRONT WHAT THE PROBLEMS ARE. BUT IF YOU REALLY,REALLY CARE ABOUT YOUR DATA, AND I THINK MORE AND MORE PEOPLEREALLY DO NOW, FOR ALL OF THE REASONS THAT WE'VE TALKED ABOUT,YOU CAN'T HAVE BAD DATA, AND ONE OF THE WAYS YOU GET BAD DATA ISFROM CHEATING ON TESTS. SO I THINK IT CAN ALL WORK TOGETHER TOCREATE A CULTURE WHERE THAT'S JUST ACCEPTED.91AUDREY AMREIN-BEARDSLEY: I THINK THAT GOES TO THE SHAREDRESPONSIBILITY, TOO, AND ALSO, WHAT BETTER LESSON FOR STUDENTS INTHE CLASSROOMS, THAT UNDERSTAND THAT THERE IS A CODE OF HONOR?THERE'S AN ETHICS OF HONOR IN TERMS OF WHAT WE'RE DOING, WHY WE'REDOING THIS. WE HAD A LOT OF CASES IN THIS STUDY WHERE PARENTSACTUALLY PARTICIPATED AND TALKED ABOUT THE STORIES THAT THEY HADAFTER THEIR CHILDREN CAME HOME AND SAID, "MOMMY, THIS DIDN'T SEEMVERY RIGHT. THIS HAPPENED IN MY CLASS TODAY DURING THIS TEST." SOWE HAVE THEM. THEY'RE PART OF THIS, AS WELL, THE PARENTS AND THESTUDENTS. I THINK THAT--FRAMING THIS ISSUE WITH HONOR AND HOW WE'REGOING TO APPROACH THIS APPROPRIATELY--I THINK THAT'S THE BESTLESSON FOR EVERYBODY.SCOTT NORTON: I THINK YOU MAKE A GREAT POINT, AND I THINK OURSYSTEMS, WHERE A STATE HAS ONE, DO A PRETTY GOOD JOB WHEN PEOPLEFOLLOW THAT CODE. IF SOMETHING HAPPENS, THERE'S A SYSTEM OF CHECKSAND BALANCES AND INVESTIGATION. WHERE WE'RE WEAKER, I THINK--ITWAS REALLY ON MY LAST SLIDE--IS WHEN THERE'S A BREAKDOWN IN THERE,WHETHER IT BE A SUPERINTENDENT, TEACHER, OR WHOEVER. THEY REALLYDON'T WANT TO REPORT UP, AND THEN WE'RE SORT OF LEFT HOLDING THEBAG AND TRYING TO FIGURE OUT WHAT REALLY HAPPENED. THAT'S WHEN ITBECOMES VERY DIFFICULT.JACK BUCKLEY: ALL RIGHT.92AUDIENCEMEMBER:EXCUSE ME. I WORK FOR THE AFT, AND I'D LIKE TO SPEAKFOR THE UNION.JACK BUCKLEY: I'LL GIVE YOU ONE MINUTE, BUT THEN I NEED TO WRAP UPAND MOVE ON TO THE NEXT PANEL.AUDIENCEMEMBER:30 SECONDS. LAST FALL, WE ADOPTED A VERY STRONGRESOLUTION ON CHEATING THAT'S AS STRONG AS ANYTHING, I SUSPECT, YOUWILL COME UP WITH TODAY. WE ENCOURAGE TEACHERS TO REPORT CHEATINGTHROUGH THEIR UNION SO THAT THEY CAN HAVE SOME SORT OF PROTECTION.IF YOU LOOK AT ATLANTA, IF YOU LOOK AT MANY OTHER PLACES AROUND THECOUNTRY, TEACHERS WHO REPORT CHEATING TEND TO BE THE ONES THAT GETFIRED OR GET IN TROUBLE, AND TO JUST SAY THAT UNIONS ARE, WHAT, THEBAD GUYS IN THIS, IS JUST ABSOLUTELY NOT TRUE.JACK BUCKLEY: I THINK I'VE HEARD A LOT OF THINGS THIS MORNING. IHAVEN'T HEARD ANYONE TRYING TO BLAME THE UNION FOR PROBLEMS WITHTESTING INTEGRITY, AND I'D ENCOURAGE YOU TO STICK AROUND FOR THENEXT TWO PANELS, WHERE WE'LL HAVE A LOT MORE TIME TO LOOK AT THATISSUE. I THINK IN TERMS OF WHAT WE'VE HEARD THIS MORNING, VERYBRIEFLY, CERTAINLY I THINK THAT THERE'S SOMEWHAT CONSENSUS THATTHERE ARE DEGREES OF CHEATING OR TESTING IRREGULARITY AND THATTHOSE TWO CONCEPTS ARE NOT NECESSARILY THE SAME THING. I THINK93CERTAINLY I'VE HEARD FROM THE PANEL AND FROM THE AUDIENCE THATMULTIPLE MECHANISMS ARE NEEDED FOR PREVENTION, BOTH AT THEOPERATIONAL LEVEL AND THEN SORT OF FURTHER REMOVED FROM THEADMINISTRATION ROOM. IT SOUNDS LIKE THERE'S A CALL FOR MORECONSISTENCY IN STANDARDIZATION IN DETECTION, AND THAT'S SOMETHINGWE'LL GET TO IN THE NEXT PANEL.I THINK A LOT OF COMMENTS THIS MORNING FOCUSED ON TRYING TOMAINTAIN A HEALTHY TESTING ENVIRONMENT, AND I THINK THE POINT ISWELL TAKEN THAT THAT STARTS AT THE TOP AND NOT NECESSARILY AT THEBOTTOM, BUT IT IS A SHARED RESPONSIBILITY AMONG STUDENTS ANDEDUCATORS AND TEST DEVELOPERS AND PARENTS AND PROFESSIONALASSOCIATIONS. I THINK THERE WAS A LOT OF TALK THAT I DIDN'TNECESSARILY EXPECT ABOUT TRAINING AND CERTIFICATION IN THEPROCTORING OR ADMINISTRATION PROCESS, AND I THINK THAT'S SOMETHINGTHAT WE'LL HAVE TO INVESTIGATE FURTHER. IT WAS VERY INTERESTING TOHEAR. AND I THINK JUST SORT OF GENERAL CALL FOR MORE COORDINATIONAMONG STATE AND LEAS, AND THAT'S TO BE EXPECTED, AS WELL. THERE'S ALOT OF PLAYERS HERE. IT'S A VERY COMPLICATED ENVIRONMENT. SO ATTHIS POINT, WE'RE GOING TO TAKE A 5-MINUTE BREAK JUST SO WE CAN,AGAIN, ROTATE THE PANELISTS UP FRONT. SO, FOLKS ON PANEL TWO,PLEASE COME UP.94THANK YOU VERY MUCH. JOIN ME IN THANKING THE PANELISTS GREATJOB.95PANEL II: DETECTION AND ANALYSIS OF IRREGULARITIES IN ACADEMICTESTINGJACK BUCKLEY: ALL RIGHT, SO, WE'RE BACK, AND OUR SECOND PANEL FOCUSESON DETECTION AND ANALYSIS OF IRREGULARITIES IN TESTING.WE'LL START WITH BRIAN JACOB, WHO IS GONNA JOIN US REMOTELY FROMANN ARBOR, WHERE HE'S THE WALTER H. ANNENBERG PROFESSOR OFEDUCATION POLICY, PROFESSOR OF ECONOMICS, AND DIRECTOR OF THECENTER ON LOCAL, STATE, AND URBAN POLICY AT THE UNIVERSITYOF MICHIGAN'S GERALD FORD CENTER OF PUBLIC POLICY. HIS PRIMARYFIELDS OF INTEREST ARE LABOR ECONOMICS, PROGRAM EVALUATION,AND THE ECONOMICS OF EDUCATION. HIS CURRENT RESEARCH FOCUSES ONURBAN SCHOOL REFORM AND TEACHER LABOR MARKETS.NEXT, WE'LL HEAR FROM CARSWELL WHITEHEAD, A DIRECTOR IN ETS' OFFICEOF TESTING INTEGRITY, WHERE SHE PLANS, DIRECTS, AND MANAGESSECURITY INVESTIGATIONS AND POLICIES DESIGNED TO PREVENT ANDDETECT SCORE IRREGULARITIES FOR COLLEGE BOARD TESTING PROGRAMS.DR. DAVID FOSTER, CURRENTLY THE CEO OF CAVEON TEST SECURITY, WILLBE OUR NEXT SPEAKER. HE'S ALSO THE CHIEF SCIENTIST AND EXECUTIVEVICE PRESIDENT OF KRYTERION, A UNIQUE INTERNET TEST ADMINISTRATIONCOMPANY WHICH HE FOUNDED. HE'S BEEN CREATING AND IMPROVING96COMPUTERIZED TESTING SYSTEMS SINCE 1982 AND HAS BEEN INFLUENTIALOVER THE PAST 28 YEARS IN INTRODUCING MANY IMPORTANT INDUSTRYINNOVATIONS AND INITIATIVES.AND THEN FINALLY, AS ASSOCIATE SUPERINTENDENT FOR ASSESSMENTAND ACCOUNTABILITY AT THE GEORGIA DEPARTMENT OF EDUCATION, MELISSAFINCHER OVERSEES THE DEVELOPMENT OF ALL OF GEORGIA'SASSESSMENT AND ACCOUNTABILITY PROGRAMS. SHE'S RESPONSIBLE FORENSURING THESE PROGRAMS MEET HIGH STANDARDS FOR TECHNICALDEFENSIBILITY. NOW I'LL TURN IT OVER TO BRIAN.BRIAN JACOB: OK, HELLO.I'M NOT SURE IF EVERYONE CAN SEE OR HEAR ME NOW, BUT I WILL PROCEEDAS IF YOU CAN.I THINK I SHOULD BE ON THE TITLE SLIDE HERE. THIS IS "DETECTION ANDANALYSIS OF IRREGULARITIES IN ACADEMIC TESTING." AND SO SLIDE 2 ISJUST A FOCUS OF THE TALK.SO, AS AN ACADEMIC ON THE PANEL, I...AM FREE TO ADMIT THAT I HAVELIMITED FAMILIARITY WITH THE PRACTICES AND POLICIES CURRENTLYUSED BY SEAS AND LEAS--SOME, BUT NOT ALL. SO, THE LESSONS I'M GONNASHARE TODAY COME FROM THE THEORY OF STATISTICAL ANALYSIS FOR THIS97TYPE OF DETECTION AND THEN FROM MY EXPERIENCE WORKING WITH SEVERALLARGE URBAN SCHOOL DISTRICTS AROUND SOME OF THESE ISSUES.I'M GOING TO FOCUS OUR REMARKS TODAY ON TEACHER AND/ORADMINISTRATOR CHEATING AS OPPOSED TO STUDENT CHEATING OR THEBROADER RANGE OF OTHER ILLICIT TESTING ACTIVITIES ANDIRREGULARITIES. I'M ALSO GOING TO FOCUS MY TALK ON STATISTICALANALYSES ONE CAN USE TO DETECT SUCH MANIPULATION AND RECOGNIZINGTHAT THIS HAS LIMITSAND THEN OTHER METHODS WILL BE NECESSARY TO COMPLEMENT THESE.MY TALK WILL FOCUS ON THE STATISTICAL ANALYSIS.SO, NEXT SLIDE.THE ROLE OF STATISTICAL ANALYSIS IN DETECTING IRREGULARITY.STATISTICAL ANALYSES HAVE A VARIETY OF BENEFITS.ONE--THEY'RE RELATIVELY LOW COST. CERTAINLY, RELATIVE TO IN-DEPTHFORENSIC INVESTIGATIONS, MONITORING OF HUNDREDS OF THOUSANDS OFCLASSROOMS. THEY CAN COVER THE ENTIRE POPULATION. UNLIKE SOMEVERY LEGITIMATE METHODS, SUCH AS TIPLINES OR INDIVIDUAL CASESTUDIES OF CERTAIN SCHOOLS, THIS IS A WAY TO LOOK AT THE ENTIRE98POPULATION OF CLASSROOMS IN AN AREA. SOME OF THESE MEASURES, WHENPROPERLY USED, CAN PROVIDE A BETTER UNDERSTANDING OF THE EXTENT OFTHE IRREGULARITY—YOU KNOW, A MEASURABLE LIKELIHOOD THAT ANIRREGULARITY WOULD HAVE OCCURRED BY CHANCE IN THE ABSENCE OF ANYMANIPULATION. THEN IT CAN ALSO IDENTIFY SYSTEMATIC PATTERNS ASWELL AS INDIVIDUAL CASES OF CONCERN.SO, ALTHOUGH STATISTICAL ANALYSES CERTAINLY HAVE LIMITATIONS, SO ITHINK A USEFUL ANALOGY IS KIND OF THE CLINICAL SCREENING FOR RAREDISEASES THAT ONE WOULD DO IN MEDICINE. STATISTICAL ANALYSES TODETECT CHEATING CAN HAVE A HIGH RATE OF FALSE POSITIVES AND/ORFALSE NEGATIVES. IT'S CRITICAL TO COMPLEMENT THEM WITH OTHERMETHODS.AND THEN IT REALLY IS IMPOSSIBLE, I THINK, TO DETERMINE THE ACTUALINDIVIDUALS RESPONSIBLE FOR MANIPULATION, EVEN IF ONE IS CONVINCEDTHAT THERE WAS IRREGULARITY AND THIS WAS DUE TO SOME ILLICIT HUMANINTERVENTION, WHETHER IT WAS THE ACTUAL TEACHER OF RECORD, THECOUNSELOR, THE ADMINISTRATOR. ONE CAN'T REALLY DETERMINE FROM ASTATISTICAL ANALYSIS. SO, NEXT SLIDE.IN GENERAL, THERE ARE 3 DIFFERENT APPROACHES FOR DETECTING TESTINGIRREGULARITIES USING STATISTICAL ANALYSES. ONE IS IN A RATIOANALYSIS, AND THIS IS SOMETHING WHERE ONE WOULD DETERMINE99GENERALLY THE AVERAGE NUMBER OF WRONG TO RIGHT ERASURES IN ACLASSROOM. AND CERTAINLY PEOPLE ON THE PANEL AND IN THE AUDIENCEARE PROBABLY MUCH MORE FAMILIAR WITH THIS THAN I AM.TESTING COMPANIES HAVE, I THINK, FAIRLY WELL-DEVELOPED METHODS FORSCANNING AND DETERMINING WHETHER SOMETHING IS A TRUE ERASURE ASOPPOSED TO A DIFFERENT TYPE OF MARK ON A TEST FORM.THEN A SECOND METHOD IS ITEM RESPONSE ANALYSIS, WHERE YOU'DBASICALLY BE LOOKING FOR UNUSUALLY COMMON RESPONSE PATTERNS ACROSSSTUDENTS WITHIN THE SAME CLASS. AND IN THIS TYPE OF ANALYSIS, YOUWOULD ALWAYS BE TRYING TO ACCOUNT FOR OTHER CHARACTERISTICS THATWOULD CAUSE THIS RESPONSE PATTERN TO BE COMMON THAT WOULDN'T BE DUETO CHEATING.AND FINALLY, LOOKING AT THE ANALYSIS OF THE TEST SCORES THEMSELVES,LOOKING FOR UNUSUALLY LARGE GAINS IN ONE YEAR AND/OR VERY LARGERELATIVE DECLINES IN THE SUBSEQUENT YEAR OR UNEXPECTED JUMPS IN THEDISTRIBUTION OF TEST SCORES. NEXT SLIDE.THIS IS A GRAPH BASED ON SOME WORK THAT COLLEAGUES OF MINE AND IDID, LOOKING AT THE NEW YORK STATE REGENTS EXAM A YEAR OR SO AGO.AND THIS IS AN EXAMPLE OF HOW ONE COULD USE STATISTICAL ANALYSIS TO100IDENTIFY SYSTEMIC IRREGULARITIES. THIS WAS NOT AN ANALYSIS TOIDENTIFY WHICH PARTICULAR SCHOOLS OR CLASSROOMS MIGHT HAVE BEEN...INVOLVED IN MANIPULATION. BUT THIS IS THE NEW YORK STATE REGENTSHIGH SCHOOL EXAM REQUIRED FOR GRADUATION.YOU SEE A GRAPHIC HERE BY SCALED SCORE ON THE VERTICAL AXIS. IT'STHE NUMBER OF STUDENTS IN THE STATE THAT RECEIVE THIS ACTUALINDIVIDUAL SCALED SCORE. AND THE TWO SOLID VERTICAL LINES AT 55 AND65 INDICATE THE PASSING THRESHOLDS FOR A REGULAR AND A MORE KIND OFBASIC DIPLOMA IN NEW YORK STATE.SO, WHAT IMMEDIATELY JUMPS OUT AT YOU IS THE NUMBER OF STUDENTSSCORING A 63 OR 64 IS MUCH, MUCH LOWER THAN THE NUMBER OF STUDENTSSCORING EXACTLY A 65, WHICH IS WHAT'S NECESSARY FOR PASSING ANDGRADUATING HIGH SCHOOL IN NEW YORK STATE. AND YOU SEE A SIMILARBLIP AT THE 54 VERSUS 55 MARGIN.AND SO BASED ON SOME STATISTICAL ANALYSES THAT UNDERLIE THIS GRAPH,YOU CAN CONCLUDE THAT THIS IS VIRTUALLY IMPOSSIBLE TO HAVE OCCURREDBY CHANCE AND SUGGEST THERE MAY HAVE BEEN SOME MANIPULATION IN THEGRADING OF THESE EXAMS, WHICH IS DONE BY THE TEACHERS IN THE ACTUALHIGH SCHOOLS FOR THEIR OWN STUDENTS. SO AGAIN, THIS IS AN EXAMPLEUSING STATISTICS TO IDENTIFY SYSTEMIC IRREGULARITIES. NEXT SLIDE.101SO, THIS IS AN EXAMPLE BASED ON SOME WORK I DID WITH STEVE LEVITTOUT IN CHICAGO IN THE EARLY 2000S LOOKING AT TESTING IRREGULARITIESUSING ACHIEVEMENT GAINS AND RESPONSE PATTERNS.THIS IS AN EXAMPLE. EACH ROW IS A DIFFERENT STUDENT. EACH COLUMN ISTHEIR RESPONSE CATEGORY FOR DIFFERENT ITEMS. NEXT SLIDE.SO, IF YOU LOOK HERE, THE FIRST THING YOU NOTICE IN THIS CLASS WASTHAT THERE WAS A VERY LARGE INCREASE IN SCORES--AN AVERAGE 4.1GRADE EQUIVALENCE JUMPING TO 5.8 GRADE EQUIVALENCE IN THATSUSPECTED YEAR. AND THEN IF YOU FOLLOW THE STUDENTS TO THE YEARAFTER WHEN THEY WERE NOT IN THIS SUSPECTED TEACHER'S ROOM, THEIRTEST SCORES ACTUALLY GO DOWN, EVEN THOUGH THEY HAD A YEAR EXTRA OFEDUCATION. SO, THIS PATTERN HERE CLEARLY RAISES SUSPICION. AND THENEXT SLIDE?HERE, THERE'S ALSO SOME STATISTICAL ANALYSES UNDERLYING THIS,WHERE YOU CAN LOOK AND IDENTIFY A NUMBER OF THE ITEMS IN THISCLASSROOM. THERE WAS—THE CORRELATION ACROSS STUDENTS WASEXTRAORDINARILY LARGE, EVEN CONDITIONAL ON WHAT YOU WOULD EXPECTGIVEN THE STUDENT CHARACTERISTIC. NEXT SLIDE.SO, FINALLY, I THINK IT'S IMPORTANT, THEN, REGARDLESS OF THESTATISTICAL ANALYSIS, TO UNDERSTAND THE TESTING IRREGULARITY DOES102NOT NECESSARILY IMPLY CHEATING. A FEW EXAMPLES I HAVE LISTED HERE:FREQUENT ERASURES MAY BE MORE COMMON AMONG CERTAIN STUDENTS THANOTHERS. COMMONALITY OF ITEMS AND RESPONSE PATTERNS ACROSSSTUDENTS MIGHT BE DUE TO CURRICULAR INSTRUCTIONAL PURPOSE OF THETEACHER. AND OBVIOUSLY LARGE TEST SCORE GAINS COULD BE THE RESULTOF AN EXTREMELY EFFECTIVE TEACHER, WHICH WOULD BE THE LAST THING WEWOULD WANT TO DISCOURAGE.SO WE ALWAYS NEED TO COMPLEMENT THESE ANALYSES WITH OTHER METHODSOF PROTECTION. NEXT SLIDE.AND SO, WHAT CAN WE DO WHILE WE ARE ANALYZING SOME OF THESE TESTINGDATA? ONE, IS THAT IT'S CRITICAL TO ACCOUNT FOR STUDENT AND/ORSCHOOL CHARACTERISTICS THAT MIGHT BE ASSOCIATED WITH ERASURES ORUNUSUAL TEST SCORE PATTERNS. I THINK THIS IS SOMETHING THAT IS LESSCOMMONBASED ON MY EXPERIENCE IN THE TESTING INDUSTRY, WHERE THERE'SUSUALLY A COMPARISON OF ONE CLASSROOM OR SCHOOL TO, LET'SSAY, THE STATE AVERAGE, EVEN IF THE STATE AVERAGE MIGHT BE A VERYDIFFERENT GROUP OF STUDENTS OR TYPE OF STUDENTS THAN THE STUDENTSIN THE SUSPECTED CLASSROOM.103SECOND, USING METRICS OR ROBUST TO OUTLIERS, SO MAYBE THE MEDIAN ASOPPOSED TO THE MEAN, WE WOULDN'T WANT TO BASE A JUDGMENT ON ACLASSROOM WHERE ONE OR TWO STUDENTS WERE DRIVING AN ENTIRECLASSROOM MEASURE, UTILIZING MULTIPLE APPROACHES WHEN POSSIBLE. ITHINK CONDUCTING SELECTIVE RETEST AUDITS MAY BE SOMETHING THAT INCERTAIN WELL-CHOSEN CASES CAN BE VERY EFFECTIVE.THIS IS SOMETHING THAT WAS DONE BY THE ADMINISTRATION IN CHICAGO ASA RESULT OF SOME OF THE WORK THAT STEVE LEVITT AND I DID. TAKINGSUSPECTED CLASSROOMS SEVERAL WEEKS AFTER THE REGULAR TESTING,RETESTING THEM IN CONTROLLED SITUATION, AND THEN RETESTING SOMERANDOMLY SAMPLED CLASSROOMS TO THE EXTENT THAT ONE FINDS THAT THERETEST RESULTS FOR THE SUSPECTED CLASSROOM ARE MUCH LOWER THAN THEORIGINAL RESULTS, ONE MIGHT HAVE MORE EVIDENCE THAT THERE WAS SOMEINITIAL MANIPULATION IN THE ORIGINAL EXAM ITSELF.AND THEN FINALLY, CONDUCTING IN-DEPTH QUALITY OF INVESTIGATIONS OFSUSPECTED INDIVIDUALS. NEXT SLIDE.SO, THERE ARE CERTAINLY SOME PRACTICAL CHALLENGES. I'M SURE ALL OFTHE OTHERS IN THE AUDIENCE AND THE PANEL WHO DEAL WITH IT DESCRIBETHEM BETTER THAN I CAN, BUT THERE'S ISSUES OF DISTRICT AND STATECAPACITY. AGENCIES OBVIOUSLY CRUNCH RESOURCES NOW, AND I THINKFINDING THE TIME AND MONEY TO DO THIS IS OBVIOUSLY DIFFICULT.104I THINK THERE ARE ISSUES THAT HAVEN'T BEEN FULLY WORKED OUT ABOUTDATA SHARING BETWEEN TEST VENDORS, STATE AGENCIES AND LOCALDISTRICTS THAT CAN, IN CERTAIN CASES, HAMPER THE DETECTION,ANALYSIS, AND EVENTUAL DEALING WITH TESTING IRREGULARITIES. ANDTHEN THERE'S THE STATISTICAL FLUENCY OF THE INTENDED AUDIENCE,WHETHER THAT'S DISTRICT OR SCHOOL LEADERS OR...EVENTUAL KIND OFADMINISTRATIVE BODIES OR EVEN COURTS. AND LAST SLIDE.SO, THAT IS IT. THANK YOU. I LOOK FORWARD TO HEARING THE OTHERPANELISTS AND THE QUESTIONS FROM THE AUDIENCE.JACK BUCKLEY: BRIAN, THANK YOU VERY MUCH.CARSWELL?CARSWELL WHITEHEAD: GOOD MORNING. THANK YOU, JACK, AND THANK YOU TOTHE FOR INVITING ME HERE TODAY.IN THE OFFICE OF TESTING INTEGRITY AT EDUCATIONAL TESTING SERVICE,WE HAVE A 3-STEP SECURITY PROCESS TO SAFEGUARD THE MILLIONS OFTESTS THAT WE ADMINISTER ANNUALLY, AND DETECTION IS A CRITICAL PARTOF105THIS PROCESS, BECAUSE WE KNOW THAT EVEN WITH THE BEST PREVENTATIVEMEASURES THAT CHEATING WILL HAPPEN, SO ONE MUST BE PREPARED TODETECT IT WHEN IT DOES. TODAY, I'LL SHARE WITH YOU THE DETECTIONPROCESS.ALL HIGH-STAKES TESTS ARE SUBJECT TO TEST TAKERS AND PROFESSIONALSWHO SEEK TO GAIN OR PROVIDE AN UNFAIR ADVANTAGE. EXPERIENCE HASTAUGHT US THAT THIS TYPE OF BEHAVIOR CAN BE DRIVEN BY ONE'S DESIRETO ACHIEVE HIGHER TEST SCORES IN ORDER TO GAIN ENTRANCE INTOCOLLEGE. IT CAN BE ALSO DRIVEN BY ONE'S DESIRE TO HAVE ONE'SSTUDENT TO DISPLAY PROGRESS AS A RESULT OF HIGH SCORES. IT ALSO CANBE TRIGGERED BECAUSE OF PERSONAL OR FINANCIAL GAIN.TO PROTECT AGAINST TEST COMPROMISE, PROCESSES MUST BE BUILT INTOTHE TESTING PLAN TO DETECT IRREGULARITIES. THE DETECTION PLANSHOULD INCLUDE PROCEDURES TO DETECT IRREGULARITIES THAT OCCURBEFORE THE TEST, DURING THE TEST, AND AFTER THE TEST.THE PRE-ADMINISTRATION PROCESS. IRREGULARITIES THAT OCCUR PRIOR TOTESTING USUALLY INVOLVE TEST MATERIAL COMPROMISE AT THE TESTCENTER. THESE IRREGULARITIES CAN BE FAR-REACHING. EARLY DETECTIONOF TEST SECURITY BREACHES IS ESSENTIAL TO THE CONTAINMENT ORWIDESPREAD COMPROMISE. A GOOD TEST SECURITY PLAN SHOULD INCLUDE AN106AUDIT PROCESS TO DETECT TAMPERING BY STUDENTS AND PROFESSIONALSBEFORE THE TEST IS ADMINISTERED.THE AUDIT SHOULD OCCUR BEFORE TEST DAY OR ON THE MORNING OF THETEST. AND IN ORDER FOR THE AUDIT TO BE EFFECTIVE, ALL TESTMATERIALS SHOULD BE INDIVIDUALLY INVENTORIED TO INSPECT FORMISSING TEST BOOKS OR SIGNS OF TEST REPRODUCTION. SCHOOL OFFICIALSSHOULD BE NOTIFIED IMMEDIATELY OF ANY DETECTION OF IRREGULARITIESSO THAT THE OFFICIALS WILL BE ABLE TO MAKE A DECISION REGARDINGTESTING.AND LASTLY, THE MOST EFFECTIVE PROCESS IS AN UNANNOUNCED AUDIT.THIS IS BECAUSE YOU'RE ABLE TO CATCH THE PERPETRATOR OFF-GUARD.TEST ADMINISTRATION. AN UNFAIR ADVANTAGE THAT CAN BE OBTAINEDDURING TESTING EITHER TO GIVE AN UNFAIR ADVANTAGE OR TO RECEIVE ANUNFAIR ADVANTAGE CAN BE DETECTED BY THE TESTING PERSONNEL IF ACTIVEPROCTORING OCCURS. THE PROCTOR SHOULD CHECK IDENTIFICATION TOCHECK AGAINST IMPERSONATION AND TO PROTECT AGAINST IMPERSONATION.AND MONITOR THE ROOMS FOR WANDERING EYES, SUCH AS COPYING ORCOMMUNICATION, SHOULD ENSURE STUDENTS ARE WORKING ON THECORRECT SECTION OF THE TEST WHEN THERE ARE SCRAMBLED FORMS OF THETEST, BECAUSE A STUDENT WHO'S NOT WORKING ON A CORRECT SECTION OFTHE TEST COULD BE COPYING OFF OF A STUDENT WHO HAS A SCRAMBLED107FORM. OR THE STUDENT COULD BE WORKING ADDITIONAL TIME ON APARTICULAR SECTION IN ORDER TO OBTAIN AN UNFAIR ADVANTAGE.THE PROCTOR SHOULD CHECK FOR UNAUTHORIZED AIDS, SUCH AS A CELLPHONE, TO TRANSMIT AN ANSWER KEY OR TO RECEIVE AN ANSWER KEY;SHOULD DOCUMENT ANY UNAUTHORIZED TEST MATERIAL ACCESS OR UNUSUALBEHAVIOR BY SCHOOL PERSONNEL, SUCH AS WHAT MANY OF YOU MENTIONEDEARLIER TODAY--SIGNALING TO STUDENTS THE CORRECT RESPONSE ORINTENTIONALLY OVERTIMING THE TEST.THE PROCTOR SHOULD ALSO, WHEN COLLECTING THE ANSWER SHEET, INSPECTTHE ANSWER SHEET TO CHECK FOR IMPERSONATION OR THE SWITCHING OF THEANSWER SHEET.SO YOU SEE, IN THIS PROCESS, ACTIVE PROCTORING IS KEY. I HEARD MANYOF YOU GUYS MENTION ABOUT THE TRAINING OF THE PROCTORS AND HAVINGPROCTORS TO BE AWARE. SO OBTAINING AN UNFAIR ADVANTAGE ACTUALLY CANBE A PREVENTIVE IF THE PROCTOR IS ABLE TO OBSERVE THAT DURING THEADMINISTRATION. SO IT SHOULD BE A GOOD EMPHASIS MADE TO THE TESTINGSTAFF TO BE VIGILANT AND TO NOT DO NON-TEST-RELATED ACTIVITIESDURING THE TEST SO THAT THEY WILL BE ABLE TO DETECT THESEIRREGULARITIES.THE POST-ADMINISTRATION PROCESS.108IRREGULARITIES CAN BE DISCOVERED AFTER THE TEST AS WELL THROUGH THESCORING PROCESS. THE SECURITY PLAN SHOULD INCLUDE POST-ANALYTICSTO CHECK FOR IRREGULARITIES SUCH AS UNUSUAL ERASURES, WHICH MANY OFYOU GUYS MENTIONED TODAY, AND I'M LOOKING AT WIDESPREAD UNUSUALERASURES, WHICH COULD BE AN INDICATION OF POSSIBLE TAMPERING OF THEANSWER SHEET.HANDWRITING OR GRIDDING IRREGULARITIES, IF WIDESPREAD, CAN BE ANINDICATION, ALSO, OF TAMPERING OF THE ANSWER SHEET. ESSAYSIMILARITY, IF WIDESPREAD, COULD BE AN INDICATION OF PREKNOWLEDGE. INCONSISTENTPERFORMANCE IN A PARTICULAR SECTION, IF THERE'S A UNIQUE SECTION OFTHE TEST, COULD BE AN INDICATION OF TAMPERING OF THE ANSWER SHEETAND ALSO COPYING.UNUSUAL ANSWER PATTERNS, IF WIDESPREAD, CAN BE AN INDICATION OFPOSSIBLE PRE-KNOWLEDGE AND IT ALSO CAN BE AN INDICATION OF ANSWERSHEET TAMPERING. TEST COMPLETION TIME AND IRREGULARITIES--THIS ISFOR YOUR COMPUTER-BASED TESTING. IF A STUDENT COMPLETES A TESTEXCEPTIONALLY FAST, THIS MIGHT BE AN INDICATION THAT THAT STUDENTHAD AN ANSWER KEY.109AND LASTLY, A LARGE SCORE DIFFERENCE FOR A REPEATER TEST OR SPIKEIN THE MEAN TEST SCORE—THESE ANALYSES CAN BE USED, SOME OF THEM,INDIVIDUALLY, AND MANY OF THEM COULD BE USED COLLECTIVELY IN THEDECISION-MAKING PROCESS.WHEN YOU LOOK AT THESE ANALYSES, YOU HAVE TO DETERMINE "WHAT DOESTHIS DATA MEAN? WHAT IS THIS DATA TELLING ME?" SO, THAT'SIMPORTANT. IT'S JUST NOT ABOUT CONDUCTING THE ANALYSES, BUT WHAT ISIT TELLING ME. SO, FOR INSTANCE, IF YOU HAVE UNUSUAL ERASURES ON ALARGE NUMBER OF ANSWER SHEETS, YOU DON'T STOP THERE. YOU CHECK TOSEE IF THERE'S AN UNUSUAL ANSWER PATTERN, AND THAT MAY HELP YOU TORENDER A DECISION, WHETHER OR NOT A STUDENT OR WHETHER OR NOT APROFESSIONAL PROVIDED AN UNFAIR ADVANTAGE.INTERPRETATION OF THE DATA.THE TEST SECURITY PLAN SHOULD HAVE ESTABLISHED GUIDELINES TOASSIST WITH THE DECISION-MAKING PROCESS. USE DETECTEDIRREGULARITIES AS FLAGS. AND I AGREE WITH A LOT OF YOU GUYS.THESE ARE FLAGS THAT REQUIRE ADDITIONAL INVESTIGATIONS TODETERMINE IF THERE IS A CONCERN. AND WE ALL KNOW WHEN ANALYZINGDATA THAT THIS IS AN ART. NO STATISTICAL EVIDENCE IS TRULYPOSITIVE. WE NEED TO—IN REVIEWING THIS EVIDENCE, WE HAVE TO MAKE110JUDGMENT CALLS. AND THEN WE ALSO HAVE TO KNOW THAT BEHIND ANYANALYSIS WILL BE POLICY QUESTIONS.IN CONCLUSION, DO THESE DETECTION PRACTICES WORK? YES, THEY DO. IDON'T DO A LOT OF INVESTIGATION FOR K-12. I DO INDIVIDUALINVESTIGATIONS FOR TEST PAPERS FOR SCORE INVALIDITY, BUT I HAVEBEEN INVOLVED IN INVESTIGATIONS FOR K-12. AND I CAN TELL YOU THATTHESE ANALYSES AND THESE PROCESSES FOR DETECTING IRREGULARITIESHAVE BEEN VERY EFFECTIVE AT CONTAINING TEST COMPROMISE--SO INSTEADOF MILLIONS OF TEST TAKERS BEING IMPACTED, MAYBE IT'S LIMITED TOA PARTICULAR SCHOOL; FOR PROVIDING EVIDENCE TO CANCEL INVALID TESTSCORES; FOR MITIGATING FURTHER RISK; AND FOR HELPING TO REFINEPROCEDURES.AND WHAT I WANT TO MENTION IS THAT WHEN AN IRREGULARITY ISDETECTED, IT'S KEY FOR THOSE WHO ARE IN CHARGE TO SHARE THISINFORMATION TO EITHER THE SCHOOL DISTRICT OR TO THE STATE AND FORAN INVESTIGATION TO OCCUR. BECAUSE THE ONLY WAY THAT YOU'LL KNOW IFANYONE IS PROVIDING AN UNFAIR ADVANTAGE IS TO CONDUCT ANINVESTIGATION. AND EVEN WHEN I REVIEW TEST SCORES ON AN INDIVIDUALBASIS, WE RECEIVE INQUIRIES INTO OUR OFFICE VIA OUR SECURITYHOTLINE. IT COULD BE AN INQUIRY FROM SOMEONE FROM THE SCHOOLDISTRICT. IT CAN BE AN INQUIRY FROM ANYONE WHO MAY HAVE KNOWLEDGETHAT SOMEONE PROVIDED AN UNFAIR ADVANTAGE.111WHEN WE RECEIVE THAT INFORMATION, WE CONDUCT AN INVESTIGATION. ANDIF NOTHING IS FOUND, WE DON'T ACT ON IT. BUT IF SOMETHING IS FOUND,WE ARE OBLIGATED TO MAKE SURE THAT THOSE SCORES DO NOT STAND.THANK YOU.JACK BUCKLEY: THANKS VERY MUCH. DAVID?DAVID FOSTER: I APPRECIATE THE OPPORTUNITY TO PRESENT TODAY ANDPARTICIPATE IN THIS PANEL. IT'S BEEN VERY INFORMATIVE SO FAR. MYPRESENTATION'S GOING TO BE SOMEWHAT HIGH-LEVEL. I THINK IPRESENT BEST PRACTICES PROBABLY ONLY AS EXAMPLES AMONG THESLIDES, BUT LET'S START OFF WITH--IN DETECTION, WHAT ARE WELOOKING FOR?WE'RE LOOKING FOR AN ACTUAL BREACH, WHETHER IT'S IN PROGRESS,OCCURRED SOME TIME IN THE PAST, OR IT'S JUST A THREAT--IT MAY COMEDOWN THE PIPE PRETTY SOON. THE GOOD NEWS IS THERE ARE ONLY TWOTYPES OF THESE BREACHES AND THREATS.THERE'S CHEATING, WHICH I'VE BEEN TALKING ABOUT AS INAPPROPRIATEATTEMPTS TO INCREASE SCORES; AND THEFT, WHICH DOESN'T CARE A WHOLELOT ABOUT THE IMMEDIATE PROSPECT OF INCREASING SCORES--LATER, OF112COURSE--BUT STEALING TEST CONTENT FOR GAIN. THE BAD NEWS IS THATTHERE ARE HUNDREDS AND PERHAPS THOUSANDS OF WAYS TO CHEAT ON EXAMSOR TO STEAL AND USE THE CONTENT.SO, THESE ARE A FEW EXAMPLES OF SOME TYPICAL SECURITY THREATS ORBREACHES. ON THE CHEATING SIDE, TEACHER TAMPERING WITH ANSWERSHEETS, TEACHERS HELPING STUDENTS DURING THE EXAM, TEACHERSPREPPING THE STUDENTS PRIOR TO THE EXAM, HAVING CERTAIN STUDENTSREMAIN AT HOME ON TEST DAYS. THEN YOU CAN SWITCH THE ATTENTION TOSTUDENTS. CHEATING ON TESTS USING PRE-KNOWLEDGE, ASSISTANCE,CHEAT SHEETS, OTHER TECHNOLOGIES OR OTHER METHODS. BUT THEN YOUALSO HAVE THE THEFT THREATS. A TEACHER OR A STUDENT CAN STEALBOOKLETS. A TEACHER OR STUDENT CAN RECORD QUESTIONS THROUGHOUTTHE EXAM.SO, TODAY, I'M GONNA DISCUSS EACH OF THESE IN ORDER—7 DETECTIONPRINCIPLES. THERE COULD BE MORE. IN FACT, SINCE I'VE SUBMITTEDTHIS, I'VE THOUGHT OF 2 OR 3 OTHERS. THERE JUST WASN'T TIME. BUTFOCUS. CONCENTRATE ON THE HIGH-RISK THREATS FIRST. ALMOST IGNORETHOSE THAT ARE INFREQUENT OR ARE LESS OF A THREAT.ADAPT. YOU NEED TO MATCH YOUR DETECTION METHODS TO THE THREAT.OFTEN WE LOOK IN DIFFERENT DIRECTIONS.113BACK-UP. USING SEVERAL LAYERS OF DETECTION METHODS.PREDICT. WATCH OUT FOR THE FUTURE. WHAT CAN WE EXPECT NEXT? LET'SNOT GET SURPRISED.FILTER. RULE OUT OTHER EXPLANATIONS.EVALUATE. USE YOUR DETECTION METHODS TO EVALUATE WHAT YOU ARE DOINGAND THEN PLANNING A LITTLE MORE SPECIFICALLY.SO, LET'S GO THROUGH THESE.SO, FOCUS. CONCENTRATE ON THE HIGH-RISK THREATS FIRST. WHATCONSTITUTES A HIGH-RISK THREAT OR A HIGH-DAMAGING BREACH? THELIKELIHOOD OF THE THREAT, HOW IMMEDIATE IT IS. WHAT'S THE AMOUNT OFPOTENTIAL DAMAGE IT CAN CAUSE? HOW DIFFICULT IS IT TO PREVENT?MAYBE IT'S BETTER TO--IF YOU CAN'T PREVENT IT, YOU MAY WANT TODETECT IT. IT'S DIFFICULT TO DETECT AND MITIGATE AND/OR IT USESVULNERABILITIES IN OUR SYSTEMS.FOR EXAMPLE, ONE TEACHER CHEATING BRINGS MUCH GREATER DAMAGE IN AHIGHER RISK THAN ONE STUDENT TEACHING--I'M SORRY. ONE STUDENTCHEATING.114ADAPT. MATCH YOUR DETECTION METHODS TO THE THREAT. NOW, THREATSDEPEND ON A NUMBER OF CIRCUMSTANCES. THEY'RE NOT THE SAME AT GRADELEVELS. THREATS AT THIRD GRADE ARE DIFFERENT THAN THREATS AT 11THGRADE. THEY'RE DIFFERENT FOR SCIENCE AND FOR MATH. THE DELIVERYMODE, WHETHER IT'S PAPER OR PENCIL OR COMPUTERIZED TESTING AFFECTSTHE RISK. THE TESTING HISTORY OF A SCHOOL, A TEACHER, A STUDENT MAYCHANGE THE WAY YOU APPROACH DETECTION.AND MOTIVATION. OFTEN, TESTS THAT WEREN'T INTENDED TO BE HIGHSTAKES HAVE BECOME HIGH STAKES. AND NOW YOU NEED TO FOCUS ATTENTIONON THAT. WE OFTEN TEND TO USE EITHER ONE TOOL FOR DETECTION ORFOCUS WHAT WE HAVE IN THE WRONG DIRECTION. AND I MENTION THATEXAMPLE THAT THE THREATS AT 3RD GRADE MAY INVOLVE MORE TEACHERCHEATING, WHEREAS THREATS AT 11TH GRADE, BECAUSE THE STAKES AREHIGHER FOR THE STUDENTS, MAY INVOLVE--THE RISK MAY BE MUCH MOREWITH STUDENT CHEATING.BACK UP. VISA USES SEVERAL METHODS TO DETECT IF YOUR CARD IS BEINGUSED INAPPROPRIATELY. LIKEWISE, WE SHOULD USE SEVERAL LAYERS OFDETECTION METHODS, AND I THINK DR. JACOB'S ALSO MENTIONED THIS.I'LL DO IT IN A LITTLE BROADER LEVEL, THOUGH. USE FORENSICSTATISTICAL METHODS AS HAVE BEEN DISCUSSED.115CONTINUOUS OBSERVATION. WHAT DOES IT TAKE TO WATCH A STUDENT WHOMYOU MAY SUSPECT OF TAMPERING WITH ANSWER SHEETS? IT'S KIND OF LIKEPROCTORING THE PROCTORS. USE OF A TIPLINE. I'VE BEEN SURPRISEDRECENTLY BY THE NUMBER OF...CHEATING AND TESTING AND THEFTPROBLEMS THAT HAVE COME TO OUR ATTENTION SIMPLY BECAUSE OF TIPS.WEB MONITORING. QUESTIONS THAT ARE STOLEN ARE ALMOST ALWAYS SHAREDACROSS THE WEB. THAT'S ONE WAY TO FIND OUT ABOUT THEM. SO, ATEACHER CHEATING—I THINK ALL OF THESE 4 METHODS HERE AND MAYBEOTHERS THAT I HAVEN'T LISTED SHOULD BE ENLISTED TO DETECT THEPROBLEM.AND JUST A LITTLE ASIDE, IT'S PROBABLY BETTER TO DETECT THESETHINGS EARLY THAN LATE. SOMETIMES WE DON'T DETECT PROBLEMS UNTILSEVERAL WEEKS, MAYBE SEVERAL MONTHS AFTER THE PROBLEM HASOCCURRED. IF YOU CAN DETECT THE THREAT AND MITIGATE THE THREAT ORDETECT THE PROBLEM IN PROCESS, THE BREACH IN PROCESS, SECURITY WILLBE ENHANCED.PREDICT. WATCH OUT FOR NEW THREATS. THEY COME ALONG EVERY DAY.CHANGING STAKES WILL AFFECT THOSE. NEW TECHNOLOGIES MAKES CHEATINGAND THEFT LESS DETECTABLE. I MEAN, THE SMALL LITTLE 2-WAY RADIOS INTHE EAR—THEY CAN'T BE SEEN BY TEACHERS, BUT A STUDENT CAN SPEAK AND116RECEIVE INFORMATION DURING AN EXAM. THE CREATIVITY OF CHEATERS ANDTHIEVES IS ALWAYS HIGH.AND I JUST PUT AN EXAMPLE HERE. THEY'RE ON THE MARKET FOR LESS THAN$100. YOU CAN BUY THESE DEVICES THAT HAVE LENSES IN THEM. THEY CANHOLD UP TO 16 GIGABYTES OF INFORMATION AND ESSENTIALLY RECORD ANENTIRE SET OF EXAM CONTENT AND BE PRETTY MUCH UNDETECTABLE BYANYBODY WALKING AROUND THE ROOM. I DON'T HAVE A GOOD SOLUTION TOTHOSE, BUT THAT'S NOT THE POINT OF MY TALK TODAY ANYWAY, SO...FILTER. THIS HAS BEEN MENTIONED A COUPLE OF TIMES. RULE OUT OTHEREXPLANATIONS. FOR FAIRNESS PURPOSES, YOU DON'T WANT TO ACCUSESOMEONE OF CHEATING OR SPEND A LOT OF EFFORT IN THAT DIRECTIONWHEN, IN FACT, CHEATING DIDN'T HAPPEN.TESTING IRREGULARITIES, FROM THE GRAPHIC ON THE SLIDE, ARE MANY.THOSE SHOULD BE FILTERED AND EFFORTS SHOULD BE MADE TO DETECTAND RESPOND TO ACTUAL CHEATING OR OTHER KIND OF TEST FRAUDEVENTS. AND THE EXAMPLE I APPLY HERE, I SAW JUST RECENTLY.A STATISTICAL ANALYSIS DETECTED A KIND OF COLLUSION AMONG ANUMBER OF STUDENTS IN A CLASSROOM, WHEN REALLY ALL THEY DID WASDECIDE TO FINISH UP THE TEST BY EVERYBODY ANSWERING "C." IT WASN'T117CHEATING. IT WAS BAD TESTING PRACTICE, BUT NOT AN EXAMPLE OFCHEATING. IT WAS CERTAINLY A TESTING IRREGULARITY.EVALUATE. YOU SHOULD USE DETECTION METHODS TO EVALUATE YOURSECURITY--BOTH HOW WELL YOU'RE DETECTING AS WELL AS OTHER METHODS.THIS IS A GRAPH OF A STATE GIVING THE SAME STATISTICAL ANALYSISOVER 5SEPARATE YEARS IN THE SPRING. AND IT SHOWS A DROP IN DETECTEDPROBLEMS, BECAUSE THEY INSTITUTED SCORE INVALIDATIONS AND BETTERTRAINING PROCEDURES AFTER THAT FIRST YEAR.PLAN. SET UP TO SUCCEED. THIS IS AN INTERESTING--YOU CAN MODIFY ANDSHOULD MODIFY YOUR DETECTION METHODS TO DETECT PARTICULARPROBLEMS. PRE-KNOWLEDGE IS VERY DIFFICULT TO DETECT. NO ONE CAN SEETHAT A STUDENT COMES TO A TEST WITH ALL THE ANSWERS IN THEIR HEADS.YOU CAN, HOWEVER, PUT IN SPECIFIC QUESTIONS IN THE EXAM TO TELLWHETHER THEY'RE USING PRE-KNOWLEDGE OR NOT.NOW, IT'S A TEST-DESIGN COLLABORATION WITH THE SECURITY FOLKS TO GOAFTER A PARTICULAR THREAT. BUT IN THIS CASE, THEY WERE USED IN 88%OF THE EXAMS, AND THIS KIND OF METHOD COULD PRETTY MUCH DETECTEVERY INSTANCE.118JUST A LITTLE BIT OF AN EXAMPLE HOW THEY MIGHT DIFFER, THEN I'MDONE. HIGH RISK AT GRADE 3 MIGHT BE TEACHER TAMPERING WITH ANSWERSHEETS. I THINK THIS HAS BEEN MENTIONED SEVERAL TIMES TODAY. DIRECTDETECTION WOULD BE TO PHYSICALLY OBSERVE THE TEACHER. IF THETEACHER CAN'T BE REMOVED AND SOMEONE ELSE REPLACED, THEN OBSERVETHAT TEACHER FROM THE TIME YOU GATHER UP THE ANSWER SHEETS TO WHENTHEY HAND THEM OFF. IF THEY ATTEMPT SOMETHING, YOU'LL DETECT IT.INDIRECTLY, YOU CAN SEE THAT THROUGH FORENSICS EFFORTS, UNUSUALLYHIGH RATES OF WRONG TO RIGHT ERASURES, AND UNEXPECTED SCORE GAINSWILL GIVE SOME INDIRECT EVIDENCE THAT'S HAPPENING.ANOTHER INDIRECT METHOD IS TO INSTITUTE A TIPLINE, WHERE SOMEONEWILL LET YOU KNOW THIS IS HAPPENING. AND THE LAST ONE IS 11TH GRADE,JUST AS A DIFFERENCE. STUDENT CHEATING BY COPYING FROM ANOTHERSTUDENT.DIRECT. YOU USE TRAINED PROCTORS WHO VIGILANTLY OBSERVE THESTUDENTS DURING THE EXAM. THAT'S A GOLD STANDARD METHOD.INDIRECTLY. SIMILARITY AND COLLUSION ANALYSES AND MAYBE LOOKING ATSOME ABERRANT RESPONDING. THEY DON'T ALWAYS GET TO SEE WHAT THEIRSTUDENTS ARE DOING.AND FINALLY, AGAIN, A TIPLINE IS OF VALUE. SO, THANK YOU VERY MUCH.119JACK BUCKLEY: DAVE, THANK YOU VERY MUCH.AND MELISSA?MELISSA FINCHER: OK.GOOD MORNING. HOPE TO BE ABLE TO SHARE SOME OF THE EXPERIENCESWE'VE GONE THROUGH IN GEORGIA, AND I HOPE THE INFORMATION ISHELPFUL.IN GEORGIA, WE HAD BROADBASED COMMITMENT OF MULTIPLE AGENCIES. OFCOURSE, THE GEORGIA DEPARTMENT OF EDUCATION WAS INVOLVED, AS WE'RERESPONSIBLE FOR THE DEVELOPMENT, ADMINISTRATION, SCORING, ANDREPORTING OF THE STATE TESTS, INCLUDING SECURITY PROCEDURES ANDPROTOCOLS. THE OFFICE OF STUDENT ACHIEVEMENT ACTUALLY LED THE WORKINTO THE DETECTION IN THE STATE OF GEORGIA. THEY ARE STATUTORIALLYCHARGED WITH THE AUDIT FUNCTION, AND WE WORKED VERY CLOSELY WITHTHE OFFICE OF STUDENT ACHIEVEMENT.THE GOVERNOR'S OFFICE BECAME INVOLVED, AND THAT WAS VERYIMPORTANT. THE GOVERNOR APPOINTED SPECIAL INVESTIGATORS, AND LATERON ANOTHER PANEL, YOU'LL HEAR FROM ONE OF OUR SPECIALINVESTIGATORS.120THE PROFESSIONAL STANDARDS COMMISSION IN THE STATE OF GEORGIA ISALSO INVOLVED IN THIS WORK IN THAT THEY HAVE ETHICS STANDARDS FOREDUCATORS, AND THEY ALSO ARE THE CREDENTIALING AGENCIES. AND INGEORGIA, THERE ARE STAKES ASSOCIATED WITH INAPPROPRIATE PRACTICESDURING ASSESSMENT. WE DO REQUIRE THAT ONLY CERTIFIED EDUCATORSADMINISTER OUR STATE ASSESSMENT PROGRAM AS ONE DETERRENT.IN TERMS OF THINGS THAT STATES NEED TO CONSIDER, IT'S VERYIMPORTANT TO UNDERSTAND THE TYPES OF ANALYSES THAT ARE AVAILABLEAND WHAT INFORMATION THEY CAN CONTRIBUTE. YOU HAVE TO LOOK AT THESTRUCTURE OF YOUR ASSESSMENT PROGRAM, THE INFORMATION THAT YOUCAPTURE.FOR INSTANCE, WE WERE ABLE TO KNOW WHO ADMINISTERED THE TEST.WHAT'S DIFFERENT IN GEORGIA--WE DO HAVE THE HEADER SHEETS THAT DR.CIZEK TALKED ABOUT, BUT WE DO ALLOW SYSTEMS TO GROUP THEIR ANSWERDOCUMENTS IN A WAY THAT IS MEANINGFUL FOR THEM, SO THAT THE RESULTSCOME BACK IN A STRUCTURALLY MEANINGFUL WAY.HOWEVER, SYSTEMS ARE RESPONSIBLE FOR MAINTAINING RECORDS ON WHOADMINISTERED THE TEST AND HOW STUDENTS WERE GROUPED. IT'S ALSOIMPORTANT TO UNDERSTAND, AS YOU'VE HEARD THROUGHOUT THIS121MORNING, THAT THERE'S NO SINGLE ANALYSIS OR COMBINATION OFANALYSES THAT IS DEFINITIVE. YOU DON'T GET THE ANSWER, ADDITIONALINVESTIGATION IS GOING TO BE WARRANTED. THAT IS SOMETHING THAT YOUCAN COUNT ON.HAVING A CLEAR RATIONALE FOR THE FLAGGING CRITERIA THAT YOU USE ISVERY IMPORTANT, AND YOU NEED TO THINK AHEAD OF TIME OF HOW YOU'REGOING TO FLAG, AT WHAT LEVEL YOU'RE GOING TO FLAG.THAT FLAGGING CRITERIA WILL, IN TURN, BECOME A WAY TO INTERPRET THERESULTS, AND IT WILL BE USED BY DISTRICTS AND SCHOOLS INUNDERSTANDING. WITH THAT SAID, IT'S VERY IMPORTANT TO BE VERY CLEARABOUT WHAT YOU'RE FLAGGING, AND THAT'S SOMETHING THAT WE'VE HEARD,AGAIN, FREQUENTLY THIS MORNING.WHAT YOU'RE FLAGGING ARE DATA ANOMALIES. YOU'RE NOT FLAGGINGSOMETHING THAT'S DEFINITIVE. YOU'RE FLAGGING SOMETHING THAT NEEDSTO BE INVESTIGATED FURTHER. THERE IS A REASON IN GEORGIA WE WANTSTUDENTS TO USE NUMBER 2 PENCILS. WE DID ERASURE ANALYSES. WE WANTSTUDENTS TO HAVE THAT ERASURE AS A TOOL THAT'S NEEDED IN ALEGITIMATE MANNER. AS YOU CAN IMAGINE, ONE OF THE FIRST REACTIONSTHAT WE GOT WAS, "WE'RE JUST GONNA REMOVE THE ERASERS." THAT'S NOTA GOOD SOLUTION.122[LAUGHTER]AS YOU'RE WORKING THROUGH THIS, AS YOU'RE CHOOSING YOURCONTRACTOR, MAKE SURE THAT THEY HAVE SUFFICIENT QUALITY CONTROLMECHANISMS IN PLACE. YOU NEED TO PLAN FOR INDEPENDENT VERIFICATIONAND REPLICATION OF THE ANALYSES. YOU CAN EXPECT TO BE CHALLENGED.AS ASSESSMENT FOLKS, WE'RE USED TO THAT, BUT THERE'S AN EMOTIONALCOMPONENT TO THIS WORK AS WELL. AS MUCH AS YOU CAN--THIS ISCOMPLICATED WORK, AND IT'S DIRTY WORK AND IT'S MESSY WORK--TRYYOUR BEST TO KEEP THINGS SIMPLE, STRAIGHTFORWARD, AND CONSISTENT.COMMUNICATION IS KEY HERE. THESE ARE TERMS AND ANALYSES THAT WE'REFAMILIAR WITH. WE DEAL WITH STANDARD DEVIATIONS QUITE A BIT IN OURWORK, BUT AT THE SCHOOL LEVEL THEY DON'T. SO THEY NEED A REFRESHERCOURSE AND NEED A LENS TO UNDERSTAND THAT WORK.SO, THEY'LL NEED QUITE A BIT OF HELP. DISTRICTS AND SCHOOLS WILLNEED QUITE A BIT OF SUPPORT AND HELP IN UNDERSTANDING ANDINTERPRETING THE DATA. OPEN AND COMPREHENSIVE COMMUNICATION ISESSENTIAL AND BEING CONSISTENT WITH THAT COMMUNICATION. YOU CANEXPECT THINGS TO RAMP UP AND GET COMPLICATED REALLY, REALLYQUICKLY. IT'S ALSO IMPORTANT THAT YOU'RE VERY CLEAR ON WHAT DATACAN BE SHARED AND WHAT DATA CAN'T BE SHARED. THIS IS BASED ON123SECURE ASSESSMENT DATA, AND SO THERE ARE SOME PIECES THAT CAN'T BESHARED.IT'S IMPORTANT TO HAVE A VERY CLEAR PLAN FOR NEXT STEPS. THEANALYSIS IS JUST ONE PART. SO, I MENTIONED THE FACT THAT FOLLOW-UPINVESTIGATIONS ARE GOING TO BE REQUIRED. HAVE A PLAN FOR WHO WILLCONDUCT THOSE. IF IT IS THE DISTRICT, YOU NEED TO EXPECT THAT THEYWILL NEED SUPPORT. THEY WON'T KNOW WHERE TO BEGIN, NECESSARILY,BECAUSE THIS WILL BE NEW WORK TO THEM. THEY'VE INVESTIGATEDINDIVIDUAL CASES IN THE PAST, BUT MAYBE NOT NECESSARILY THINGS ONTHE MAGNITUDE OF WHAT WE SAW IN THE STATE OF GEORGIA.CERTAINLY, THE CHAIN OF CUSTODY IS CRITICAL, BUT THAT SHOULD NOT BETHE SOLE FOCUS. AND THEN, QUITE FRANKLY, YOU NEED TO HAVE A PLANFOR WHAT WE ENCOUNTERED IN GEORGIA, AND THAT IS WHEN THE DISTRICTREFUSES TO INVESTIGATE OR DOESN'T PUT FORTH THEIR BEST EFFORT. MAKESURE THAT YOUR ASSESSMENT POLICIES ADDRESS RECORDS RETENTION. ITALKED ABOUT THAT A LITTLE BIT ALREADY IN MAKING SURE THAT THOSERECORDS ARE RETAINED. WHO ADMINISTERED THE TEST? TO WHICHSTUDENTS? WHO PROCTORED? WE'VE SEEN SEATING CHARTS, SOMETHINGTHAT WE PROBABLY NEED TO ADD IN THE STATE OF GEORGIA. WHO HADPOSSESSION OF THE SECURE MATERIALS AND FOR HOW LONG? WE HAVE CHECKIN AND CHECK-OUT PROCEDURES RECORDING TIMES SO THAT IF MATERIALSAREN'T RETURNED FOR AN HOUR AFTER ADMINISTRATION, THAT IS124DOCUMENTED AND THAT'S SOMETHING THAT CAN BE INVESTIGATED.KNOWING WHO HAS ACCESS TO THAT SECURE LOCATION WHERE MATERIALS ARESTORED OVERNIGHT, THAT KIND OF THING. THOSE ARE VERY IMPORTANTPROTOCOLS THAT NEED TO BE IN PLACE, AND THOSE RECORDS NEED TO BEMAINTAINED.THINK ABOUT THE CHANGES IN YOUR TEST PROCEDURES THAT WILL TAKEPLACE DURING THE NEXT ADMINISTRATION. IN GEORGIA, THE OFFICE OFSTUDENT ACHIEVEMENT REQUIRED THE ROTATION OF TEACHERS IN SCHOOLSTHAT WERE FLAGGED SEVERE AND MODERATE. WE SENT IN STATE MONITORS.WE'D ALWAYS DONE RANDOM CHECKS OF STATE MONITORS, BUT HERE WAS ACONCERTED EFFORT TO SEND IN STATE MONITORS FOR THE DURATIONOF THE ADMINISTRATION. THE MONITORS OBSERVE THE TEST ENVIRONMENT,AND AT THE CONCLUSION OF TESTING EACH DAY, THEY SECURED THE ANSWERDOCUMENTS THEMSELVES AND WERE THERE IN THE MORNING TO OPEN ANDRETRIEVE THE ANSWER DOCUMENTS.SO WE RESTRICTED ACCESS TO THE ANSWER DOCUMENTS. AN IMPORTANTCONSIDERATION FOR STATES AND DISTRICTS GOING INTO THIS IN TERMS OFSTATE MONITORS, THE SELECTION AND TRAINING OF THOSE MONITORS ISESSENTIAL. CERTAINLY, WE DO THAT IN THE STATE OF GEORGIA, BUT IT'SIMPORTANT THAT THE MONITORS NOT BE OVERZEALOUS AND UNDERSTAND125THAT THE SCHOOL IS STILL RESPONSIBLE FOR THE ADMINISTRATION.THEY'RE THERE TO OBSERVE THE TEST ENVIRONMENT AND TO SECURE THEANSWER DOCUMENTS AND THEN REPORT ANYTHING THAT IS OF CONCERN. YOUNEED TO TAKE IT TO THE NEXT LEVEL IF INAPPROPRIATE BEHAVIOR ISFOUND.THINK ABOUT WHAT SUPPORTS WILL BE PROVIDED TO THE STUDENTS WHORECEIVED INACCURATE SCORES. HOW WILL YOUR ACCOUNTABILITYDETERMINATIONS BE ADJUSTED? WHAT OTHER PROGRAMS AND DECISIONS AREIMPACTED BY THIS BAD DATA? IT'S VERY IMPORTANT THAT YOU HAVE THATPLANNING DONE UP FRONT. CONSIDER THE CAPACITY AT ALL LEVELS-CERTAINLY, AT THE SCHOOL DISTRICT, STATE, AND CONTRACTOR LEVEL.THIS HAS REALLY PUSHED OUR CAPACITY. IT'S BEEN A GOOD THING,NONETHELESS. IT'S SOMETHING THAT NEEDED TO OCCUR.CONSIDER THE TIMING OF THE REPORTS AND THE INVESTIGATIONS. IFYOU'RE LIKE MY STATE, THEY WANT THE DATA IMMEDIATELY FOLLOWING.THEY WANT IT NEXT DAY, FASTER, MORE, MORE, MORE, BECAUSE THEY NEEDTO MAKE DECISIONS. WE HAVE PROMOTION AND RETENTION. SO, OFTEN, THEDATA WILL COME BACK AFTER SCORES ARE BEING REPORTED, AND IT'S VERYHARD TO INVALIDATE AND PULL BACK A SCORE ONCE IT'S BEEN RELEASED.SO, CONSIDER THAT DOMINO EFFECT.126WHO WILL NEED TO BE NOTIFIED OF THE SUBSTANTIATED DATAINACCURACIES. THERE ARE USERS OF THE DATA THAT HAVE ALREADYRECEIVED DATA FOR GRANTS OR WHAT-HAVE-YOU THAT NEED TO BE NOTIFIEDTHAT THE DATA IS NOT ACCURATE. AGAIN, KEEP DETAILED RECORDS OF ALLTHE STEPS FOLLOWED, AND THEN THINK AS BROADLY AND ASCOMPREHENSIVELY AS YOU CAN.JACK BUCKLEY: MELISSA, THANK YOU VERY MUCH.SO, AGAIN, AS BEFORE, WE'LL BEGIN BY ME TRYING TO THROW A COUPLEQUESTIONS AT THE PANEL AND BEFORE I OPEN IT UP TO THE REST OF US.SO, DAVID, I THINK I'M STEALING THIS FROM YOU, BUT WE'RE TALKINGABOUT DETECTION. I THINK WE CAN CATEGORIZE METHODS INTO TWO BROADFAMILIES.FIRST, THEY'RE SORT OF DIRECT, RIGHT? SO, IN THIS CASE, WE'RETALKING--WHAT I'M HEARING IS THINGS WE'RE DOING IN OBSERVATION ORMONITORING OR PROCTORING. BUT ALSO, CARSWELL, DISCUSSIONS ABOUTTHINGS LIKE MATERIALIC INSPECTION, SO LOOKING AT SHRINK WRAP,ACTUALLY DOING PHYSICAL INSPECTION OF THE SPACE OR THE TESTMATERIALS.127AND SECONDLY, I THINK, DAVID, WHAT YOU CHARACTERIZE AS INDIRECT,WHICH IS PROBABLY WHAT MOST PEOPLE THOUGHT WE WERE GONNA SPEND MOSTOF THE TIME TALKING ABOUT HERE, WHICH IS SORT OF STATISTICALFORENSICS, TIPLINES, OTHER SORT OF REPORTING MECHANISMS.I WANT TO GET TO MELISSA, SOMETHING YOU SAID, BECAUSE IT ALSOSPEAKS TO WHAT CHANCELLOR HENDERSON SAID THIS MORNING ABOUTTRANSPARENCY AROUND THE STANDARDS IN INDIRECT REPORTING,PARTICULARLY STATISTICAL FORENSICS.SO YOU MENTIONED SOMETHING ABOUT THE IMPORTANCE OF COMMUNICATIONAND MAKING SURE THAT YOU TELL PEOPLE ESSENTIALLY WHAT ARE YOU GONNABE FLAGGING. WHAT IS YOUR THRESHOLD FOR DETECTING AN IRREGULARITYIN ADVANCE? I'M CURIOUS. HOW DO YOU DO THAT? HOW DO YOU SET THAT?WHAT KINDS OF COMMUNICATION METHODS DO YOU USE, AND DOES EVERYONEELSE AGREE THAT SORT OF TRANSPARENCY THERE IS IMPORTANT, THAT WEWANT TO BE VERY CLEAR ABOUT WHAT WE'RE USING TO LEAS AND TOSCHOOLS, OR IS THERE A COUNTERPOINT WHERE MAYBE WE DON'T WANT TOTELL THEM NECESSARILY WHAT THE THRESHOLDS ARE IN ADVANCE?MELISSA FINCHER: I THINK IN GEORGIA, IN ALL HONESTY--I DON'T MEAN TOSPEAK FOR THE OFFICE OF STUDENT ACHIEVEMENT, BUT THEY ARE THE ONESTHAT SET THAT FLAGGING CRITERIA AND CARRY THE BURDEN OF MUCH OFTHAT COMMUNICATION. TO A CERTAIN EXTENT, IT WAS TRIAL AND ERROR.128THE FIRST TIME OUT. IT WAS LOOKING AT THE DATA, TRYING TO MAKESENSE OF IT, FIELDING QUESTIONS FROM THE FIELD IN WAYS TO HELP THEMBETTER UNDERSTAND. YOU HAVE TO LOOK AT THIS AS A PROCESS OFCONTINUAL IMPROVEMENT, BUT I THINK IT IS IMPORTANT SINCE THOSEFLAGS ARE GOING TO BE USED AS INTERPRETATION TO BE VERY CLEAR ABOUTWHAT THEY MEAN. I'M NOT SURE THAT WE HAVE IT RIGHT YET. WE USE 3STANDARD DEVIATIONS IN THE STATE OF GEORGIA. I NOTICE THAT SEVERALSTATES USE 4 STANDARD DEVIATIONS. I DON'T KNOW THAT WE DON'T HAVEIT RIGHT, BUT HAVING A RATIONALE FOR WHAT YOU'RE USING IS VERYIMPORTANT.JACK BUCKLEY: AND WHEN YOU SAY 3 STANDARD DEVIATIONS, WE'RE TALKINGABOUT--MELISSA FINCHER: WRONG TO RIGHT.JACK BUCKLEY: FREQUENCY OF ERASURES OR WTR, WRONG TO RIGHT ERASURESIN THAT CASE.MELISSA FINCHER: YES.JACK BUCKLEY: BRIAN, YOU MENTIONED THE NEED FOR ROBUST STATISTICSWHEN129YOU'RE SETTING THRESHOLDS. WE'RE WORRIED ABOUT OUTLIERS THAT MIGHTBE--I'M SORRY. IN ONE OF YOUR POINTS, YOU HAD MENTIONED THE NEED ORRECOMMENDED THE USE OF ROBUST STATISTICS IN TERMS OF STATISTICALFORENSICS. SO ANY PARTICULAR RECOMMENDATIONS? AGAIN? SO, WE WOULDSEND SOMETHING--BRIAN JACOB: [INDISTINCT] I THINK AS MUCH AS KIND OF ONE SINGLEMEASURE WOULD BE THE GENERAL RULE OF TESTING [INDISTINCT] THESENSITIVITY OF FLAGGING OR OTHER ANALYSIS TO A VARIETY OFAPPROACHES. FOR EXAMPLE, IF YOU'RE LOOKING AT CLASSROOM OR SCHOOLSTATISTICS OR NUMBER OF WRONG TO RIGHT, PROBABLY THE MEDIAN RATHERTHAN THE MEAN NUMBER OF WRONG TO RIGHT WOULD BE A MORE ROBUSTSTATISTIC. NOW, AT THE SCHOOL LEVEL, IT MADE A WEALTH OF DIFFERENCEFOR THIS VERY LARGE SAMPLE, BUT IN SOME SMALL CLASSES WHERE YOUONLY HAVE 10 OR 11 STUDENTS, YOUR ONE STUDENT GOING OFF GRID ANDHAVING TO ERASE HALF THE TEST CAN CHANGE YOUR AVERAGE WRONG TORIGHT FOR THE ENTIRE CLASSROOM. NOW, YOU EITHER USE THE[INDISTINCT] LIKE THAT OR HOPEFULLY THAT SORT OF THING IS CAUGHT INKIND OF A FOLLOW-UP INVESTIGATION WHERE YOUR [INDISTINCT]IRREGULARITY.JACK BUCKLEY: ANY OTHER THOUGHTS ON TRANSPARENCY HERE IN TERMS OFWHAT SHOULD YOU COMMUNICATE TO THE FOLKS WHO ARE ACTUALLY130ADMINISTERING THE ASSESSMENT IN TERMS OF WHAT YOU WILL BE FLAGGINGOR LOOKING AT? DAVID, WHAT DO YOU GUYS RECOMMEND?DAVID FOSTER: I'M A BIG FAN OF TRANSPARENCY AND DISCLOSURE MAINLYBECAUSE IT DETERS AND PREVENTS CHEATING. IF THEY KNOW YOU HAVE ASTRONG, SOPHISTICATED, COMPREHENSIVE SYSTEM IN PLACE, PEOPLE ARELESS LIKELY TO TRY TO GAME AND GO UP AGAINST IT, BECAUSE THE FEARWILL BE--SO, THE MORE YOU CAN PROVIDE INFORMATION ON HOW THAT'SDONE, THE MORE CONFIDENCE THE INDUSTRY AND PEOPLE HAVE IN WHATYOU'RE DOING AND IT WILL AFFECT THESE BEHAVIORS WE'VE BEEN TALKINGABOUT. SO, I'M A BIG FAN OF IT.CARSWELL WHITEHEAD: BUT AS IT RELATES AS FAR AS THE AUDITING PROCESS,THAT'S THE REASON WHY I SAID THAT AN UNANNOUNCED AUDIT IS MOREEFFECTIVE, BECAUSE YOU'RE CATCHING ANYONE WHO'S ATTEMPTING TOREMOVE TEST MATERIALS OR SHARE TEST MATERIALS WITH STUDENTS BEFORETHE TEST. YOU'RE ABLE TO DETECT THAT BY HAVING AN UNANNOUNCEDAUDIT. WHAT YOU CAN ALSO HAVE IS AN AUDIT THAT WILL OCCUR AFTER THETEST TO ENSURE THAT ALL OF THE MATERIALS HAVE BEEN RETRIEVED FROMTHE TESTING STAFF AND ALL OF THE MATERIALS ARE ACCOUNTABLE, BECAUSEWHAT YOU'RE LOOKING AT NOW IS MAKING SURE THAT YOU HAVE ALL OF THEMATERIALS WILL THEN MITIGATE ANY TYPE OF RISK FOR A COMPROMISE AT AFUTURE ADMINISTRATION. AND I MENTIONED, TOO, THAT WHEN THESE TYPESOF IRREGULARITIES IN REGARDS TO TEST MATERIALS ARE DETECTED, YOU131MUST ACT VERY QUICKLY, BECAUSE YOU'RE NOT ONLY LOOKING AT IMPACTINGTHE LOCATION WHERE THE IRREGULARITY OCCURRED. YOU'RE LOOKING ATIMPACTING MANY SCHOOLS THAT WILL BE ADMINISTERING THAT TEST.TRAINING OTHER TESTING STAFF IS VERY KEY FOR THE TESTING STAFF TOBE VIGILANT. AND I ALSO THINK, TOO, AS FAR AS WITH STUDENTS, HAVINGA WAY FOR STUDENTS TO BE ABLE TO SHARE IF THEY RECEIVED ANY TYPE OFPRE-KNOWLEDGE. IN MY OFFICE, WE HAVE A HOTLINE WHERE TEST TAKERSARE ABLE TO CALL IN OR TEST TAKERS ARE ABLE TO SEND AN E-MAIL TOADVISORS IF ANYONE RECEIVED AN UNFAIR ADVANTAGE. IF THEY HAVE THISOPPORTUNITY TO SHARE THIS INFORMATION, THEY WILL BRING THISINFORMATION TO OUR ATTENTION VERY EARLY IN THE PROCESS EVEN BEFORESCORES ARE REPORTED, BECAUSE WHAT I'M HEARING IS THAT SOMETIMESYOU'RE NOT FINDING OUT ABOUT THESE IRREGULARITIES UNTIL SCORES AREREPORTED. IF YOU HAVE A WAY FOR THIS INFORMATION TO BE CONVEYED TOYOU PRIOR TO SCORES BEING REPORTED, THEN YOU'RE ABLE TO PREVENTTHOSE SCORES FROM BEING RELEASED.JACK BUCKLEY: ON THAT TOPIC OF TIPLINES, DAVID, I THINK YOURECOMMENDED THEM AS WELL IN THE K-12 ENVIRONMENT. IN A TYPICALSTATE TESTING PROGRAM, WHAT'S THE SORT OF VOLUME OF TIPS AND WHAT'STHE FALSE POSITIVE/FALSE NEGATIVE RATE LOOK LIKE, JUST OUT OFCURIOSITY, IF ANYONE HAS EXPERIENCED IT?132MELISSA FINCHER: I CAN'T TELL YOU ABOUT THE VOLUME. WE DEFINITELY GETIT. OFTENTIMES, IT COMES FROM PARENTS, FROM STUDENTS GOING HOME ANDTALKING ABOUT THE TEST ENVIRONMENT AND TALKING, PERHAPS, ABOUT HOW"IT WASN'T AS HARD AS I THOUGHT IT WAS. IT WAS JUST EVERYTHING THETEACHER COVERED THE DAY BEFORE" OR "THE TEACHER WAS VERY HELPFUL,"THAT KIND OF THING. WE PROBABLY GET 20 TO 30 OF THOSE EACHADMINISTRATION. WHAT WE'RE SEEING NOW IS--WHICH DOESN'T SOUND LIKEIT'S VERY MANY. WHAT WE'RE SEEING NOW IS THINGS COMING IN AFTER THEFACT, RATHER THAN DURING THE ADMINISTRATION. AND THAT'S A LITTLEBIT MORE DIFFICULT TO DEAL WITH.DAVID FOSTER: AND I DON'T HAVE ANY COMPARATIVE EXPERIENCE LIKEMELISSA DOES, BUT I'D BE VERY CONCERNED ABOUT WHAT IS COMING IN.THERE MUST BE A LOT OF SOCIAL CONSTRAINTS TO NOT BLOW THE WHISTLE,NOT TO TIP, AND SO IT SHOULD BE ONE--AS OTHERS HAVE MENTIONED, TOO-IT SHOULD BE ONE OF MANY WAYS WE DO TO TRY TO DETECT THE KINDS OFTHINGS THAT ARE GOING ON. I DON'T REALLY HAVE A FEELING FOR THERELIABILITY OF TIPLINES AND HOW EFFECTIVE THEY REALLY ARE. MAYBETHAT'S A GOOD AREA OF RESEARCH, ESPECIALLY BECAUSE IT'S A MAJOR WAYOF DETECTION TODAY.CARSWELL WHITEHEAD: WELL, IN MY WORLD, THE TIPLINE IS VERY EFFECTIVE.AND HOW ARE STUDENTS ABLE TO SHARE INFORMATION WITH THE SCHOOL IFTHEY HAVE A CONCERN THAT THE TEST QUESTIONS THAT THEY SAW THAT133PARTICULAR DAY ARE TEST QUESTIONS THAT WERE REVIEWED WITH THEM THEDAY PRIOR?MELISSA FINCHER: THEY WOULD GO TO THE PRINCIPAL, TYPICALLY, IF THAT'STHE CASE. IT DEPENDS ON THE LOCAL STRUCTURE. AS WE HEARD EARLIERTODAY AND AS YOU ALL PROBABLY KNOW, UNFORTUNATELY SOME OF THEEDUCATORS WHO REPORTED INAPPROPRIATE PRACTICE WERE PERSECUTED. SO,IT'S MAKING SURE THAT THE PROTECTIONS ARE THERE SO THE INFORMATIONCAN BE FORTHCOMING.JACK BUCKLEY: ALL RIGHT, SOMETHING ELSE THAT CERTAINLY EMERGED FORME AS A COMMON THEME ACROSS YOUR COMMENTS WAS THE NEED TO NOT FOCUSON A SINGLE ANALYSIS. SO, I THINK YOU COULD CALL THIS SOME SORT OFDEFENSE IN DEPTH STRATEGY, SOMETHING WHERE WE NEED TO BE LOOKING ATA RANGE OF BOTH DIRECT AND INDIRECT DETECTION METHODS. GIVEN SORTOF REAL-LIFE SCARCE RESOURCE CONSTRAINTS, IF YOU'RE IMPLEMENTINGAN ASSESSMENT PROGRAM, SAY, AT THE DISTRICT LEVEL, AMONG THISDIFFERENT RANGE OF METHODS TO CHOOSE FROM, WHAT SHOULD BE THE FIRSTFEW THAT YOU WOULD ABSOLUTELY RECOMMEND IN A K-12 SORT OF STANDARDESEA-NCLB-TYPE TESTING ENVIRONMENT? WHAT ARE YOUR FIRST 3 GO-TOMETHODS?CARSWELL WHITEHEAD: I WILL LOOK AT THE ERASURES, IF WIDESPREAD, ANDALSO AN ANSWER PATTERN. LIKE I SAID, YOU CAN USE THOSE ANALYSES IN134CONJUNCTION. I HEARD SOMEONE MENTION EARLIER THAT IT MAY NOT BEUNUSUAL FOR A LARGE NUMBER OF ERASURES. YES, ON AN INDIVIDUALBASIS, YES. BUT IF YOU SEE HUNDREDS OF ANSWER SHEETS THAT COME INAND YOU SEE A LOT OF ERASURES--AND YOU WOULD HAVE TO HAVE ACRITERION. WHAT ARE YOU LOOKING FOR IN ERASURES? WHAT IS CONSIDEREDUNUSUAL ERASURES? BECAUSE YOU HAVE TO HAVE STANDARDS IN ORDER TOUSE. AND IF YOU SEE THAT YOU HAVE HUNDREDS OF ANSWER SHEETS WITHERASURES, THAT DEFINITELY WOULD BE A FLAG. NOW, AS FAR AS MYPROCESS, AS FAR AS INDIVIDUAL CASES OR GROUP CASES, I WOULD LOOK ATPATTERN OF RESPONSES TO CHECK FOR EITHER PRE-KNOWLEDGE OR TO CHECKFOR SOME FORM OF COPYING OR COMMUNICATION. SO, THOSE ARE THE ONESTHAT I WOULD LOOK AT. AND THEN IF YOU'RE LOOKING AT A COMPUTERBASED TEST, IF STUDENTS ARE TAKING A TEST ON A COMPUTER AND YOU'RESEEING THAT THEY'RE COMPLETING THE TEST UNUSUALLY FAST, THAT MIGHTBE AN INDICATOR THAT YOU WANT TO LOOK AT AS WELL.DAVID FOSTER: I DON'T THINK I CAN ANSWER THE QUESTION. MAYBE IT'S MYLACK OF EXPERIENCE IN THIS AREA, BUT EACH OF THESE HAS THEIRADVANTAGES AND THEIR DRAWBACKS. ERASURE ANALYSIS--AS THE WORLD ISGOING MORE TOWARD COMPUTERIZING EXAMS, IT'S A DATED METHODOLOGY.SO, ADVICE TO USE IT IS "USE IT FOR A WHILE. IT'S GONNA GO AWAY."AND REPLACE IT WITH OTHER STATISTICAL ANALYSES. I DON'T KNOW.AGAIN, EFFECTIVE SECURITY IS GOING TO BE ABLE TO CAPTURE--MAYBE YOU135COULD JUST USE EACH OF THEM ON A SMALLER SCALE, AND THAT MIGHT BEMORE EFFECTIVE. THAT'S PROBABLY THE WAY I WOULD ANSWER IT, I GUESS.BRIAN JACOB: I WAS JUST THINKING... THE THING ABOUT COST AND COSTBENEFIT. I THINK THERE'S PROBABLY VASTLY DIFFERENT COSTS FOR SOMEOF THESE STRATEGIES. I WOULD SUGGEST THAT ERASURE ANALYSIS AND SOMEOF THE STATISTICAL-BASED TEST SCORE ANALYSIS, I THINK, SHOULD BEVIRTUALLY COSTLESS ONCE THE SYSTEM IS PUT IN PLACE. I MEAN THERE'SA COMPUTER PROGRAM THAT SCANS THE TESTS THAT POPS OUT THE AVERAGEOR THE MEDIAN NUMBER OF WRONG TO RIGHT ERASURES. YOU NOW HAVE APROGRAM THAT CALCULATES THE AVERAGE TEST SCORE GAIN FROM ONE YEARTO THE NEXT. SO WE SHOULD BE DOING ALL OF THOSE THINGS THAT ARECHEAP ALL OF THE TIME. AND THEN, FOR THINGS THAT ARE ACTUALLYEXPENSIVE, LIKE MANUAL INSPECTION, INVESTIGATION OF SPECIFICIRREGULARITIES—THOSE THINGS, I THINK, WE WOULD HAVE TO RATION. ITHINK DAVID'S POINT ABOUT FOCUSING ON HIGH-RISK CASES IS A GOODCRITERIA OR LOOKING AT THE MEDICAL SCREENING MODEL. YOU DO THE LOWCOST THINGS FOR EVERYONE AND THEN YOU ONLY DO MORE IN-DEPTHINVESTIGATIONS WHEN THERE'S INDICATION OF IRREGULARITY EITHER FROMA TIP OR FROM ONE OF THESE LOW-COST SCREENS.JACK BUCKLEY: BEFORE I OPEN IT UP, JUST ONE MORE QUICK POINT, BACK TOONE OF THE THINGS YOU SAID AS A BEST PRACTICE, 'CAUSE IT'SSOMETHING THAT I'M NOT SURE THAT I'D HEARD ELSEWHERE. YOU DISCUSSED136ACTUALLY CONDITIONING. SO, IF YOU'RE PERFORMING SOME SORT OFSTATISTICAL FORENSIC ANALYSIS TO DO THAT ANALYSIS CONDITIONAL ONSCHOOL AND/OR STUDENT CHARACTERISTICS, CAN YOU EXPLAIN THERATIONALE FOR THAT AND WHY IF FOLKS AREN'T DOING THAT THAT THEYSHOULD BE, IN YOUR OPINION?BRIAN JACOB: I THINK THE RATIONALE GETS BACK TO THE UNDERLYINGCONCEPT BEHIND THE STATISTICAL ANALYSIS. YOU'RE LOOKING FOR ADEVIATION FROM WHAT IS EXPECTED OR NORMAL. AND THAT DEVIATION,YOU'RE TAKING AT LEAST AS AN INITIAL SUGGESTION OF IRREGULARITY,NOT NECESSARILY CHEATING, OF COURSE. AND...SO I GUESS THE QUESTIONIS WHAT IS EXPECTED. AND IF THERE ARE CERTAIN CHARACTERISTICS OFSTUDENTS OR SCHOOLS THAT ARE ASSOCIATED WITH ERASURES OR CERTAINTEST SCORE PATTERNS OR TEST SCORE GAINS, THEN YOU WANT TO BECOMPARING TO THE EXPECTED FOR THAT GROUP AND NOT NECESSARILY FORTHE EXPECTED FOR ALL GROUPS. I THINK IN MANY CASES, IN MOST CASES,IF THE FLAGGING CRITERIA IS SUFFICIENTLY HIGH--I THINK, CERTAINLY,3, 4, 5 STANDARD DEVIATIONS--AND THERE'S NOT A HUGE CORRELATIONBETWEEN STUDENT OR SCHOOL CHARACTERISTICS AND THESE MEASURES, INPRACTICE, IT PROBABLY WILL NOT HAVE A LARGE IMPACT IN MANY CASES.BUT I THINK IT'S CERTAINLY EASY ENOUGH TO DO AND CONCEPTUALLYSHOULD BE A COHERENT THING TO DO.137JACK BUCKLEY: TO OUR OTHER PANELISTS AND, OF COURSE, THE AUDIENCE.JOHN.JOHN FREMER: I WANT TO PUT OUT A PERCEPTION AND SEE IF THERE'SAGREEMENT OR DISAGREEMENT ON IT. MY PERCEPTION IS THAT THERE IS NOTA DISAGREEMENT ABOUT WHETHER YOU SHOULD DO MULTIPLE FORENSICSANALYSES FIRST DAY ASSESSMENT. I THINK THAT IS NOW THE STANDARD,THE BEST PRACTICE. THE ISSUE IS HOW TO GET TO THAT, GIVEN THAT YOUHAVE THESE ONGOING PROGRAMS, YOU HAVE THESE CONTRACTS WITHVENDORS, YOU HAVE THE CONSORTIUM TRAINS COMING DOWN THE TRACK. BUTFROM MY CONVERSATIONS WITH PEOPLE, FROM THE THINGS I READ, FROM THECONFERENCES I GO TO, I VIEW THAT AS ALREADY SETTLED. YOU NEED TO BEDOING THAT AND IF YOU'RE NOT DOING THAT, YOU PROBABLY ALREADY KNOWIT AND YOU HAVE IT IN YOUR HEAD OR YOU'RE HAVING MEETINGS OR PLANSOF HOW TO GET TO THAT POINT.BRIAN JACOB: MAYBE I COULD FOLLOW UP TO THAT AND DO A QUESTION. FORSTATE OR DISTRICT CONTRAST WITH TESTING COMPANIES AND VENDORS, HOWCOMMON ARE THE PROVISION OF ERASURE STATISTICS TO THE DISTRICT ORSTATE? I WOULD THINK, GIVEN THE EASE, THAT THAT SHOULD JUST HAVEALWAYS BEEN AN AUTOMATIC SOMETHING THAT IS INCLUDED EVERY TIME JUSTAS A MATTER OF COURSE. IS THAT THE CASE, OR IS IT VERY UNUSUAL ORADDITIONALLY EXPENSIVE FOR SOME REASON FOR A STATE TO GET ERASURE138STATISTICS ALONG WITH THE OTHER TEST STATISTICS IT GETS BACK FROMTHE VENDOR?STEVE FERRARA: I CURRENTLY WORK FOR PEARSON, AND I WORKED FOR OTHERTESTING COMPANIES. WE ALL OFFER A SUITE OF FORENSIC ANALYSES. ANDTHE QUESTION IS WHETHER THE RFP CALLS FOR IT OR WHETHER THE STATEASKS FOR IT IN AN ADDENDUM. SO THAT'S MY RESPONSE TO THAT. I'DACTUALLY LIKE TORAISE ANOTHER POINT, UNLESS YOUWANT TO CONTINUETHIS DISCUSSION.JACK BUCKLEY: PLEASE.STEVE FERRARA: ALL RIGHT, SO THERE'S BEEN SOME DISCUSSION OF ROBUSTSTATISTICS. AND JACK, I THINK YOU ASKED MELISSA A QUESTION THAT LEDHER TO SAY GEORGIA USES 3 STANDARD DEVIATIONS IN THE WRONG TO RIGHTANALYSIS. OTHER STATES USE 4. THAT'S CONSISTENT WITH WHAT I KNOW.AND I THINK IT BECOMES A MATTER OF POLICY AND PRACTICALITY INMAKING DECISIONS ABOUT THE STANDARDS FOR FORENSIC ANALYSES ANDOTHER DECISION-MAKING IN THE PROCESS. AND SOME PRACTICAL MATTERSCONCERN THINGS LIKE WHETHER YOU USE 3 OR 4 STANDARD DEVIATIONS-YOU CAN MAKE A DECISION LIKE THAT USING HISTORICAL DATA, DEPENDINGON THE FALSE POSITIVE AND NEGATIVE RATES THAT YOU GET IN HISTORICALDATA. ANOTHER CONSIDERATION IS HOW MANY INVESTIGATIONS YOU CANAFFORD TO CONDUCT. THEY'RE EXPENSIVE. THEY'RE HARD TO DO. THEY'RE139POLITICALLY SENSITIVE, AND THEY GET IN THE PAPERS. SO, IF YOU USE 2OR 3 STANDARD DEVIATIONS, YOU'RE GONNA FLAG A LOT OF CLASSROOMS, ALOT OF TESTING GROUPS. YOU CAN'T NOT FOLLOW UP ON ANY OF THOSE. YOUHAVE TO FOLLOW UP ON EVERY ONE. IF YOU USE 4 STANDARD DEVIATIONS,YOU FLAG CONSIDERABLY SMALLER NUMBERS OF TESTING GROUPS. IT GIVESYOU MORE RESOURCES TO FOCUS ON THE ONES THAT IN SUBSEQUENTINVESTIGATION YOU NEED TO GO INTO THE SCHOOLS AND DO MORE. AND YOUALSO HAVE THE BENEFIT OF BEING ABLE TO EXPLAIN "WE'RE USING 4STANDARD DEVIATIONS. THIS IS A REALLY STRICT CRITERION. IT FLAGSVERY FEW CLASSROOMS." IT GIVES YOU SOME POLITICAL CAPITAL TO MAKEYOUR JOB EASIER OF DETECTING AND INVESTIGATING.JACK BUCKLEY: ANY COMMENT FROM THE PANEL?AUDIENCEMEMBER:GOOD AFTERNOON. MY NAME IS DR. ... I'M ON THE D.C.STATE BOARD OF EDUCATION. MY CONCERN IS THE YOUNG PEOPLE THAT--[MICROPHONE CUTS OUT]...MY CONCERN IS THAT-- I'D LIKE TO STAND--IS THE TESTING. THE KIDSARE ACTUALLY BEING DAMAGED BECAUSE OF THE FACT, AS ADULTS, WE HAVEA MORAL OBLIGATION TO MAKE SURE THEY HAVE THE BEST QUALITYEDUCATION AND MAKE SURE THE SERVICE THAT WE GIVE THEM ARE THE BES140SERVICES THAT ARE GIVEN TO THEM. WE'RE DAMAGING OUR KIDS BECAUSETHIS HAS BEEN GOING ON FOR YEARS. I'M IN THE SCHOOL SYSTEM FOR OVER15 YEARS, ALONG WITH ANNETTE. AND ONCE YOU GIVE THESE TESTINGS,WHAT'S IN PLACE FO THEM TO HELP THEM TO SAY THAT THEY'RE DOING THERIGHT THING? BECAUSE THEY'RE LISTENING TO US, AND THEY HAVE THEIRTRUST IN US TO SAY THAT WE'RE GIVING THEM THE BEST TESTING—CURRICULUM THAT WE'RE ASSIGNED TO GIVE THEM. WHAT'S IN PLACE TOHELP THEM? BECAUSE THEY'RE MENTALLY BEING DISTURBED AND ALSOTHEY'VE BEEN OUTCAST BY THEIR FRIENDS BY CALLING THEM "CHEATERS"AND STUFF. IT'S ALMOST THE SAME AS BEING BULLIED. SO WE NEED TOHAVE SOMETHING IN PLACE TO MAKE SURE THAT THEY'RE DOING THE RIGHTTHING, BECAUSE THEY PUT THEIR TRUST IN US TO SAY THAT WE'RE DOINGTHE RIGHT THING. SO, WHAT'S IN PLACE? I KNOW DAVID TALKED ABOUTGIVING ANOTHER TEST WHEN THOSE DIDN'T DO RIGHT TO SAY WHATEVER.THAT'S DEVASTATING WHEN YOU HAVE TO GO BACK AND FORTH AGAIN TO SEEHOW THAT TEST IS BEING DONE WHEN YO THINK YOU DID A GREAT JOB ANDYOU GOTTA GO BACK AND TAKE THE TEST ALL OVER AGAIN. BECAUSE TO ME,IF THE PACKAGE WAS DONE IN THE CORRECT WAY IN THE BEGINNING, COMINGOUT THE TESTING, WE WOULDN'T HAVE THIS ISSUE. A LOT OF TIMES WHENYOU GET--JUST LIKE YOU TAKE YOUR DRIVERS' LICENSE. FIRST TIME YOUMIGHT FAIL, YOU HAVE TO GO BACK AND TAKE THE TEST AGAIN. AND THESAME THING WHEN YOU GET INSTRUCTIONS IN GIFTS OR CARS ANDAIRPLANES. IF THOSE EXAMPLES IN THERE IS NOT DONE RIGHT, THEN WHATHAPPEN? YOU BE PUTTING ALL THE GIDGITS TOGETHER AND YOU STILL FIND141SOME DEFAULT OR OTHER INFORMATION THAT'S NOT CORRECT. WE NEED TOHAVE SOMETHING IN PLACE THAT OUR KIDS DON'T HAVE TO BE GOINGTHROUGH THIS AND ESPECIALLY THE PARENTS, 'CAUSE THIS STAYS WITH YOUFOR A LIFETIME. SO, WHAT'S IN PLACE-- WHAT I WANT TO KNOW FROMSTATE BOARD OF EDUCATION AND FROM PARENTS THAT I'M GETTINGFEEDBACK--THAT WHAT ARE WE GONNA DO TO MAKE THIS BETTER? THANK YOU.JACK BUCKLEY: I THINK THERE'S A LOT IN THERE. IF I COULD TAKE ONEPIECE OUT, IT WOULD BE REALLY THE QUESTION ABOUT FOLLOW-UP. SO,LET'S SAY WE'VE SET A FAIRLY RIGOROUS THRESHOLD FOR DETECTINGIRREGULARITY. ONCE WE'VE DETECTED IT--WE'RE GONNA TALK ABOUTINVESTIGATION ON THE NEXT PANEL. BUT IN TERMS OF WITHIN THENARROWER CONTEXT, WHAT DO THE DETECTION PEOPLE DO NEXT AND HOWOFTEN DO THEY DO IT? WHERE DO WE SEND THIS TO AND WHERE DO WE GOFROM THERE IN THE IMMEDIATE NEXT STEPS?DAVID FOSTER: I'M GONNA MAKE A VERY GENERAL COMMENT, AND I DON'T KNOWHOW HELPFUL IT WILL BE, BUT I THINK IN MANY OF THESE AREAS WE'RESTILL IN THE INFANCY. OUR PROCTORING METHODS ARE BROUGHT INTOQUESTION. OUR STATISTICAL METHODS ARE FAIRLY NEW, AND THERE AREN'TAS MANY USED AS WE WOULD LIKE. TIPLINES. WE DON'T REALLY KNOW HOWEFFECTIVE THEY ARE. WHAT WE NEED TO DO FROM THE DETECTION SIDE ISKEEP REFINING AND IMPROVING THEM. THERE'S A LOT OF ROOM FORIMPROVEMENT, AND THE PROBLEM IS, SAY WITH PROCTORING, IS WE GET142SET. TEACHERS ARE GREAT AND THEY'RE ALWAYS GONNA PROCTOR WELL ANDTHAT'S A STANDARD AND WE CAN FORGET ABOUT IT NOW FOR 50 YEARS. ANDTHAT'S REALLY NOT THE CASE. WE NEED TO EVALUATE EACH OF THE THINGSTHAT WE DO SO THAT WE CAN MAKE SURE WE'RE DETECTING PROPERLY WITHFEW FALSE POSITIVES AND FALSE NEGATIVES. I THINK THERE'S ROOM FORIMPROVEMENT. THAT'S A VERY GENERAL ANSWER AND PROBABLY DOESN'TSPECIFICALLY ANSWER ANY INDIVIDUAL CONCERN BUT--MELISSA FINCHER: I THINK IT'S SOMETHING THAT AT THE STATE DEPARTMENTWE DON'T TAKE LIGHTLY. IT IS VERY DISHEARTENING WHEN YOU HAVE TOINVALIDATE A STUDENT'S SCORE THROUGH NO CAUSE OF THE STUDENT HIM ORHERSELF, THROUGH ACTIONS THAT IMPROPER ADMINISTRATION OR SOMEOTHER FORCE HAS OCCURRED. CERTAINLY, AS A STATE, WE'RE WORKING TOIDENTIFY THOSE STUDENTS WHO DID NOT HAVE AN ACCURATEADMINISTRATION AND MAKING SURE THAT THEY RECEIVE SUPPORT SERVICESTHAT THEY NEED SO THAT THEY CAN BE SUCCESSFUL, BUT IT'S VERYDIFFICULT.JACK BUCKLEY: SCOTT.SCOTT NORTON: I HAVE A QUICK QUESTION, REALLY. I THINK IT'S MOSTLYFOR MELISSA, BUT ANYBODY COULD RESPOND. IF YOU WOULD, SAY A WORD ORTWO ABOUT THE PROS AND CONS OF HAVING THIS OTHER AGENCY THAT YOUDEAL WITH THAT TAKES A LOT OF THE BURDEN FOR THE INVESTIGATION. I143DIDN'T QUITE GET ALL OF IT, BUT THAT'S PROBABLY ATYPICAL COMPAREDTO MOST? JUST FOR OUR INFORMATION, HOW DOES THAT GO?MELISSA FINCHER: SURE. THE GOVERNOR'S OFFICE OF STUDENT ACHIEVEMENTIS A SEPARATE AGENCY FROM THE GEORGIA DEPARTMENT OF EDUCATION, ANDTHEY ARE STATUTORILY RESPONSIBLE FOR EDUCATIONAL AUDITING, AND SOTHEY ARE THE ONES THAT EMBARKED ON THIS PATH IN THE STATE OFGEORGIA. AND IN MANY WAYS, IT HAS WORKED VERY NICELY FOR US,BECAUSE MY DIVISION CAN CONTINUE TO FOCUS ON THE DEVELOPMENT OF THETEST, ON THE ADMINISTRATION SUPPORTING DISTRICTS. WE WORK VERYCOLLABORATIVELY WITH THEM IN THAT THEY CONTRACT WITH OUR VENDOR ANDOFTEN THROUGH OUR OWN CONTRACT, BUT THE DATA GO DIRECTLY TO THEMAND NOT TO MY OFFICE. WE REVIEW THE FINDINGS OF THE INVESTIGATIONSAND BUMP THAT UP AGAINST OUR PROTOCOLS. I WORK WITH THE OFFICE OFSTUDENT ACHIEVEMENT FOR ANY FINDINGS THAT THEY HAVE OCCURRED, ANDTHAT'S ONE OF THE TIPLINES THAT I DIDN'T TALK ABOUT IS THAT OFFICEALSO RECEIVES INQUIRIES AND TIPS FROM THE FIELD THAT THEY FOLLOW ONTHAT I'M NOT PRIVY TO. BUT WE'RE CONSTANTLY--IT'S A RECURSIVEPROCESS. IN LOOKING AT OUR SECURITY PROTOCOLS AND, UM...MONITORING THEM AND REVISING THEM, WHAT WE HAVE FOUND AS A RESULTOF THE LATEST INVESTIGATION IS THAT THEY WERE USING SOME OF OURPROTOCOLS AS AN EXCUSE TO DO SOME OF THE THINGS THEY WERE DOING. SOWE HAD TO ADJUST BASED ON THAT. BUT IT'S MADE MY LIFE, QUITEFRANKLY, SCOTT, A LOT EASIER HAVING THAT OUTSIDE AGENCY LEAD THAT144WORK BECAUSE EVEN WITH IT, IT'S BECOME A SECOND FULL-TIME JOB FORME.AUDIENCEMEMBER:LET ME JUST SAY THIS. MY NAME IS STEVE JACKSON. I'MTHE PRINCIPAL OF DUNBAR HIGH SCHOOL, AND I WAS ALSO THE PRINCIPALOF TWO DIFFERENT HIGH SCHOOLS IN NEW YORK STATE, AND I'VE BEENDOING THIS WORK FOR OVER 25 YEARS. NUMBER ONE, I THINK IT'S SADTHAT WE'RE HAVING THIS TYPE OF CONVERSATION INVOLVING INTEGRITYSIMPLY BECAUSE AS AN EDUCATOR, ALL OF US SHOULD HAVE SOME LEVEL OFINTEGRITY, ESPECIALLY AS IT PERTAINS TO TESTING. WE TALKED A LITTLEBIT ABOUT TRANSPARENCY A LITTLE EARLIER. I THINK IT'S IMPORTANT TOHAVE A LEVEL OF TRANSPARENCY, BUT MORE IMPORTANTLY I DO THINK THATADMINISTRATORS, DISTRICTS, AND OTHERS NEED TO BE TRAINEDSPECIFICALLY ON TESTING INTEGRITY, BECAUSE IT'S CLEAR TO ME THATTHROUGHOUT THE COUNTRY, WE'RE HAVING A LOT OF PROBLEMS IN DIFFERENTDISTRICTS AND DIFFERENT SCHOOLS IN TERMS OF TESTING INTEGRITY ASWELL AS THE RESPONSIBILITY OF EDUCATORS. SO, FOR ME, THAT'SEXTREMELY IMPORTANT, BUT EVEN AS WE LOOK AT THE HIGH-STAKE TESTINGAND AS WE MOVE TOWARD THE COMMON CORE, THERE ARE TEACHERS NOW THATARE BEING HELD RESPONSIBLE FOR WHETHER OR NOT THEIR PARTICULARSTUDENTS PASS OR FAIL. AND AS WE MOVE AROUND THE COUNTRY ANDTHEY'RE HELD RESPONSIBLE FOR WHETHER OR NOT THEIR STUDENTS PASS ORTHEIR STUDENTS FAIL, THEN SOME OF THESE PEOPLE WHO MAY NOT HAVEINTEGRITY WILL CHEAT. SO I THINK IT'S IMPORTANT FOR US TO BEGIN TO145HAVE THESE CONVERSATIONS IN DISTRICTS THROUGHOUT THE COUNTRY SOTHAT PEOPLE ARE REAL CLEAR NOT ONLY WHAT THEIR ROLE ANDRESPONSIBILITY IS BUT ALSO WHAT SOME OF THE CONSEQUENCES MAY BE FORCHEATING.JACK BUCKLEY: AND I THINK THAT SUMMARIZES MUCH OF THE INTENT OF THISENTIRE DAY, SO WOULD AGREE. YES.AUDIENCEMEMBER:EXCUSE ME.[MICROPHONE CRACKLING]I'D LIKE, UM--[MICROPHONE FEEDBACK]I JUST WOULD LIKE A REACTION FROM THE PANEL MEMBERS ABOUT THISIDEA. THIS IS BASED ON THE COMMENT THAT WAS MADE BY DR. FOSTER THATTHERE ARE HUNDREDS, MAYBE THOUSANDS OF WAYS THAT THERE'S CHEATINGAND STEALING AND ALSO DR. CIZEK'S COMMENT ON SHAREDRESPONSIBILITY. ISN'T THE TIME THAT THERE IS SOME KIND OF EITHER ALIST OR A DATABASE OR SOME KIND OF CLEARINGHOUSE THAT IF THERE'SSOME NEW WAY OF CHEATING OR STEALING THAT EVERYONE'S MADE AWARE OFIT AND MAYBE WITH A CORRESPONDING WAY OF DEALING WITH IT? WHETHER146IT'S A NEW WAY OF PROCTORING, A NEW STATISTICAL INDEX, OR MAYBEEVEN NEW COMPUTER PROGRAM THAT MIGHT EVEN BE SHARED AS AN OPENSOURCE. AND YES, AS MEMBERS OF NCME, WE DO PRESENT OUR RESEARCHABOUT THESE THINGS ONCE A YEAR, BUT SOMETIMES THE NEED IS MOREIMMEDIATE. AND I'M THINKING ALONG THE SAME LINE AS FOR IN THE I.T.INDUSTRY WHEN THERE ARE NEW VIRUSES. NORTON KEEPS TRACK OF IT, ANDTHEN THEY HAVE A SOLUTION AND THEY UPDATE YOUR COMPUTERS WITH IT.SO I JUST LIKE A REACTION FROM THE PANEL MEMBERS. THANK YOU.CARSWELL WHITEHEAD: AS FAR AS TEST CENTER, I CAN SPEAK IN REGARDS TOTEST CENTER SUPERVISORS BECAUSE THAT'S WHERE I'M PRIMARILY INCONTACT WITH. WE DO SHARE THAT TYPE OF INFORMATION WITH THESUPERVISORS DURING TRAINING SESSIONS SO THEY CAN BE VIGILANTDURING THE TEST AND BE ABLE TO DETECT THOSE TYPES OFIRREGULARITIES. SO IT'S AN ONGOING PROCESS WHERE THAT INFORMATIONIS SHARED EITHER VIA ONLINE TRAINING OR IN PERSON WE SHARE THATINFORMATION.DAVID FOSTER: YOU KNOW, I AGREE THAT IT'S NOT VERY SYSTEMATIC. WHENWE STARTED CAVEON 10 YEARS AGO, WE STARTED AT CHEATING IN THE NEWS,A NEWSLETTER, AND INTENDED TO SHOWCASE EVERY COUPLE OF WEEKS 5 OR 6INCIDENTS THAT CAME OUT OF THE NEWS. EVERYBODY ENJOYED READING IT.YOU DO LEARN SOME THINGS, BUT IT ISN'T SYSTEMATIC LIKE YOU'RESUGGESTING. IT'S NOT A DATABASE, AND I THINK SOMETHING LIKE THAT IS147NEEDED. HOW WE GATHER THAT INFORMATION AND PRESENT IT IN AREASONABLE WAY, I'M NOT REALLY SURE. BUT I CERTAINLY AGREE AS TOTHE NEED FOR THAT.JACK BUCKLEY: I THINK WE HAVE ONE QUESTION FROM THE WEB AUDIENCE.JAMES ELIAS: WE DO, ACTUALLY. IT'S FROM JIM WOLLACK AT THEUNIVERSITY OF WISCONSIN. JIM WANTS TO KNOW ABOUT DEVELOPING ACRITERION BASED ON LIKELIHOOD OF COMMITTING A FALSE POSITIVE ANDTHEN USING STATISTICAL CORRECTIONS LIKE BONFERRONI TO ADJUST FORTHE TOTAL NUMBER OF ANALYSES BEING PERFORMED. HE WAS OFFERING THATAS AN ALTERNATIVE TO USING 3 VERSUS 4 STANDARD DEVIATIONS.JACK BUCKLEY: I'M HAPPY TO SPEAK TO THAT, BUT IF ANYONE ELSE ON THEPANEL WOULD RATHER... I THINK THIS IS SOMETHING WE'VE CERTAINLYSEEN IN MANY OF THE RFI COMMENTS THE DEPARTMENT HAS ALREADYRECEIVED. AT LEAST IN SEVERAL PLACES, THERE'S A QUESTION ABOUTPEOPLE UNDERSTANDING THAT IF YOU'RE DATA DREDGING OR NOT LOOKING TOTEST A PARTICULAR HYPOTHESES THAT YOU NEED TO CONSIDER VERYCAREFULLY THE FALSE POSITIVE AND FALSE NEGATIVE RATES. AND BRIAN,YOU TOUCHED ON THIS BEFORE. YOU CAN DO THAT BY SETTING YOURTHRESHOLDS VERY FAR UP, BUT YOU CAN ALSO DO THAT BY CONSIDERING THEENTIRE SORT OF FAMILY-WISE ERROR RATE OR WHATEVER IT IS ACROSS YOURDIFFERENT ANALYSES. I THINK YOU CAN SHOW THERE'S A VARIETY OF148METHODS THERE, AND SOME OF THEM ARE EQUIVALENT STATISTICALLY, ANDSOME OF THEM SORT OF SERVE DIFFERENT PURPOSES. BUT I DON'T THINKTHAT THERE'S ANY CLEAR STANDARDS YET THAT ANYONE HAS AGREED ON,WHICH IS SOMETHING I'M CERTAINLY HEARING TODAY. I DON'T KNOW IFANYONE ELSE HAS A COMMENT ON THAT.QUICK QUESTION? I GOT TIME FOR ONE MORE, THEN WE'VE GOTTA WRAP UP.[MICROPHONE FEEDBACK]AUDIENCEMEMBER:I'M CATE SWINBURN. I'M THE CHIEF OF DATA ANDACCOUNTABILITY IN D.C. PUBLIC SCHOOLS. NOT SURE IF MY MICROPHONE[INDISTINCT].I WONDER IF ANY OF THE ACADEMICS WHO'VE DONE THE STATISTICALANALYSIS HAVE ALSO LOOKED AT TEACHER EVALUATION SCORES OR THIRDPARTY EVALUATIONS, SO AS WE ARE STEPPING INTO THIS IN D.C. PUBLICSCHOOLS THIS YEAR FOR SOME OF OUR ANALYSES, I'M LOOKING AT THOSEFLAGGED TEACHERS TO LOOK AT THEIR OBSERVATION SCORES ON OUREVALUATION, WHICH FOR THOSE OF YOU WHO DON'T KNOW, FOR D.C. PUBLICSCHOOLS, WE'VE GOT—TWO OF THE TEACHERS' OBSERVATIONS A YEAR AREDONE BY A THIRD-PARTY EVALUATOR, SO IT'S NOT JUST A PRINCIPAL'SEVALUATION SCORE. ADDITIONALLY, OUR TEACHERS ARE EVALUATED ON COREPROFESSIONALISM AS PART OF THEIR ANNUAL SCORE. SO, CERTAINLY, FOR149THIS YEAR, WE ARE INVESTIGATING EVERYONE. BUT THERE'S A PART OF METHAT WONDERS IF OVER MULTIPLE YEARS IF WE CAN BE CORRELATING SORTOF DETERMINATIONS OF IMPROPRIETY ALONG WITH VERY HIGH OBSERVATIONSCORES OR IF THERE'S THE ABILITY TO SAY IF SOMEBODY'S NOT HITTINGFOR WRONG TO RIGHT BUT THEY'RE HITTING FOR BEING A HIGH FLYER ONGROWTH, BUT THEY ARE ALSO VERY HIGHLY RATED FOR OBSERVATIONS ANDCORE PROFESSIONALISM, COULD WE EXCLUDE THEM? NOT THAT WE ARE NOW,BUT WONDERING IF ANYBODY'S ACTUALLY DONE ANY LOOK AT CORRELATINGTEACHER PERFORMANCE EVALUATIONS TO IMPROPRIETIES AS PART OF MAKINGTHIS MORE ROBUST.JACK BUCKLEY: MY GUESS IS NO, BUT IF ANYONE ELSE DOES KNOW ANEXAMPLE... I THINK THIS FALLS IN THE VERY BROAD CATEGORY OF USINGALL THE INFORMATION YOU CAN, AND I THINK THAT'S A GOOD IDEA.AUDREY AMREIN-BEARDSLEY: I CAN ANSWER THAT A LITTLE BIT. I KNOW THATTHE CORRELATION IN THE OFFICES THAT ARE JUST STARTING TO BECONDUCTED, PARTICULARLY ON WHAT YOU ALL ARE DOING IN OTHER STATESON THE GROWTH AND THE VALUE-ADDED IS ONLY ON THE GROWTH ASCORRELATED WITH OBSERVATIONS RIGHT NOW. I'M NOT AWARE OF ANYTHINGTHAT'S LOOKED AT IMPROPRIETARIES. IT'S JUST WE'RE AT THE FIRSTLEVEL ANALYSES AT THIS POINT IN RUNNING THOSE CORRELATIONS.150DAVID FOSTER: AND I'LL ANSWER IT IN A DIFFERENT WAY. I AGREE THATTHERE'S A LOT MORE DATA THAT SHOULD BE GATHERED IN ON THESEANALYSES. A RECENT ONE I'M AWARE OF--A PROBLEM WAS DISCOVEREDTHROUGH A FORENSIC ANALYSIS SIMPLY BECAUSE TWO TEST TAKERS SHAREDTHE SAME E-MAIL ADDRESS. IT'S REALLY A VERY SPECIFIC PIECE OFINFORMATION, BUT IT LED TO AN INVESTIGATION AND DISCOVERY THATTHERE WAS A PROXY TEST TAKING RING GOING ON, AND SO I BRING THAT UPJUST TO ILLUSTRATE THAT THERE'S A LOT OF DATA THAT CAN BEASSOCIATED WITH THE TESTS THAT WE GENERALLY DON'T USE. AND THAT'SAN EXAMPLE OF ONE THAT HAPPENED TO CAUSE A HIT AT THAT TIME, BUTCERTAINLY WE SHOULD DO MORE OF THAT WORK.JACK BUCKLEY: ALL RIGHT, SO, TO WRAP UP, CERTAINLY WE'VE HEARD FROMTHIS VERY INTERESTING PANEL THAT WE NEED TO USE A RANGE OFSTATISTICAL AND NON-STATISTICAL METHODS FOR DETECTION ANDVERIFICATION. CERTAINLY NO ONE METHOD IS DEFINITIVE, AND WE NEED TODETECT EARLY AND USE EARLY DETECTION METHODS TO EVALUATE SECURITYPROCEDURES AS WELL. THIS IS NOT JUST A POST-OP STATISTICAL PROCESS.CERTAINLY, I THINK SOMETHING WE'VE HEARD IN THE WRITTEN COMMENTSAND ALSO TODAY IS SOMETHING YOU'LL HEAR ALL THE TIME IN SECURITYTHAT IT'S A PROCESS, NOT A PRODUCT. THERE'S ALWAYS NEW METHODS OFCHEATING DEVELOPED. THERE ARE ALWAYS SORT OF A CONSTANT RACEBETWEEN PEOPLE TRYING TO GET AWAY WITH SOMETHING AND PEOPLE TRYINGTO DETECT THEM, AND THIS IS CERTAINLY NO EXCEPTION. I THINK THE151OTHER PIECE THAT I'M HEARING THAT'S VERY INTERESTING COMING OUT OFTHIS PANEL IS, AGAIN, THE NEED FOR TRANSPARENCY AND CLEAR CRITERIA.WE DIDN'T GET A CHANCE TO TALK TO MAYBE ABOUT POLICY-WISE--STEVE,YOU WERE STARTING TO GET THERE-- BUT WHO SHOULD SET THOSECRITERIAS. IS THAT A STATE BOARD? IS THERE A TECHNICAL ADVISORYPANEL? AND THAT'S SOMETHING I'D LIKE TO HEAR MORE ABOUT IF WE HAVETHE TIME, BUT CLEARLY WE NEED TO COMMUNICATE THESE THRESHOLDS TOTHE AUDIENCE. SO, ON THAT NOTE, I'D LIKE TO THANK THIS PANEL FORANOTHER VERY INTERESTING SESSION.[APPLAUSE]WE'RE NOW GONNA SHIFT TO LUNCH ON YOUR OWN. SO, YOU GOT AN HOUR TOEAT LUNCH, AND WE WILL START PROMPTLY AT 1:30, BECAUSE WE HAVE TWOMORE VERY GOOD SESSIONS TO GET THROUGH THIS AFTERNOON.THANK YOU VERY MUCH.152PANEL III: RESPONSE AND INVESTIGATION OF ALLEGED AND/OR ACTUALIRREGULARITIES IN ACADEMIC TESTINGJACK BUCKLEY: WELCOME BACK, EVERYONE.OUR THIRD PANEL OF THE DAY FOCUSES NOW ON RESPONSE TO ANDINVESTIGATION OF ALLEGED AND/OR ACTUAL IRREGULARITIES IN ACADEMICTESTING.WE'LL START BY HEARING FROM BOB WILSON OF WILSON, MORTON, AND DOWNSAND A FORMER DISTRICT ATTORNEY FOR DEKALB COUNTY, GEORGIA, OFCOURSE. BOB IS ONE OF THE SPECIAL INVESTIGATORS APPOINTED BY FORMERGOVERNOR SONNY PERDUE AND GEORGIA GOVERNOR NATHAN DEAL TOINVESTIGATE CHEATING ON THE CRCT IN THE ATLANTA PUBLIC SCHOOLSAND DOUGHERTY COUNTY SCHOOL SYSTEM.NEXT, WE'LL BE JOINED BY TISHA EDWARDS, WHO'S SPEAKING ON BEHALF OFDR. ANDRES ALONSO, CEO OF BALTIMORE CITY PUBLIC SCHOOLS.TISHA IS THE CHIEF OF STAFF OF BALTIMORE CITY, WHERE SHECOORDINATES THE ACTIVITIES OF THE SYSTEM OFFICERS FOR THEEFFICIENT OPERATION OF ALL SCHOOLS AND OFFICES. SHE'S ALSO THEFOUNDER OF BALTIMORE FREEDOM ACADEMY, A CHARTER HIGH SCHOOL IN THECITY.153NEXT, WE'LL HEAR FROM STEVE FERRARA, WHO'S VICE PRESIDENT AND CODIRECTOR OF THE CENTER FOR PERFORMANCE ASSESSMENT AT PEARSON.PRIOR TO JOINING PEARSON, STEVE WAS A PRINCIPAL RESEARCH SCIENTISTAT CTB/MCGRAW-HILL, MANAGING RESEARCH DIRECTOR IN THE ASSESSMENTPROGRAM AT THE AMERICAN INSTITUTES FOR RESEARCH, AND STATEASSESSMENT DIRECTOR IN MARYLAND.AND FINALLY, LOU FABRIZIO IS THE DIRECTOR OF THE DIVISION OF DATA,RESEARCH, AND FEDERAL POLICY AT THE NORTH CAROLINA DEPARTMENT OFPUBLIC INSTRUCTION. HE CONTINUES TO SERVE AS THE STATE'S FEDERALLIAISON WITH THE U.S. DEPARTMENT OF EDUCATION AND ALSO SERVES ASTHE SENIOR ADVISOR TO STAFF IN THE ACCOUNTABILITY SERVICESDIVISION REGARDING THE STATE'S ABCS ACCOUNTABILITY PROGRAM, THESTATEWIDE TESTING PROGRAM, AND THE STATE'S ACCOUNTABILITY ANDASSESSMENT PLANS.SO, BOB?BOB WILSON: GOOD AFTERNOON.IT'S ALWAYS GREAT TO BE FIRST AFTER LUNCH WHEN EVERYBODY WANTS TOTAKE A NAP. THE ATLANTA INVESTIGATION, WHICH I WAS HONORED ALONG154WITH A GENTLEMAN NAMED RICHARD HYDE AND MIKE BOWERS TO HEAD UP, WASA MASSIVE INVESTIGATION, AND UNFORTUNATELY, IT REVEALED ANUGLY AND DISTURBING TRUTH.TO GIVE YOU SOME CONCEPT OF THE SIZE, THE INVESTIGATIVE TEAMINCLUDED OVER 60 STATE AGENTS, ATTORNEYS, PARALEGALS. IN THE END,WE ACTUALLY USED OVER 100 DIFFERENT PEOPLE IN THAT INVESTIGATION.THAT INVESTIGATION, BY ANY STATISTICAL ASSESSMENT, WOULD NORMALLYHAVE TAKEN 14 TO 18 MONTHS TO 18 MONTHS, AND WE PRESSED IT INTO 10MONTHS. SO IT WAS VERY EXHAUSTIVE FOR EVERYONE INVOLVED IN IT.WE INTERVIEWED APPROXIMATELY 2,000 PEOPLE AND A TOTAL OF 2,200INTERVIEWS. THAT'S A LOT OF INTERVIEWS.WE WENT THROUGH OVER 800,000 DOCUMENTS.THERE WERE 56 SCHOOLS THAT WERE INVOLVED IN THE INVESTIGATION,AND WE FOUND CHEATING AT 44 OF THEM. IN 30 OF THE SCHOOLS, 82PRINCIPALS AND TEACHERS CONFESSED.OVER 180 EDUCATORS WERE INVOLVED ULTIMATELY IN THE END BY NAME.IT'S IMPORTANT TO NOTE THAT IN OUR REPORT, BASED ON ALL OF THEEVIDENCE THAT WE FOUND, WE ESTIMATED THAT THE NUMBER OF TEACHERS155AND ADMINISTRATORS INVOLVED WAS SOMETHING LIKE TWO TO THREE TIMESGREATER THAN THOSE WE COULD NAME.THERE WERE 38 PRINCIPALS INVOLVED. IF YOU'RE IN THE SYSTEM OUTTHERE IN AMERICA, YOU DON'T WANT TO BE WHERE ATLANTA WAS.THE HEADLINES WERE AWFUL. YOU DON'T WANT TO READ HEADLINES LIKE,"WHISTLE BLOWERS GET PUNISHED," "A.P.S. SUPPRESSED SCANDAL,""ATLANTA SCHOOLS SOFT ON CHEATERS," "UNETHICAL BEHAVIOR ACROSSEVERY LEVEL"--THAT'S A HEADLINE--WITH PICTURES OF TOPADMINISTRATORS.IT HURTS THE SYSTEM. IT HURTS THE IMAGE OF EDUCATION IN THISCOUNTRY.WHEN YOU LOOK AT THE FINAL POINT ON THIS PARTICULAR SLIDE, YOU SEETHAT IT'S TIME-CONSUMING, AND IT'S CERTAINLY OFF MISSION. ITABSOLUTELY WILL DIVERT YOUR SYSTEM FROM WHERE IT OUGHT TO BE TOWHERE IT SHOULD NOT BE. IN THE ATLANTA SYSTEM...THEY WERE NOT HAPPYTO WORK WITH NO CHILD LEFT BEHIND. THEY WANTED TO CREATE TARGETSTHAT WERE FAR ADVANCED BEYOND WHAT NO CHILD LEFT BEHIND REQUIRED.THEY CREATED THEM, AND THE CUMULATIVE EFFECT WAS DISASTROUS ANDDESTRUCTIVE FOR THEM. THEY NOT ONLY CREATED TARGETS THAT EXCEEDED156NO CHILD LEFT BEHIND, BUT THEY IMPOSED PRESSURES TO ACHIEVE THOSETARGETS, AND THOSE PRESSURES WERE BOTH NEGATIVE AND POSITIVE.THE NEGATIVES WERE, YOU WERE GOING TO BE PUBLICLY HUMILIATED IF YOUDIDN'T MEET TARGETS. IF YOUR SCHOOL DIDN'T MAKE IT, YOU, AS THEPRINCIPAL WERE GOING TO BE ON THE HOOK. IF YOU, AS A TEACHER,DIDN'T MAKE IT, THEN YOU WERE GOING TO BE ON THE HOOK, AND YOU WEREGOING TO BE PUBLICLY HUMILIATED IN FRONT OF OTHER PRINCIPALS OROTHER TEACHERS, AND YOUR JOB WAS IN DANGER.IF YOU MADE IT, WELL, YOU GOT RECOGNITION. YOU GOT BONUSES. YOU GOTPRAISE. YOU GOT JOB SECURITY. THE END RESULT WAS ABSOLUTELY ADISASTER.THE SUPERINTENDENT SAID THE FOLLOWING. SHE QUOTED HERSELF ASTELLING HER PRINCIPALS WHEN SHE HIRED ONE, "YOU HAVE 3 YEARS TOMEET TARGETS, OR I WILL FIND SOMEONE WHO WILL. THERE WILL BE NOEXCEPTIONS AND NO EXCUSES." NOT ONLY DID SHE SAY THAT IN ONEINTERVIEW. SHE SAID IT IN YET A SECOND INTERVIEW.WHAT MESSAGE DID THAT DELIVER?157IF I REQUIRE MY LAWYERS IN MY FIRM TO BILL A CERTAIN NUMBER OFHOURS A YEAR THAT IS OUTRAGEOUS AND TOO MUCH, THEY ARE EITHER GOINGTO QUIT OR THEY ARE GOING TO LIE AND CHEAT.WE MUST SET REASONABLE GOALS, NOT UNREASONABLE GOALS.WE HAVE HEARD ABOUT SOME OF THESE TARGETS AND WARNINGS, SIGNIFICANTINCREASES IN SCORES OR SIGNIFICANT DECREASES IN SCORES, ANDPROBABLY THE BETTER WORD IS UNUSUAL INCREASES OR DECREASES.SOME PEOPLE NEVER THINK ABOUT THE DECREASES, BUT THE DECREASES THISYEAR MIGHT TELL YOU SOMETHING ABOUT LAST YEAR--UNUSUAL ANSWERPATTERNS, HIGH WRONG-TO-RIGHT ERASURES, REPORT OF SUSPICIOUSCONDUCT, REPORT OF CHEATING.THOSE ARE DIFFERENT THINGS.PEOPLE TELL AND REPORT SOMETHING THAT THEY SAW THAT JUST DIDN'TSEEM RIGHT. PEOPLE SAW THINGS AND THOUGHT IT WAS CHEATING. YOU HAVETO PAY ATTENTION TO THOSE THINGS.GREG CIZEK AND I WERE TALKING DURING LUNCH. THE HIGH WRONG-TO-RIGHTERASURES THAT WERE MENTIONED ARE SELDOM, IF EVER, SKEWED BY ASINGLE STUDENT TO THE EXTENT THAT IT WOULD CAUSE YOU TO THINK THAT158CHEATING HAD HAPPENED IN THAT SCHOOL OR EVEN THAT CLASSROOM.WE LOOKED AT THOUSANDS OF TESTS. WE HAD STUDENT-LEVEL DATA. WECOULD GO TO EACH STUDENT'S TEST AND HOW MANY WRONG-TO-RIGHTERASURES THAT PARTICULAR STUDENT HAD ON EACH OF THE TESTS.WE NEVER FOUND ONE WHERE SOME STUDENT BY THEMSELVES BY GETTINGOFF TRACK AND CORRECTING THEIR SITUATION JUST THREW THE WHOLENUMBERS FOR THAT CLASS OFF SO BADLY THAT IT WOULD LEAD YOU TOBELIEVE THERE WAS JUST CHEATING THERE.YOU CAN GET THE STUDENT-LEVEL DATA. THE IDEA THAT WE WON'T BE ABLETO FOLLOW THIS IF WE GO TO COMPUTERIZED TESTS IS HOGWASH. COMPUTERSCAN DO ABOUT ANYTHING. THEY CAN TELL YOU WHEN PLUG IN AN ANSWER ANDWHEN YOU CHANGED THE ANSWER AND HOW MANY TIMES YOU CHANGED IT. YOUWILL JUST HAVE A NEW FORM OF ERASURE, IS WHAT YOU WILL HAVE.WHAT DO YOU DO WHEN YOU GET THESE REPORTS?WELL, YOU DON'T WANT TO BE A PENN STATE. YOU REPORT IT, AND YOUKEEP REPORTING IT, AND YOU REPORT IT UP THE LINE: ONE, TO SCHOOLLEADERSHIP--THE PRINCIPAL, THE ASSISTANT PRINCIPAL, THE TESTCOORDINATOR.159IF YOU ARE A STAFF MEMBER OR A TEACHER, YOU REPORT IT, AND YOUREPORT IT UP THE LINE UNTIL SOMEBODY PAYS ATTENTION.THE SCHOOL HAS TO REPORT IT TO THE SYSTEM LEADERSHIP AND TAKE ITUP, ASSISTANT SUPERINTENDENTS AND SUPERINTENDENTS OF THE SYSTEM.THE SYSTEM NEEDS TO LET THE LEGAL COUNSEL FOR THE SYSTEM KNOW WHATTHE PROBLEM MAY BE. THE LEGAL COUNSEL NEEDS TO BE INVOLVED.YOU GENERALLY ARE GOING TO WANT TO NOTIFY THE STATE EDUCATIONAUTHORITIES IN YOUR STATE. IT MAY NOT BE BECAUSE THERE IS ANYTHINGMORE THAN YOU WANT TO LET THEM KNOW WHAT YOU ARE LOOKING INTO, ANDDO THEY HAVE ANY ADVICE FOR COUNSEL FOR YOU?YOU MAY WANT TO LET YOUR STATE LICENSING AUTHORITY KNOW, NOTBECAUSE YOU HAVE ANYTHING OTHER THAN TO LET THEM KNOW THAT THEYMAY, TOO, HAVE SOME ADVICE FOR YOU AS TO HOW YOU MIGHT GO ABOUT IT.YOU MAY NEED TO LET YOUR SCHOOL RESOURCE OFFICERS OR LAWENFORCEMENT KNOW. IT DEPENDS ON WHO YOU ARE GOING TO USE FOR YOURINVESTIGATION, BUT YOU NEVER EVER GO WRONG BY REPORTING IT UP THECHAIN, AND IF THE PERSON AHEAD OF YOU DOES NOT DO ANYTHING, GOABOVE THEM AND BEYOND THEM.160KEY COMPONENTS.NEVER EVER IGNORE ANY COMPLAINT. ALL REPORTINGS ARE IMPORTANT.THEY MUST BE VIGOROUSLY AND THOROUGHLY INVESTIGATED, AND IF IT ISSOMETHING PRETTY BIG, YOU ARE GOING TO NEED SOME SERIOUSORGANIZATION AND PLANNING BEFORE YOU DO IT.EVERY WITNESS MUST BE INTERVIEWED, EVERYONE WHO MAY KNOWSOMETHING, IF IT IS POSSIBLE THEY KNOW SOMETHING, AND IT IS NOTABOUT WHAT THEY KNOW DIRECTLY.IT IS ABOUT WHAT THEY KNOW IN ANY REGARDS.WE FOUND THAT THE PRELIMINARY INVESTIGATION DONE BY ATLANTA,WETELL THE PEOPLE, "WELL, WE JUST ONLY WANT TO KNOW WHAT YOU KNOW.""WELL, I HEARD--""WELL, I DON'T KNOW WHAT YOU HEARD."YOU WANT HEARSAY. YOU WANT GOSSIP. YOU WANT INFORMATION FROMWHATEVER SOURCE IF YOU'RE ACTUALLY INVESTIGATING IT. WHY? BECAUSESOMEONE UNRELATED TO THE WITNESS YOU ARE TALKING TO MAY VERY WELLHAVE TOLD YOU THAT SAME KIND OF INFORMATION.161THESE PEOPLE HAVE NO CONNECTION. IT TELLS YOU THERE IS SMOKE OUTTHERE.YOU NEED TO FIGURE OUT WHERE IT IS COMING FROM.SCHOOL PEOPLE ARE NOT REALLY WELL-EQUIPPED TO BE INVESTIGATORS.THAT'S NOT WHAT YOU ARE TRYING TO DO. YOU ARE TRYING TO EDUCATE.THAT'S WHY YOU GOT TO USE RESOURCES OF ATTORNEYS, INVESTIGATORS,LAW ENFORCEMENT WHERE APPROPRIATE, BUT EVERYBODY NEEDS TO BEINTERVIEWED.WHEN INTERVIEWING STUDENTS, YOU GOT TO BE CAREFUL, NO DOUBTABOUT IT, BUT STAFF, CUSTODIAL STAFF, THEY KNOW A LOT. THEY HAVETHEIR EYES OPEN. I'LL GIVE YOU SOME OF THE STORIES.YOU JUST HAVE TO BE CAREFUL.TEACHER HAD 6 DIFFERENT STUDENTS, WHO SAY SHE GAVE THEM ANSWERS.HER RESPONSE WHEN ASKED, "WHY ARE 6 STUDENTS TELLING US YOU GAVETHEM ANSWERS ON THOSE TESTS?" HER BEST RESPONSE WAS, "PROBABLY ACASE OF MISTAKEN IDENTITY."NOW, DID THAT ANSWER TELL YOU ANYTHING?162IT CERTAINLY DID US.REINTERVIEWS. THIS IS IMPORTANT TO NOTE. ANYBODY WHO REMEMBERS"COLUMBO," THIS IS HOW IT WORKS. YOU KEEP GOING BACK. GOODINVESTIGATORS ACT LIKE THEY'RE DUMB AS A BRICK, BUT THEY KEEP GOINGBACK, JUST LIKE HE DID.IN THE ATLANTA INVESTIGATION, NOT ONE TEACHER OR ADMINISTRATOREVER CONFESSED AND ADMITTED WRONGDOING ON THE FIRST INTERVIEW, NOTONE, YET 82 ENDED UP CONFESSING.YOU MUST GO BACK, AND THE MORE INFORMATION YOU GAIN, THE MOREPRESSURE YOU PUT ON AS THE QUESTIONS GO FORWARD. YOU HAVE TOCONSIDER POLYGRAPHS.THESE ARE EDITORIAL COMMENTS BY SOMEBODY WHO WAS INVESTIGATED INTHE SYSTEM.I'M NOT A PROFESSIONAL EDUCATOR. I AM A SCHOOL BOARD LAWYER. THETESTS SHOULD BE ABOUT THE CHILDREN, NOT ADULTS. IF YOU TAKE THEADULTS OUT OF IT WHERE THE TEST IS NOT ABOUT THEM, THEY HAVE NOREASON TO CHEAT.163WHEN THE TEST BECAME ABOUT TEACHERS AND SCHOOLS, TEACHERS ANDSCHOOLS HAD SOMETHING AT STAKE, A REASON TO CHEAT.BECAUSE THE TEST TIED TO ADULTS, IMAGE BECOMES IMPORTANT. IT WAS ANOVERRIDING FACTOR IN ATLANTA. IMAGE IS NOT MORE IMPORTANT THANCHILDREN, AND NOTHING, ABSOLUTELY NOTHING, IS MORE IMPORTANT THANCREATING A CULTURE OF INTEGRITY WHERE AN HONOR SYSTEM CONTROLS ANDSUSPICIOUS ACTIVITY IS REPORTED AND ENCOURAGED TO BE REPORTED.IN ATLANTA, THEY HAD JUST THE OPPOSITE. THEY HAD A CULTURE OF FEARAND A CULTURE OF INTIMIDATION, AND PEOPLE WHO REPORTED GOT PUNISHEDAND LOST THEIR JOBS. THAT SILENCES OTHER PEOPLE.IN A SYSTEM WHERE PEOPLE REPORT AND SOMETHING IS DONE ABOUT IT,THAT MAKES OTHERS WILLING TO DO IT, AS WELL, AND THIS GOES FOR BOTHADULTS AND STUDENTS. IT SHOULD BE AN HONOR SYSTEM.THAT'S IT.JACK BUCKLEY: THANKS VERY MUCH. TISHA?TISHA EDWARDS: GOOD AFTERNOON.164MY PRESENTATION WILL FOCUS ON SOME LESSONS LEARNED FROM BALTIMORECITY PUBLIC SCHOOLS TEST MONITORING AND INVESTIGATION PROCESS. IWILL PROVIDE A SENSE OF OUR SCHOOL SYSTEM'S BEST PRACTICES ON ITSTEST MONITORING AND INVESTIGATION PROCESS AND SHARERECOMMENDATIONS FOR RESTORING CREDIBILITY.WE HAD A PARTICULAR INCIDENT THAT REALLY RAISED OUR ORGANIZATIONALAWARENESS AROUND ISSUES RELATING TO CHEATING IN OUR SCHOOL SYSTEM.ONE OF OUR SCHOOLS WAS ALLEGED TO HAVE ALTERED ITS TEST SCORES FROMTHE MARYLAND STATE ASSESSMENT IN ORDER TO QUALIFY FOR BLUE RIBBONSTATUS, AND THERE HAD BEEN PREVIOUS ALLEGATIONS CONCERNING THISSCHOOL THAT WERE FOUND TO BE UNSUBSTANTIATED, AND TO BOB'SPOINT, OUR ALLEGATIONS ACTUALLY STARTED FROM A PARENT, AND THISPARENT WAS PRETTY ADAMANT ABOUT TAKING THE ALLEGATIONS TO THE TOP.NOT ONLY DID SHE NOTIFY THE SUPERINTENDENT. SHE ALSO NOTIFIED THESTATE, AND SHE TOOK IT UPON HERSELF TO COME TO EVERY BOARD MEETINGFOR ALMOST 6 MONTHS AND TESTIFY PUBLICLY, AND SHE HAD NO CONCERNSABOUT BEING ANONYMOUS. SHE WANTED PEOPLE TO KNOW THAT SHE HAD THISCONCERN, AND SHE PUT THE ORGANIZATION ON NOTICE THAT THERE WAS THISALLEGATION. THIS PARTICULAR INVESTIGATION LASTED ABOUT 18 MONTHS,WHICH REALLY PUT THE STRAIN ON THE ORGANIZATION BECAUSE THERE WASTHIS SENSE THAT WE WERE NOT DOING ANYTHING, BUT IN ALL ACTUALITY,WE WERE MOVING REALLY FEVERISHLY TO ASCERTAIN WHAT WAS HAPPENING,165AND WE WERE ENGAGED IN LOTS OF ANALYSIS, LOTS OF INTERVIEWING ANDREINTERVIEWING, AS BOB COMMUNICATED IN HIS PRESENTATION.CITY SCHOOLS' EXPERIENCECHALLENGES ESTABLISHING CULPABILITY. ONE OF THE REALLY DIFFICULTTHINGS--AND I DON'T THINK THAT WE REALLY TALKED ABOUT THAT TODAY-IS, ONCE YOU FIND OUT THAT THERE HAS BEEN WIDESPREAD CHEATING, ITIS A WHOLE OTHER SITUATION TO FIND OUT WHO IS THE CULPRIT IN THATSITUATION, AND WE HAD A REAL HARD TIME ESTABLISHING CULPABILITY.PEOPLE WERE NOT TALKING.THERE WAS OVERSENSITIVITY IN REPORTING OF ALLEGATIONS. THERE WASBACKTRACKING OF ALLEGATIONS ONCE STAFF AND PARENTS WEREINTERVIEWED. AS YOU ALL MAY KNOW, SCHOOLS ARE, IN MANY CASES,FAMILIES. THEY ARE PART OF A COMMUNITY, AND ONCE PEOPLE FIND OUTTHAT THERE IS GOING TO BE HIGH ACCOUNTABILITY, A LOT OF TIMES, THEYBACKTRACK FROM PREVIOUS STATEMENTS. SO THAT MADE IT VERY DIFFICULTFOR US.BY CONDUCTING MULTIPLE ANALYSES—COMMUNICATION ANALYSIS, ERASUREANALYSIS—CITY SCHOOLS ESTABLISHED A VERY CLEAR CASE THAT THERE WASLIKELY SYSTEMIC CHEATING OCCURRING AT THIS PARTICULAR SCHOOL. OUR166LESSONS LEARNED IS THAT ANALYSIS IS NEVER ENOUGH TO ESTABLISHCULPABILITY.UNLIKE ATLANTA, WE DIDN'T HAVE ANYBODY CONFESS, AND SO IT MADE ITREALLY DIFFICULT FOR US TO FIGURE OUT WHAT THE INDIVIDUALACCOUNTABILITY SHOULD LOOK LIKE, EVEN THOUGH WE WERE PRETTY SURETHAT CHEATING HAD OCCURRED. UNLESS COMPREHENSIVE INVESTIGATIONSOCCUR, THE CREDIBILITY OF DISTRICTS AND SCHOOLS SUFFER.WHILE THE ORGANIZATION IS UNDER INVESTIGATION, WE KNEW THAT WEWERE GOING TO TAKE A HIT, BUT WE FELT LIKE THE PRICE WAS WORTH IT.AGAIN, IT WAS AN 18-MONTH INVESTIGATION.EVERY BOARD MEETING, WE HAD SOMEONE CALLING US TO THE CARPET.WE HAD FOLKS, BECAUSE OF THE TIME IT WAS TAKING, DIDN'T FEEL LIKEWE WERE MOVING FAST ENOUGH, AND SO THERE WAS THIS MOMENT IN TIMEWHERE IT WAS VERY DIFFICULT FOR THE ORGANIZATION, BUT OURSUPERINTENDENT WAS VERY CLEAR THAT NOTHING MATTERED MORE THANTHE INTEGRITY OF OUR SCHOOL SYSTEM AND THAT WE WOULD NOT STAND BYAND ALLOW ANYONE TO CALL INTO QUESTION THE CAPACITY OF OURCHILDREN.167HE REALLY SAW THIS AS AN ATTACK ON TEACHING AND LEARNING IN OURDISTRICT.WE, AFTER THIS PARTICULAR CASE, DID A LOT OF WORK TO STRENGTHEN ALLASPECTS OF OUR INTERNAL CONTROLS. THE MARYLAND STATE DEPARTMENT OFEDUCATION HAS PUBLICLY SAID THAT BALTIMORE CITY HAS ONE OF THE MOSTSTRINGENT TESTING PROTOCOLS IN THE STATE AS A RESULT OF THISINCIDENT. SO WE TRIED TO LEARN FROM IT, AND WE TRIED TO CREATE ITAS AN OPPORTUNITY TO BE BETTER AS AN ORGANIZATION.WE DID CONDUCT MULTIPLE TYPES OF ANALYSIS IN ORDER TO ESTABLISHCULPABILITY WITH CERTAINTY, AND WE ARE CONTINUING TO ADHERE TO THECODE OF MARYLAND STATUTES FOR CONDUCTING TESTING INVESTIGATIONS.JACK BUCKLEY: THANK YOU VERY MUCH.TISHA EDWARDS: I’M NOT FINISHED.JACK BUCKLEY: I'M SORRY.[LAUGHTER]JACK BUCKLEY: SUCH A PREGNANT PAUSE, I THOUGHT--168TISHA EDWARDS: THANK YOU. THANK YOU.SO WE TOOK STEPS FOR PREVENTING POTENTIAL VIOLATIONS. WEESTABLISHED AND IMPLEMENTED INTERNAL CONTROLS THAT WOULD SIGNALPROBLEMS. WE REALLY HAVE WORKED HARD TO BE PROACTIVE. WE USED SOMEOF THE ANALYSES THAT YOU'VE TALKED ABOUT EARLIER TODAY--GROWTHANALYSIS, BENCHMARK ANALYSIS, LEAVER ANALYSIS WHERE WE LOOK ATSTUDENTS AND HOW THEY ACHIEVE ONCE THEY LEFT THE SCHOOL WHERE THEYWERE TESTED THE PREVIOUS YEAR, AND COHORT ANALYSIS.WE HAVE ALSO DEVELOPED A VERY DETAILED MONITORING PLAN WHERE WEMONITORED ALL OUR HIGH-STAKES ASSESSMENT ANNUALLY. NONE OF THIS ISDONE BY STATE MANDATE OR DIRECTIVE. THIS IS ALL BASED ONORGANIZATIONAL WILL AND OUR DESIRE TO PUT OURSELVES IN A POSITIONWHERE WE NEVER HAVE THE SITUATION THAT HAS HAPPENED IN THE PAST TOHAPPEN AGAIN, AND WE'VE ENHANCED THE EDUCATION AND AWARENESS OF ALLOF OUR STAKEHOLDERS.WE NOW PROVIDE MANDATORY TRAINING FOR ALL PERSONS PARTICIPATING INTEST ADMINISTRATION. IN THE PAST, IT WAS ONLY LIMITED TO TESTINGCOORDINATORS, SCHOOL-BASED TESTING COORDINATORS. NOW ALLPRINCIPALS, ALL TEACHERS, ALL TEST MONITORS, EVERYONE WHO HAS ANYROLE IN TESTING ADMINISTRATION MUST GO THROUGH AN ANNUAL TESTINGTRAINING DONE BY OUR ORGANIZATION.169WE DISSEMINATE A VIDEO FOR EVERYONE IN THE SCHOOL SYSTEM TO LOOK ATEVERY YEAR. IT'S A VIDEO WHERE DR. ALONSO GETS ON CAMERA AND, FORTHE RECORD, COMMUNICATES HOW IMPORTANT TESTING INTEGRITY IS.THIS IS A VIDEO THAT THE CUSTODIAN WATCHES, THE SECRETARY WATCHES,THE TEACHERS WATCH. IT HAS WIDESPREAD DISTRIBUTION THROUGHOUT THEORGANIZATION.ALL OF OUR PERSONNEL WHO ARE INVOLVED IN THE ADMINISTRATION AREALSO REQUIRED TO SIGN A NONDISCLOSURE FORM, AND WE COLLABORATE VERYSTRONGLY WITH THE STATE TO ENSURE, AGAIN, THAT OUR TESTINGPROTOCOLS ARE ALIGNED WITH BEST PRACTICES.DETECTING POTENTIAL VIOLATIONS.WE HAVE ESTABLISHED WELL-DEFINED PROTOCOLS. WE MADE AN INVESTMENTOF ABOUT HALF A MILLION DOLLARS A YEAR IN TESTING MONITORING ANDENSURING THAT WE HAVE TIGHT PROTOCOLS AND WE'VE COMMUNICATED THOSEAND MEMORIALIZED THEM, AND WE FLAG POTENTIAL CONCERN WITH SUPPORTFROM OUR STAKEHOLDERS. AGAIN, THE STATE IS A REAL WONDERFUL PARTNERIN THIS PROCESS.170THE STATE PROVIDES US WITH SCORING ALERTS, WHICH WE ACT ON VERYSWIFTLY, AND THE STATE IS ALSO RESPONSIBLE FOR PERFORMING THEERASURE ANALYSES WHEN NECESSARY.IN TERMS OF INVESTIGATING POTENTIAL VIOLATIONS, AS YOU'VEDISCUSSED EARLIER TODAY, WE DO HAVE A TIP LINE. WE GET INFORMATIONBOTH ON THE STATE LEVEL AND AT THE LOCAL LEVEL.WE TALKED ABOUT SOME OF THE PROTOCOLS, AND WE DO HAVE A STANDARDAND EXPECTATION THAT EVERY ALLEGATION IS INVESTIGATED. THERE ISNOT A SINGLE ALLEGATION THAT IS IGNORED OR PUT IN A FILE.WE'VE DIRECTED THE RESOURCES SO THAT WE CAN INVESTIGATE EVERYALLEGATION, AND WE'VE STARTED TO IMPLEMENT STRONG QUALITATIVEEVIDENCE IN HELPING US TO HAVE AN UNDERSTANDING OF WHAT'S HAPPENINGIN SCHOOLS AND, IN PARTICULAR, TESTING ADMINISTRATIONS.ONE OF THE THINGS THAT WE WOULD ADVISE OTHER LOCAL SCHOOLSYSTEMS IS THAT YOU REALLY NEED TO HAVE A STRONG INVESTIGATIVEREPORT. YOUR CONCLUSIONS NEED TO BE BASED ON SOLID DATA. WHEREPOSSIBLE, YOU NEED TO TRIANGULATE THE DATA. WE'VE TALKED A LITTLEBIT ABOUT LOOKING AT DIFFERENT DATA SETS TO GET A SENSE OF HOW ISONE171THING HAPPENING IN ONE ENVIRONMENT AND THEN THERE'S ANOTHER DATASET THAT COMPLETELY CONTRADICTS THAT REALITY. SO TRIANGULATING THEDATA IS VERY IMPORTANT.AND YOU NEED TO PRESENT YOUR FINDINGS TO PROPER INTERNALAUTHORITIES. OUR CEO IS MADE AWARE OF EVERY INVESTIGATION ALONG THEWAY. WE MEET WITH HIM ON A REGULAR BASIS SO THAT HE UNDERSTANDSWHERE WE ARE IN THE INVESTIGATIVE PROCESS.THE OTHER THING WE SHOULD SAY IS, YOU SHOULD PLAN FOR LITIGATION.THERE IS NO WAY THAT YOU ARE GOING TO HAVE A STRONG INVESTIGATIVEPROTOCOL AND PROCESS IN YOUR DISTRICT AND NOT BE INVOLVED IN FUTURELITIGATION. SO YOU HAVE TO BE PREPARED FOR LITIGATION ON THE FRONTEND.MAINTAINING CREDIBILITY.THIS IS SOMETHING, AGAIN, THAT BOB REFERRED TO IN HIS PRESENTATION.WE FELT LIKE BEING OUT FRONT, AND OUR SUPERINTENDENT ACTUALLY HAD APRESS CONFERENCE AROUND THE ALLEGATIONS IN OUR SCHOOL SYSTEM.IT WAS IMPORTANT THAT HE ESTABLISH CREDIBILITY FOR THE DISTRICT,AND EVEN THOUGH WE HAD THESE PARTICULAR INCIDENCES WHERE WE HADCONCERNS, WE ALSO KNEW THAT WE HAD TO HAVE THE PUBLIC CONTINUE TO172BUY INTO OUR SCHOOL SYSTEM AND TO KNOW THAT WE WERE GOING TO DO THERIGHT THING FOR OUR CHILDREN.MY TIME IS UP.JACK BUCKLEY: KEEP GOING. I OWE YOU--TISHA EDWARDS: NO. I'M GOOD.[LAUGHTER]I'M GOOD.JACK BUCKLEY: YOU ARE?TISHA EDWARDS: YEAH. SURE.JACK BUCKLEY: STEVE.STEVE FERRARA: THANK YOU.GOOD AFTERNOON, EVERYONE, AND THANK YOU, JACK AND EUNICE, FORGIVING ME THE OPPORTUNITY TO SPEAK.173I'M GONNA TRY TO ADDRESS ISSUES AROUND INVESTIGATION, AND THIS ISBASED ON MY--OH, YEAH; THAT'S A GOOD IDEA--EXPERIENCE AS A FORMERSTATE ASSESSMENT DIRECTOR. SOME PEOPLE WOULD SAY I'M ONE OF THEMANY RECOVERING STATE ASSESSMENT DIRECTORS.SO...JACK BUCKLEY: TRY THE ONE ON THE LEFT.STEVE FERRARA: THERE WE GO. OK.SO AS AN OVERVIEW, I WANT TO MAKE SOME BACKGROUND COMMENTS, AND IHOPE YOU'LL INDULGE ME. SOME OF THEM ECHO THINGS THAT YOU'VE HEARDBEFORE, BUT THEY'RE BACKGROUND TO WHAT I WANT TO SAY ABOUTCONDUCTING INVESTIGATIONS AND CONSIDERATIONS FOR CONDUCTING THOSEINVESTIGATIONS.SO TO START WITH, I WANT TO DRAW A CONCEPTUAL DISTINCTION BETWEENTEST SECURITY AND TESTING INTEGRITY, AND I WAS PLEASED TO SEE THATNCES HAS TREATED THIS AS AN INTEGRITY ISSUE, NOT JUST A TESTINGSECURITY ISSUE.TESTING INTEGRITY OBVIOUSLY REQUIRES TEST SECURITY—SECURITY OF THEMATERIALS, APPROPRIATENESS OF THE PROCEDURES, AND SO FORTH. IT174ALSO REQUIRES PROFESSIONAL ETHICAL BEHAVIOR AND THINKING, AND WEHAVE HEARD REFERENCES TO THAT THROUGH THE MORNING AND THROUGH THESEPREVIOUS PRESENTATIONS.WITHOUT PROFESSIONAL ETHICAL BEHAVIOR, YOU CAN'T REALLY HAVE TESTSECURITY. WHEN YOU HAVE THOSE TWO THINGS TOGETHER, WHAT YOU CAN GETIS WHAT WE ARE ALL LOOKING FOR--TRUSTWORTHY DATA AND SOMETHING THATOFTEN GETS LOST. I'VE ONLY HEARD ONE OR TWO MENTIONS OF IT TODAY.IT HELPS--BY HAVING TESTING INTEGRITY, YOU AVOID INAPPROPRIATEMESSAGES TO CHILDREN. WHEN YOU THINK ABOUT IT--I'VE HAD TO SAY THISIN PUBLIC FORUMS BEFORE, INCLUDING A COURTROOM--WHEN A TEACHERCHEATS WITH STUDENTS, THE MESSAGE TO THE STUDENTS IS, "CHEATING ISOK, AND MAYBE THERE'S SOMETHING WRONG WITH ME. THAT'S WHY THETEACHER IS CHEATING."SO THE WAY I THINK OF THIS IS A MUTUALLY SUPPORTIVE TEST SECURITYSYSTEM AND A CULTURE OF PROFESSIONAL ETHICS. WE'VE HEARDREFERENCES TO CULTURE ALL DAY SO FAR. SO THE SECURITY SYSTEMPROVIDES RULES, GUIDELINES, AND SO FORTH. IT SHOULD REQUIRETRAINING.175THERE SHOULD BE EXTENSIVE TRAINING LIKE THE TYPE THAT TISHA HASREFERRED TO. I THINK YOUR SYSTEM IS PRETTY IMPRESSIVE, WHAT I'VEHEARD FROM YOU SO FAR.THIS NOTION OF A CULTURE--SCHOOLS AND SCHOOL SYSTEMS ARE MINICULTURES. AT LEAST, I VIEW THEM THAT WAY.AND WHAT IS A CULTURE? WELL, IF YOU GO TO THAT--IF YOU LOOK UPDEFINITIONS OF "CULTURE," I CHOSE AN EASY ONE. IT IS A SET OFSHARED ATTITUDES, VALUES, GOALS, AND PRACTICES THAT CHARACTERIZEAN INSTITUTION OR ORGANIZATION.I THINK THAT APPLIES VERY WELL TO WHEN YOU THINK ABOUT SCHOOLSAND SCHOOL SYSTEMS AS CULTURES. IN ORDER TO SUPPORT THE SECURITYRULES AND PROCEDURES AND SO FORTH THAT STATES AND LOCAL SCHOOLSYSTEMS SET UP, YOU NEED THAT CULTURE OF PROFESSIONAL ETHICALTHINKING AND BEHAVIOR,AND YOU DON'T NECESSARILY HAVE TO PERSUADE PEOPLE TO BUY INTO THECULTURE. YOU DO HAVE TO CONVINCE THEM THAT THEY HAVE TO ABIDE BYIT.WHEN YOU HAVE THOSE TWO THINGS TOGETHER, THEN YOU HAVE THE CHANCETO HAVE A SYSTEM WITH GREAT INTEGRITY AND MINIMIZE VIOLATIONS,176WHETHER THEY ARE INADVERTENT OR ADVERTENT. SO HOW DOES THE SYSTEMAND CULTURE WORKING TOGETHER SUPPORT INVESTIGATIONS? I'M GOING TODO THIS VERY BRIEFLY.YOU CAN THINK HYPOTHETICALLY ABOUT 3 GROUPS OF PEOPLE.LEADERS IN THE SCHOOL WHO THINK AND PRACTICE IN PROFESSIONALLYETHICAL WAYS, THEY BUY INTO THE NOTION. THEY ACT ACCORDINGLY, ANDTHEY'RE FAIRLY VOCAL ABOUT IT. THEY ARE LEADERS ABOUT THINGS LIKEPROFESSIONAL ETHICS.THERE MAY BE A SECOND GROUP THAT ACTS APPROPRIATELY. THEY MAY NOTBUY INTO THE SYSTEM. THEY MAY NOT BE VOCAL ABOUT IT, BUT THEY GOALONG WITH THE SYSTEM, AND THEN THERE ARE OTHER PEOPLE.NOW, THE REASON I LIKE TO THINK ABOUT THOSE 3 GROUPS IS BECAUSE ITHELPS YOU THINK ABOUT WHEN YOU'VE DETECTED ANOMALIES AND YOU HAVETO FOLLOW UP WITH INVESTIGATION. I AGREE YOU HAVE TO FOLLOW UP ONEVERYTHING. IT ENCOURAGES AND SUPPORTS PROFESSIONALLY ETHICALTHINKING AND BEHAVIOR, AND IT ENCOURAGES OTHER HELPFUL PARTS OF THESYSTEM, LIKE REPORTING OF IRREGULARITIES AND COOPERATIONS WITHINVESTIGATIONS WHEN IRREGULARITIES MAY OCCUR.177SO WITH THAT AS A BACKGROUND, I'M NOT GOING TO TALK EVEN FASTERTHAN I ALREADY AM. I WANT TO TALK ABOUT A COUPLE OF VIGNETTES THATI THINK ILLUSTRATE A COUPLE OF POINTS. ONE OF THE THINGS THAT ILEARNED--I THINK YOU CAN JUST ARRIVE AT BY LOGIC--IS, WHAT HAPPENSWHEN A SCHOOL IS INVESTIGATED FOR POTENTIAL VIOLATIONS, WHETHERTHE VIOLATIONS OCCURRED OR NOT, WHETHER ANYBODY IS CAUGHT ATSOMETHING INAPPROPRIATE OR NOT.WHAT OFTEN HAPPENS IS, THE BEHAVIOR IN THE SCHOOL CHANGES,TYPICALLY IN A POSITIVE DIRECTION. YOU CAN SEE THAT IF YOU JUST GOBACK TO THE SCHOOL AFTER THE INVESTIGATION. YOU SEE THAT BEHAVIORHAS CHANGED.ONE OF THE THINGS THAT ALWAYS WORRIED ME IS THAT IT CAN DRIVEINAPPROPRIATE BEHAVIOR FURTHER UNDERGROUND. I HAVE SEEN INCIDENTS-AND I'M GOING TO TELL YOU A BRIEF EXAMPLE OF THIS--WHERE THEBEHAVIOR WAS ACTUALLY QUITE BRAZEN, AND SO WHEN THERE IS ANINVESTIGATION, THAT BRAZEN BEHAVIOR DISAPPEARS. THE QUESTION IS,WHAT SORT OF UNDERGROUND STUFF MAY BE GOING ON?SO TWO QUICK VIGNETTES RELATED TO INVESTIGATIONS AND WHAT CAN GO ONAT SCHOOLS. WHEN I WAS A STATE ASSESSMENT DIRECTOR, I WENT TO VISITA BLUE RIBBON SCHOOL TO WATCH AN ADMINISTRATION OF THE STATE178ASSESSMENT. IT WAS THE MARYLAND SCHOOL PERFORMANCE ASSESSMENTPROGRAM.THE PRINCIPAL ASKED ME TO GO AND OBSERVE THE STATE TEACHER OF THEYEAR IN HER SCHOOL. SHE WAS OBVIOUSLY AND APPROPRIATELY VERY PROUDOF THIS WOMAN.SO EVERYBODY KNOWS WHO I AM. I'M SITTING IN THE CLASSROOM. 10MINUTES BEFORE THE ADMINISTRATION IS GOING TO START, THIS TEACHEROF THE YEAR WAS CHARISMATIC, ABSOLUTELY CHARISMATIC. SHE BROUGHTTHE THIRD-GRADERS AROUND HER.SHE SAT IN A ROCKING CHAIR. SHE READ THEM A STORY, AND SHE SAID, "IWANT YOU TO LISTEN CAREFULLY TO THIS STORY. IT'S GOING TO HELP YOURELAX AND GET READY FOR THE ASSESSMENT," AND THEY WERE ENRAPTLISTENING TO THE STORY AND LISTENING TO THIS CHARISMATIC PERSON.SHE PUT THE KIDS AT THEIR DESKS, DO THE TEST ADMINISTRATION. THEYBEGAN TO ADMINISTER THE ASSESSMENT.I WAS ALREADY NERVOUS. THE STORY SHE HAD READ TO THE KIDS WASRELEVANT TO THE THEME OF THE READING PASSAGES THE KIDS WERE READINGIN THE ASSESSMENT. NOW, IN A CASE LIKE THAT, THERE WASN'T MUCHINVESTIGATION THAT I HAD TO DO--IT WAS A FOLLOW-UP--BUT I THINK IT179MAKES THE POINT OF HOW--I THINK IT ILLUSTRATES HOW DELICATE THINGSCAN BECOME WHEN YOU DETECT CHEATING AND WHEN YOU HAVE TO CONDUCT ANINVESTIGATION.THIS IS ALL LEADING TO A POINT ABOUT INVESTIGATIONS.A SECOND WAS SAME PROGRAM, MSPAP, ALL CONSTRUCTED-RESPONSE ITEMS.STUDENTS WROTE THE RESPONSES TO ALL ITEMS. WE DISCOVERED JUST BYDUMB LUCK THAT THE TEACHER HAD DICTATED RESPONSES TO ALL OF THESTUDENTS IN HER TESTING GROUP.NOW, NO ONE SAW HER DO THAT. WE FOUND IT IN THE SCORING PROCESS.ALL STUDENTS HAD EXACTLY THE SAME RESPONSES TO ALL ITEMS OR ALMOSTEXACTLY THE SAME RESPONSES TO ALL ITEMS.THE INVESTIGATION THERE WAS NOT THAT DIFFICULT BECAUSE THEEVIDENCE WAS SO STRONG. NONETHELESS, INVESTIGATION WAS REQUIRED TOBUILD THE CASE WE NEEDED TO PROCEED.SO AGAIN, THAT'S MORE BACKGROUND TO THESE CONSIDERATIONS THAT I'DLIKE TO BRING TO YOUR ATTENTION BEFORE I FINISH UP.180FIRST OF ALL, TYPICALLY, TEST DEVELOPMENT AND DELIVERY CONTRACTORSAREN'T NECESSARILY INVOLVED IN INVESTIGATIONS ONCE AN ANOMALY ISDETECTED.NOT TYPICALLY, AS I SAID EARLIER IN RESPONSE TO JACK'S QUESTION, WEMAY BE INVOLVED IN TEST SECURITY DETECTION. WE ARE DEFINITELYINVOLVED IN PROTECTING SECURITY.TYPICALLY, WE ARE NOT INVOLVED IN INVESTIGATIONS, AND I THINK IT ISIMPORTANT TO KEEP THE ROLES AND RESPONSIBILITIES CLEAR.SECOND OF ALL, PRINCIPLES AND PROCEDURES FOR INVESTIGATIONS, THEYMUST BE CLEAR. THEY MUST WITHSTAND POTENTIAL CHALLENGES INADMINISTRATIVE LAW COURTS. EDUCATORS--PRINCIPALS, ASSISTANTPRINCIPALS, ASSISTANT SUPERINTENDENTS--WHO ARE ASKED TO CONDUCTTHESE INVESTIGATIONS DON'T REALLY KNOW ANYTHING ABOUTINVESTIGATIVE PROCEDURES. THEY'RE NOT TRAINED TO DO THAT SORT OFTHING. THEY DON'T HAVE THE MIND-SET THAT'S INVOLVED. I THINK WHATWE NEED--THE KIND OF THING THAT TISHA REFERRED TO THAT'S GOING ONIN BALTIMORE CITY, THE KIND OF THING THAT BOB REFERRED TO--WE NEEDTRAINING FOR THE PEOPLE WHO ARE GOING TO CONDUCT THOSEINVESTIGATIONS.I'D SUGGEST THAT IT EVEN BE OUTSIDE PEOPLE.181THERE ARE IMPORTANT INVESTIGATION PROCEDURES, PROTOCOLS THATPEOPLE FOLLOW. THERE ARE ALSO SUBTLE THINGS, LIKE FBI AGENTS ANDOTHER INVESTIGATORS ARE TRAINED TO DETECT EVASIVE BEHAVIOR ANDLYING, AND THAT CAN HELP MAKE A DIFFERENCE ON HOW MUCH FURTHERTHE INVESTIGATION HAS TO GO.VERY QUICKLY, THIS WILL COME UP IN THE NEXT PANEL, I'M SURE, BUT IWAS ASKED TO BRING IT UP. THERE ARE CHEATING RISKS WITH NEW DEVICESTHAT ARE GOING TO HAVE A HUGE IMPACT ON HOW WE GO ABOUTINVESTIGATING WHEN WE FIND ANOMALIES. APPARENTLY, WHEN WE DOONLINE TESTING, WE CAN LOCK THINGS DOWN. KIDS CAN'T GO OUTSIDE OFTHE TEST. THEY CAN'T GO OUT ONTO THE WEB TO LOOK UP THINGS, AND WECAN DEAL WITH WHO IS SITTING IN FRONT OF THE COMPUTER AND ALL THATSORT OF THING.TABLETS INTRODUCE A WHOLE NEW SET OF RISKS, LIKE TASK SWAPPING,GOING OUT ONTO THE WEB, AND SCREENSHOT CAPABILITIES.THERE IS A VERY SIMPLE WAY TO TAKE SCREENSHOTS ON AN IPAD. SOEVENTUALLY, WE'LL HAVE TO WORK OUT HOW TO LOCK THAT STUFF DOWN.THE PROBLEM WILL BE THAT NOT EVERY KID IS GOING TO HAVE SOME SORTOF DEVICE. IF THE SCHOOLS DECIDE TO ALLOW KIDS TO BRING THEIR OWN182DEVICES, THINK OF THE PROBLEMS OF TRYING TO INVESTIGATE WHENTHERE'S BEEN AN ANOMALY WHEN A STUDENT HAS USED HIS OWN DEVICE.LAST COMMENT, SOMETHING BOB ALREADY SAID, IT'S NECESSARY TOINVESTIGATE, ENFORCE, AND SANCTION RIGOROUSLY AND CONSISTENTLY ANDFOLLOW UP ABSOLUTELY ON EVERYTHING.THANK YOU.JACK BUCKLEY: THANKS VERY MUCH. LOU?LOU FABRIZIO: ALL RIGHT.GOOD AFTERNOON.I WOULD LIKE TO THANK THE DEPARTMENT FOR GIVING ME THE OPPORTUNITYTO BE HERE TODAY. THIS HAS BEEN AN AMAZING DAY LISTENING TO ALL THEDIFFERENT PRESENTERS TALKING ABOUT ALL THESE VARIOUS ASPECTS OFTESTING INTEGRITY, BUT THERE OBVIOUSLY ARE LOTS OF THEMES THATWE'RE HEARING IN THE DIFFERENT SESSIONS OVER AND OVER AGAIN, ANDI'D LIKE TO JUST START OUT BY TALKING ABOUT THE RULES AND REGS INTHE ADMINISTRATIVE CODE.183IN NORTH CAROLINA, THESE ARE THE 3 MAIN THINGS THAT WE HOLD OUT ASTHE ADMINISTRATIVE CODE WHICH ARE ALL STATE BOARD REGULATIONS,RULES AND REGULATIONS, AND THE FIRST ONE IS THE TESTING CODE OFETHICS, AND I WANT TO BRING UP AN ISSUE THAT OCCURRED WHEN WEWENT THROUGH THE ADMINISTRATIVE PROCEDURES ACT IN NORTH CAROLINA.THIS WAS IN 1997, AND IN THE MATERIALS THAT WE SUBMITTED TO GOTHROUGH THE APA PROCESS, THE ATTORNEYS DIDN'T LIKE THE WORDING,WHICH WAS THAT THE STATE BOARD STRONGLY RECOMMENDS THE USE OFPROCTORS IN THE CLASSROOM, AND THE ATTORNEYS CAME BACK TO US ANDSAID, "OK. LOOK. THIS IS ADMINISTRATIVE CODE. YOU EITHER HAVE TOSAY YOU HAVE TO HAVE THE PROCTORS, OR YOU DON'T MAKE ANY MENTION OFIT," AND WE SAID WE HAVE TO HAVE THEM, AND WE HAVE HAD THEM EVERSINCE.IT HAS BEEN REQUIRED, AND EVERY YEAR, WE GET HATE MAIL FROM PEOPLEWHO ARGUE, "WHY DON'T WE TRUST THE TEACHERS? WHY ARE WE FORCINGTEACHERS TO HAVE TO SPEND TIME PROCTORING IN OTHER CLASSROOMS?"AND OUR RESPONSE IS, "BECAUSE WE WANT TO ENSURE THE TESTINGINTEGRITY OF THE SYSTEM." SO I DID WANT TO BRING THAT UP AS THEMIND-SETS THAT YOU ARE HAVING TO CHANGE IN ORDER TO MAKE THIS ALLBE EFFECTIVE.184WE HAVE TALKED ABOUT THE FACT THAT WE NEED TO BE RECORDING ANDMONITORING IRREGULARITIES, BUT I DIDN'T HEAR ANYONE TALK ABOUT ASPECIFIC SYSTEM. WE HAVE AN ACTUAL ONLINE SYSTEM WHERE DATA ISENTERED ONLINE AND CAN BE TRACKED. ALL THE STEPS OF THE PROCESS AREIN THAT SYSTEM. WE REQUIRE LOCAL INVESTIGATIONS BY THE L.E.A. WEHAVE STAFF WHOSE JOB IT IS TO ASSIST THE SCHOOL DISTRICTS IF THEYHAVE ANY QUESTIONS ABOUT HOW TO CONDUCT THAT INVESTIGATION, AND ANYSERIOUS IRREGULARITIES THAT GET IDENTIFIED REQUIRE, THEN, THE USEOF A CHECKLIST THAT MUST THEN BE SUBMITTED.ONCE THE L.E.A. SUBMITS ITS REPORT, THEN IT IS UP TO THE STAFF INACCOUNTABILITY SERVICES TO REVIEW THE ACTUAL REPORT.IF THE REPORT IS OF ONE OF THOSE SERIOUS IRREGULARITIES, WE THENFORWARD THAT REPORT TO THE ATTORNEY WITH THE STATE BOARD, AND THATPERSON, IN CONJUNCTION WITH THE ATTORNEY GENERAL'S OFFICE, MAKEDECISIONS AS TO WHETHER OR NOT A FURTHER INVESTIGATION NEEDS TOOCCUR OR WHETHER OR NOT THEY ARE OK WITH WHAT'S HAPPENED.BUT ONE OF THE THINGS THAT WE HAVE NOT SPENT A LOT OF TIME TALKINGABOUT, ALTHOUGH IT HAS BEEN MENTIONED A LITTLE BIT, ARE THESEBARRIERS, AND ONE OF THE BARRIERS THAT I KNOW WE ARE NOT ALONE INNORTH CAROLINA IS THE LIMITED RESOURCES.185EVERY YEAR, THE GENERAL ASSEMBLY CONTINUES TO MAKE CUTS TO THEBUDGET, AND THINGS HAVE TO BE ELIMINATED. WE DO NOT HAVE ANYPREASSIGNED STAFF WHOSE SOLE RESPONSIBILITY IS TO DOINVESTIGATIONS, AND WE DO NOT HAVE ANY SPECIFIC LINE ITEMS SETASIDE FOR DOING INVESTIGATIONS, BUT BECAUSE OF THAT, THE FOLLOW-UPINVESTIGATIONS THAT DO OCCUR TEND TO TAKE A LITTLE LONGER THAN WEWOULD LIKE.BALTIMORE IS A GOOD EXAMPLE, 18 MONTHS. THESE THINGS DO TAKE TIME.BASED ON THE INVESTIGATIONS THAT ARE THEN DONE BY THE STATE BOARDATTORNEY AND IN CONJUNCTION WITH THE ATTORNEY GENERAL'S OFFICE,SOME OF THE INDIVIDUALS THEN COME BEFORE WHAT'S CALLED THESTATE SUPERINTENDENT'S ETHICS ADVISORY COMMITTEE, AND THESEARE INSTANCES WHERE REVOCATION OF LICENSES WILL OCCUR, AND SOMEINVESTIGATIONS MAY ALSO RESULT IN CIVIL ACTION AND/OR, AND IALREADY SAID, THE LOSS OF THE LICENSE.ALL OF THAT KIND OF INFORMATION IS LAID OUT IN THAT TESTING CODE OFETHICS THAT I STARTED OUT WITH. THAT TESTING CODE OF ETHICS SAYSWHAT THE SUPERINTENDENT IS SUPPOSED TO DO, WHAT THE TEACHER ISSUPPOSED TO DO, WHAT THE PROCTOR IS SUPPOSED TO DO. IT EVEN GIVESEXAMPLES OF UNETHICAL PRACTICES SO FOLKS CAN'T SAY, "WELL, I DIDN'TKNOW THAT THAT WASN'T ALLOWED," BECAUSE IT IS IN THAT CODE OF186ETHICS, AND THEN THAT CODE OF ETHICS ENDS WITH THE STATEMENTTHAT CIVIL ACTION AND POSSIBLE REVOCATION OF LICENSE MAY OCCURIF ONE IS ACCUSED OF THE VIOLATION OF THE TESTING CODE OF ETHICS.SO ALL OF THAT STUFF IS LAID OUT IN THE PROCESS OR SYSTEM THAT WEHAVE. JUST SOME OTHER QUICK THOUGHTS.ONCE WE GET THE WORD THAT THERE IS SOMETHING THAT APPEARS PRETTYDARN SERIOUS--AND THAT NORMALLY OCCURS BECAUSE ONCE SOMEBODYENTERS ANYTHING INTO THAT ONLINE SYSTEM THAT WE HAVE, ITAUTOMATICALLY SENDS INFORMATION TO DIFFERENT INDIVIDUALS TOALERT THEM--WHAT WE HAVE LEARNED IS THAT IF YOU HEAR ABOUTSOMETHING THAT YOU THINK IS GOING TO BE VERY SERIOUS, IT IS ALWAYSBEST TO LET THE STATE SUPERINTENDENT KNOW, TO LET OTHER MEMBERS OFTHE TOP LEADERSHIP BE AWARE OF IT, ALSO TO MAKE OUR COMMUNICATIONSAND INFORMATION STAFF AWARE OF IT BECAUSE WHAT YOU DON'T EVER WANTIS TO BE CAUGHT OR FOR THOSE INDIVIDUALS TO BE CAUGHT ABOUTSOMETHING THAT THEY HAVE HEARD THROUGH SOMEBODY, AND IT IS ALWAYSGREAT WHEN WE ARE ABLE TO SAY, "YES. WE'RE AWARE OF THAT, AND WEARE UNDERGOING A STUDY IN RELATIONSHIP TO IT."I MUST HAVE HEARD THIS ON AT LEAST TWO OR 3 OTHER--OR SEEN IT ON TWOOR 3 OTHER SLIDES. PREVENTION IS REALLY THE BEST SOLUTION, WHICH187THEN GETS TO ANOTHER THING THAT YOU HAVE BEEN HEARING OVER AND OVERAGAIN.I HAVE BEEN TALKING ABOUT THESE COMMON THEMES, THIS CULTURE OFHONESTY. TO US, IT IS FANTASTIC WHEN YOU HAVE STUDENTS THAT END UPSAYING THINGS LIKE, "BOY, THIS LOOKS JUST LIKE THE QUESTIONS WE HADTWO WEEKS AGO," OR WHEN YOU HAVE TEACHERS THAT WILL SAY, "I THINKTHE ASSISTANT PRINCIPAL HAS BEEN DOING CERTAIN THINGS THAT HE ORSHE SHOULD NOT BE DOING," OR WHEN YOU HAVE A PRINCIPAL REPORTING ONSOMETHING THAT'S HAPPENING.IT IS ONLY WHEN THOSE KINDS OF SYSTEMS ARE IN PLACE WHERE THERE ISNO RETALIATION AND THERE IS NO PUNISHMENT FOR IT THAT I THINK ITREALLY WORKS WELL, AND WE HAVE BEEN WORKING VERY HARD.SOMETIMES I HAVE TO END UP SAYING TO FOLKS THAT WE'VE MAYBE DONESUCH A JOB OF MAKING FOLKS SO AWARE OF THE RIGIDITY OF THE SYSTEMTHAT THEY ARE AFRAID TO TAKE ANY TINY DEVIATION WHICH THEYTECHNICALLY COULD, BUT THEY WILL CALL TO ASK IF IT IS OK, AND THEYWILL APOLOGIZE, AND OUR RESPONSE IS ALWAYS, "NO. WE WOULD MUCHRATHER YOU CALL TO CHECK IF IT IS OK THAN TO HAVE GONE AHEAD ANDTHEN FIND OUT AFTERWARDS THAT WAS NOT THE RIGHT THING TO DO."188SO THAT WHOLE ISSUE OF PREVENTION, THAT WHOLE ISSUE OF THE CULTUREOF HONESTY, I THINK, IS WHAT REALLY MAKES THE WHOLE SYSTEM WORK,AND IT'S NOT EASY. IT TAKES TIME.WE DON'T KNOW ALL THE EXACT RIGHT THINGS TO DO, BUT MY TIME IS UP,SO I WILL STOP.JACK BUCKLEY: HA HA! THANKS VERY MUCH.ALL RIGHT. SO I THINK THERE'S CERTAINLY A FEW THEMES THAT I'MHEARING AGAIN THAT I'D LIKE TO ASK THE PANEL MAYBE TO ELABORATE ON.THE FIRST ONE, I'M NOT SURE IF THERE'S CONSENSUS ON, BUT, BOB, YOUCAME DOWN PRETTY HEAVILY IN FAVOR OF THE NEED TO BRING IN OUTSIDEPERSONNEL, AND OTHERS TOUCHED ON THE FACT THAT CERTAINLY AT THESCHOOL BUILDING LEVEL AND PERHAPS ALSO AT THE L.E.A. LEVEL, EXCEPTFOR OUR LARGEST AND MOST SOPHISTICATED ORGANIZATIONS, WE PROBABLYDON'T HAVE THE PERSONNEL RIGHT THERE ON THE SCENE WHO CAN ACTUALLYCONDUCT AN INVESTIGATION DISPASSIONATELY.IS THAT FAIR TO CALL THAT CONSENSUS, OR DO WE THINK AT LEAST MAYBEFOR SOME TYPES OF INTEGRITY VIOLATIONS, THE LOCAL FOLKS CAN HANDLEIT, OR DO WE NEED TO BRING IN SOMEBODY ELSE?189STEVE FERRARA: I DON'T FULLY AGREE WITH THAT, I THINK. I MADE THEPOINT THAT EDUCATORS ARE NOT TRAINED TO BE INVESTIGATORS, AND, INFACT, THERE ARE REASONS WHY YOU DON'T WANT TO PUT A SCHOOL LEADEROR MANAGER IN THE POSITION OF HAVING TO INVESTIGATE ONE OF THEIRSTAFF. I THINK YOU NEED PEOPLE WHO ARE TRAINED IN CONDUCTINGINVESTIGATIONS.MAYBE THEY ARE AN EMPLOYEE OF THE SCHOOL SYSTEM. MAYBE THEY'RESHARED ACROSS LOTS OF LARGE SCHOOL SYSTEMS. MAYBE IT IS A STATEPERSON. THOSE ARE THINGS THAT NEED TO BE WORKED OUT, AND THERE AREPOLICY CONSIDERATIONS, BUT I DON'T THINK IT HAS TO BE AN OUTSIDEAGENCY.BOB WILSON: I DON'T THINK IT HAS TO BE AN OUTSIDE AGENCY, AND I THINKTHERE ARE SOME THINGS THAT COME UP THAT I MIGHT DISAGREE WITH STEVEJUST A LITTLE BIT THAT THE PRINCIPAL CAN TAKE A PRELIMINARY LOOKAT, BUT IF IT'S GOING TO BE SOMETHING WHERE IT LOOKS LIKE THERE MAYBE SOMETHING HERE, I THINK YOU NEED SOMEBODY FROM THE OUTSIDE OFTHAT SCHOOL AND OUTSIDE OF THAT PARTICULAR ADMINISTRATION OF THATSCHOOL HANDLING THAT INVESTIGATION, AND IT MAY BE YOUR LEGALCOUNSEL WHO DOES IT, BUT THERE ARE A LOT OF TRICKS TO THE TRADE TOPROPER INVESTIGATIONS, AND THAT IS JUST NOT WHAT EDUCATORS ARETAUGHT TO DO.190EDUCATORS BY NATURE ARE COMPASSIONATE PEOPLE. INVESTIGATORS BYNATURE ARE...DIFFERENT. ARE DIFFERENT. IT IS A QUESTION-AND-ANSWERPROCESS THAT TAKES A LOT OF PATIENCE, AND IT TAKES A LOT OF TIME TODO IT RIGHT.LIKE I SAID, OUT OF ALL THOSE INTERVIEWS, WE DID NOT HAVE A SINGLECONFESSION ON THE FIRST INTERVIEW, AND THOSE WERE BEING DONE BYSTATE INVESTIGATORS, OUR GEORGIA BUREAU OF INVESTIGATION. ITWASN'T UNTIL WE UPPED THE HEAT, CHANGED THE ATMOSPHERE, GOT THEMOUT OF THE SCHOOLS, GOT THEM INTO LAW OFFICES, GOT THEM SURROUNDEDBY SOME LAWYERS AND SOME OF THOSE STATE AGENTS AND BEGIN TO ASKTHEM QUESTIONS THAT TRIANGULATED, AS TISHA SAID, SOMETIMES MORETHAN THAT, POINTS OF REFERENCE THAT THEY COULD NOT EXPLAIN THAT YOUBEGAN TO GET SOMEWHERE.THAT'S JUST NOT WHAT EDUCATORS ARE UP TO. I THINK IT SORT OFDEPENDS. DOES IT APPEAR TO BE SOME ISOLATED SITUATION IN THECLASSROOM THAT DOESN'T APPEAR TO BE TOO SERIOUS? FINE. YOU MIGHT BEABLE TO HANDLE IT IN HOUSE, BUT ONCE IT LOOKS LIKE HONEST-TO-GODCHEATING OF A RATHER SIGNIFICANT NATURE AND MAY INVOLVE MORE THANONE CLASS, I THINK YOU DEFINITELY GOT TO GO OUTSIDE. DOESN'T HAVETO BE ANOTHER AGENCY. IT CAN BE A LAW FIRM. IT CAN BE PRIVATEINVESTIGATORS THAT WORK FOR THE SYSTEM, YOU KNOW, OR GET HIRED,CONTRACT EMPLOYEES, SOMETHING LIKE THAT.191JACK BUCKLEY: TISHA, WHAT DID YOU GUYS USE?TISHA EDWARDS: I AGREE. WE DID NOT ALLOW ANY OF THE SCHOOL-BASEDSTAFF TO PARTICIPATE IN ANY OF OUR INVESTIGATIONS. WE DO HAVE ANINVESTIGATIONS UNIT IN OUR ACHIEVEMENT AND ACCOUNTABILITY OFFICE,AND WHEN WE RECEIVE ALLEGATIONS, PARTICULARLY THOSE THAT ARE VERYSUBSTANTIAL IN NATURE, WE DO GO OUT AND CONTRACT WITH INDIVIDUALSWHO HAVE EXPERTISE IN LAW ENFORCEMENT TO PARTNER WITH US AND TOBUILD THE CAPACITY OF OUR ACHIEVEMENT AND ACCOUNTABILITY OFFICE,BUT WE HAVE NEVER ALLOWED ANY SCHOOL-BASED STAFF TO PARTICIPATE INTHE INVESTIGATING OF ANY TESTING ALLEGATION.STEVE FERRARA: BY THE WAY, SORT OF A SIDE COMMENT, I'M THINKING 5 OR10 YEARS DOWN THE ROAD, AND WE HAVE ALL THESE PROCEDURES IN PLACEFOR CONDUCTING INVESTIGATIONS AND GETTING CONFESSIONS FROM PEOPLELIKE HAPPENED IN ATLANTA. IF YOU READ THE SUNDAY "NEW YORK TIMES,"THERE WAS A VERY INTERESTING STORY ABOUT THE FREQUENCY AND THE SORTOF PSYCHOLOGICAL EXPLANATIONS FOR PEOPLE MAKING FALSE CONFESSIONSIN LEGAL SITUATIONS. SO WE NEED TO MAYBE KEEP HALF AN EYE OUT FORTHAT IN THE FUTURE.JACK BUCKLEY: I THINK, TISHA, YOU MENTIONED MARYLAND'S STATUTES FORCONDUCTING TESTING INVESTIGATIONS, AND, LOU, CERTAINLY YOU WERE192DESCRIBING SOME, SOUNDS TO BE SOME PRETTY DETAILED REGULATIONS.WHAT KIND OF DETAIL IS REALLY IN THERE, AND IS THERE A MODEL THATWE CAN POINT TO ACROSS STATES OR DISTRICT LEVEL FOR SORT OFCODIFYING THIS?TISHA EDWARDS: I WOULDN'T BE AS BOLD TO SAY THAT MARYLAND SHOULD BEHELD OUT TO NECESSARILY BE THE STANDARD FOR EVERYONE ELSE. I THINKTHAT THE PARTNERSHIP THAT OUR SUPERINTENDENT, ANDRES ALONSO, WHOIS A VERY BOLD LEADER, REALLY REQUIRED THAT WE PARTNER WITH THESTATE IN ORDER TO FIGURE OUT HOW TO DO THE RIGHT THING.WE ARE THE ONLY L.E.A. IN THE STATE WHO HAD EVER HAD A PUBLIC PRESSCONFERENCE AND WHO HAD GONE OUT THERE PUBLICLY TO OWN THE FACT THATTESTING IRREGULARITIES HAD HAPPENED IN OUR SCHOOL SYSTEM, ANDACTUALLY IN THE STATE, WE HAD ACTUALLY RECEIVED ADVICE AND COUNSELTHAT, EVEN THOUGH THERE WERE OTHER JURISDICTIONS WHERE THERE WEREKNOWN TESTING IRREGULARITIES, THAT THERE HAD BEEN A DECISION MADETHAT THAT INFORMATION WOULD NOT BE GIVEN OUT TO THE PUBLIC.SO WE WERE SETTING A NEW STANDARD AROUND TRANSPARENCY, AND ONE OFTHE OTHER THINGS I WILL SAY IS THAT ALTHOUGH THE STATE HADREGULATIONS AROUND ANY L.E.A.'S ABILITY TO REVOKE A LICENSE, MOSTL.E.A.S RESISTED DOING THAT WHEN THEY COULD NOT ESTABLISHCULPABILITY, EVEN IF THERE WERE SIGNS OF CHEATING, AND ANDRES, WHO193IS ALSO AN ATTORNEY IN HIS FORMER LIFE, WAS VERY BOLD ABOUT THEFACT THAT WE WERE GOING TO USE THE REASONABLE MAN STANDARD, AND HESAID THE OLD SAYING, WHAT, "IF IT QUACK LIKE A DUCK AND WALKS LIKEA DUCK, IT'S A DUCK, AND WE'RE GOING TO HAVE ACCOUNTABILITY FOR THEDUCKS."SO HE WAS OUT THERE SAYING TO SCHOOL LEADERS AND TO TEACHERS THAT-EVEN THOUGH WITHIN THE INVESTIGATIVE PROCESS, HE COULD NOTDETERMINE SPECIFICALLY WHO HAD DONE WHAT--WE WERE ALL CLEAR THATSOMETHING HAD HAPPENED, AND IT HAS BEEN A HUGE STRETCH FROM ALITIGATION PERSPECTIVE FOR US AND THE STATE TO BE ABLE TO MAINTAINTHE REASONABLE MAN STANDARD WHEN PURSUING SOMEONE'S CERTIFICATE.SO I DON'T WANT TO GO AS FAR AS TO SAY THAT MARYLAND SHOULD BEESTABLISHING THE STANDARD, BUT I DO THINK THAT WE ARE TREADING INGROUNDS THAT WE HAVE NEVER DONE BEFORE, AND BALTIMORE IS KIND OFPUSHING THIS ISSUE OF TRANSPARENCY AND VIGILANCE AROUND THESETYPES OF ISSUES.LOU FABRIZIO: THE COMMENTS THAT TISHA JUST MADE REMIND ME OFINSTANCES WHERE WE HAVE DONE INVESTIGATIONS AND THEN WHEN WEPRESENT THE SO-CALLED EVIDENCE TO THE ATTORNEYS, INCLUDING ANSWERSHEETS OR TEST BOOKLETS WHERE STUDENTS WITH DISABILITIES HADMARKED IN THE TEST BOOKLET AND THEN THOSE RESPONSES NEEDED TO BE194TRANSFERRED TO THE ANSWER SHEET, DETERMINING THAT THEY DIDN'TMATCH-UP AND SOMETHING CHANGED SOMEWHERE, AND THEN YOU GET INTOTHAT WHOLE ISSUE OF DID ANYBODY SEE THE PERSON DO IT?AND THAT'S A FRUSTRATION THAT I THINK WE AT THE STATE LEVELSOMETIMES HAVE, IS THAT WE WILL BE TOLD, YOU KNOW, "WE'VE GOTBIGGER FISH TO FRY, AND WE NEED TO WORRY ABOUT THAT, AND WE'RE NOTGOING TO WIN THIS ONE," AND THAT'S A FRUSTRATION. TO ANSWER YOURQUESTION, JACK, I'M NOT SAYING WE'VE GOT THE BEST SET OF THINGS,BUT WE'D BE MORE THAN WILLING TO SHARE THEM WITH THE DEPARTMENT. IDID JUST LOOK AT THE LIST OF IRREGULARITIES THAT WE MONITOR, ANDTHERE ARE 60 OF THEM.JACK BUCKLEY: ALL RIGHT. WELL, AT THAT, I THINK WE CAN THROW IT OPENTO THE BROADER PANEL IN THE AUDIENCE.WAYNE?WAYNE CAMARA: I WANT TO COME BACK TO THE ISSUE OF THE DETECTION WHENSTAFF AT A SCHOOL OR A PARENT OR A STUDENT MAY THINK THAT THERE ISSOME IRREGULARITY. THE OTHER PANEL TALKED A LITTLE BIT ABOUT THETIP HOTLINE, BUT IT ALSO SOUNDS LIKE THERE ARE MANY INSTANCES WHEREA DISTRICT, AN L.E.A., THE PRIME WAY OF REPORTING THESE INSTANCES195IS GOING WITHIN THE SCHOOL BUILDING, AND, I MEAN, I QUESTIONWHETHER THAT IS THE BEST VEHICLE.IF I'M A COLLEAGUE OF SOMEONE WORKING IN A SCHOOL, IF I'M A PARENT-MY WIFE IS A FIRST GRADE TEACHER, AND SHE INSTILLED INTO ME THATWE'VE HAD KIDS IN SCHOOL, TOO, AND AS A PARENT OR AS A COLLEAGUE,YOU ARE A LITTLE BIT RELUCTANT TO BLOW THE WHISTLE ON A SCHOOL OR ATEACHER. YOU ARE WORRIED ABOUT WHAT TEACHER YOUR CHILDREN ARE GOINGTO BE PLACED WITH NEXT YEAR, AND YOU ALSO ARE NOT SURE WHATHAPPENED, AND SO WE HAVE A LOT OF SUCCESS WITH AN ANONYMOUS TIPHOTLINE.WE GET A LOT OF FALSE POSITIVES, A LOT OF FALSE POSITIVES, BUT ITALLOWS ME, AS ANOTHER TEACHER OR AS A PARENT, TO NOT FEEL THATTHERE'S GOING TO BE ANY RETALIATION, AND I CAN REPORT SOMETHINGWHEN I AM CLEARLY LESS THAN 100% CERTAINTY, AND I AM WONDERING TOWHAT EXTENT YOU FEEL IT IS IMPORTANT TO GO THROUGH THE SCHOOLBUILDING ROUTE, OR IS IT MORE ADVISABLE TO HAVE SOME ENTITY,WHETHER IT IS THE STATE OR WHETHER IT'S SOMEBODY ELSE. I'M JUSTWONDERING, CONTRASTING THE BUILDING SOLUTION VERSUS AN OUTSIDESOLUTION.TISHA EDWARDS: I WOULD REITERATE WHAT BOB SAID IN HIS PRESENTATIONABOUT YOU NEED TO FOLLOW UP ON GOSSIP. A LOT OF TIMES, PEOPLE DON'T196KNOW THAT THEY ARE REPORTING SOMETHING, AND THEY ACTUALLY ARE,THROUGH CASUAL CONVERSATIONS WITH FOLKS, WHETHER IT IS AT A PTAMEETING OR A PARENT-TEACHER CONFERENCE.WE HAD LOTS OF SITUATIONS--SHOULDN'T SAY LOTS. THAT'S ANEXAGGERATION. WE HAVE HAD SITUATIONS WHERE PARENTS HAVE BEENSITTING AT A SPECIAL ED. IEP MEETING AND HAVE SAID, "I DON'T REALLYUNDERSTAND THESE TEST SCORES BECAUSE I KNOW THAT MY SON OR MYDAUGHTER IS NOT PERFORMING AT THESE LEVELS, RIGHT? THIS JUSTDOESN'T MAKE SENSE TO ME," AND SAY THAT IN AN OPEN FORMAT WHEREPEOPLE ARE TAKING MINUTES, AND THAT INFORMATION IS FILTERED UP TOTHE SCHOOL SYSTEM THROUGH OUR INVESTIGATIVE OFFICE TO SAY THATTHAT'S NOT WHERE A PARENT IS ALLEGING CHEATING PER SE, BUT A PARENTIS SAYING, "THE DATA ABOUT MY CHILD DOESN'T MAKE SENSE TO ME," ANDSO I THINK THAT I DO AGREE THAT IT IS NATURAL FOR PEOPLE TO HAVECONCERNS ABOUT BRINGING INFORMATION FORWARD ABOUT PEOPLE THAT THEYARE IN A RELATIONSHIP WITH, WHETHER THAT'S A WORK RELATIONSHIP ORSOME OTHER KIND OF PERSONAL RELATIONSHIP.I THINK WHAT WE WOULD ADVISE SCHOOL SYSTEMS TO DO IS HAVE AS MANYVEHICLES AS POSSIBLE AND LET THE INFORMATION FLOW INTO THEORGANIZATION THROUGH A VARIETY OF MECHANISMS, BUT MAKE SURE THATWHEREVER THAT INFORMATION IS COMING IN TO, THAT THAT INFORMATION ISLOGGED, THAT INFORMATION IS TRACKED, AND THAT THERE'S AN197ACCOUNTABILITY SYSTEM FOR FOLLOW-UP, BUT, AS I SAID, THE WOMAN WHOHELPED TO RAISE OUR AWARENESS HAD NO PROBLEMS COMING INTO A SCHOOLBOARD MEETING, AND SHE DIDN'T CARE WHO WAS GOING TO BE TEACHING HERCHILD THE NEXT YEAR. SHE FELT LIKE THIS WAS A GRAVE INJUSTICE THATNEEDED PUBLIC ACCOUNTABILITY.SO WE DON'T MAKE A VALUE JUDGMENT ABOUT HOW THE INFORMATION COMESIN. OUR OBLIGATION IS JUST TO FOLLOW UP ON ALL INFORMATION.BOB WILSON: I RECOGNIZE VERY CLEARLY THE DANGERS, AS ANYONE WOULD,TO ANONYMOUS TIPS. YOU CAN HAVE YOUR MORTAL ENEMY CALLING IN JUSTTO TRY TO CAUSE YOU PROBLEMS, BUT NOT EVERYBODY IS GOING TO HAVETHE COURAGE THAT IT MIGHT TAKE TO STAND UP AND PUBLICLY PROCLAIM TOTHE PRINCIPAL OR THE SUPERINTENDENT OR BOARD OF EDUCATION THAT, "ITHINK THIS PROBLEM EXISTS." YET THEIR CONCERNS ARE ACTUALLY VALID,AND THEY NEED TO HAVE AN OUTLET AND A WAY TO EXPRESS THEM. IT MAY BEBECAUSE OF PERSONAL RELATIONSHIPS. "THE TEACHER I WANT TO COMPLAINABOUT NOT ONLY TEACHES MY DAUGHTER, BUT LIVES TWO HOUSES DOWN FROMME, AND MY HUSBAND PLAYS GOLF WITH HER HUSBAND, BUT SOMEBODY NEEDSTO LOOK AT THIS."SO I THINK IF YOU DON'T HAVE THOSE OUTLETS, THAT DOESN'T MEAN YOUDON'T HAVE A SIFTER AND YOU DON'T TRY TO PROFESSIONALLY ASSESS THISINFORMATION, BUT I THINK YOU HAVE TO HAVE A WAY THROUGH TIP LINES198AND SO FORTH FOR PEOPLE TO BE ABLE TO CALL AND PROVIDE INFORMATION,EVEN ANONYMOUSLY. I THINK IT'S A MISTAKE IF YOU DON'T.TISHA EDWARDS: I WOULD ALSO ARGUE THAT IT IS GOOD PRACTICE FOR THEORGANIZATION TO BE TRANSPARENT WITH ITS EMPLOYEES THAT NOT ONLY AREWE WAITING ON SOMEONE TO GIVE US A TIP, BUT WE'RE DOING STATISTICALANALYSIS ON THE BACK END TO FIND THOSE INCONSISTENCIES AND THOSEANOMALIES.FOLKS IN OUR SCHOOL SYSTEM KNOW, OF COURSE, WHEN PEOPLE BRINGINFORMATION FORWARD, WE WELCOME THAT, BUT THEY ALSO KNOW THAT WEHAVE A SET OF RESOURCES THAT ARE LOOKING FOR DATA, LOOKING AT DATA,AND WILL QUESTION THAT DATA IF THERE ARE THOSE ANOMALIES THATEXIST, WHICH HAS BEEN THE CASE FOR US.EVEN WHEN THAT SCHOOL COMMUNITY IS VERY TIGHT-LIPPED AND IT ISIMPOSSIBLE TO GET PEOPLE TO TALK TO YOU ABOUT WHAT'S HAPPENING, WEHAVE PARTICULAR FLAGS OR ALERTS AT THE STATE LEVEL AND AT THESCHOOL SYSTEM LEVEL WHICH WE USE TO HELP US TO THINK THROUGH WHERETHERE MIGHT BE POSSIBILITIES OF INCONSISTENCIES ANDIRREGULARITIES.BOB WILSON: LET ME THROW ONE OTHER THING IN THIS. GO BACK TO THATSTATEMENT THAT'S BEEN MADE A NUMBER OF TIMES, AND I DO BELIEVE IT199STARTS WITH THE LEADERSHIP OF THE SYSTEM, THE YOU MUST PROMOTE ACULTURE OF INTEGRITY IN THAT SYSTEM BECAUSE PEOPLE WON'T EVEN CALLIN WITH ANONYMOUS TIPS IF THEY BELIEVE NOTHING WILL HAPPEN TO IT.WE HAD A SCHOOL IN OUR A.P.S. SYSTEM THAT THE FOURTH GRADE ONE YEARWAS 813TH IN ITS MATH SCORES OUT OF 1,200 SCHOOLS IN THE STATE. THENEXT YEAR, IT WAS NUMBER ONE. NOW, SOME OF US ARE OLD ENOUGH TOREMEMBER THE METS' LAST-TO-FIRST, BUT THEY ONLY CLIMBED OVER 7TEAMS. THIS GROUP WENT FROM 813TH TO FIRST, AND A.P.S., WHEN THENEWSPAPER QUESTIONED IT, SAID, "OH, NO. IT IS ALL DUE TO OURINCREDIBLE PROGRAMS AND TEACHING." WELL, IF THEY WERE THAT DARNGOOD, I WANT THEM FOR EVERYBODY ELSE.THAT SAME YEAR, ANOTHER SCHOOL THAT WAS IN THE LOWEST GROUP THEYEAR BEFORE IN THIRD GRADE IN MATH, THOSE SAME STUDENTS NOW IN THEFOURTH GRADE WERE FOURTH IN THE STATE. A.P.S. DEFENDED IT.THE MESSAGE IS, YOU CAN HIT ME ALL YOU WANT WITH FACTS AND TRUTH,BUT WE ARE DOING IT RIGHT, AND WE ARE GOING TO IGNORE THE OBVIOUS,THAT SOMETHING IS WRONG. PEOPLE WON'T EVEN CALL IN WITH ANONYMOUSCALLS THEN BECAUSE THEY WON'T DO ANYTHING ABOUT IT. IF YOU FOSTERTHAT CULTURE AND YOU ACTUALLY DO SOMETHING ABOUT IT, AS WE'REHEARING ABOUT IN BALTIMORE, I THINK YOU WILL GET PEOPLE MOREWILLING TO TAKE WHATEVER AVENUES ARE OUT THERE--WHETHER IT'S200WALKING INTO THE PRINCIPAL'S OFFICE, SPEAKING AT A BOARD MEETING,WRITING THE SUPERINTENDENT, OR JUST GETTING ENOUGH COURAGE TO CALLTHE ANONYMOUS LINE--YOU WILL GET MORE INFORMATION ABOUT YOURSYSTEM.JACK BUCKLEY: I THINK WE'VE GOT A QUESTION FROM THE ONLINE AUDIENCE.JAMES ELIAS: WE DO. SO THIS WAS ACTUALLY PROMPTED BY STEVE FERRARA'SDESCRIPTION OF GROUPS OF PEOPLE. THIS QUESTIONER WANTS TO KNOW, "ISIT ETHICAL TO WORK COUNTER TO A SYSTEM THAT HAS LOST CREDIBILITY INTERMS OF ITS GOALS?"STEVE FERRARA: I'M SORRY. COULD YOU REPEAT THAT? I'M GLAD THAT'S YOURQUESTION.JAMES ELIAS: BEYOND YOUR GROUP OF PEOPLE, IS IT ETHICAL TO WORKCOUNTER TO A SYSTEM THAT HAS LOST CREDIBILITY IN TERMS OF ITSGOALS?STEVE FERRARA: YEAH. YOU KNOW, I THINK WE NEED TO WRITE TO THEETHICIST COLUMN IN THE "NEW YORK TIMES" MAGAZINE. I DON'T THINKTHAT IS THE KIND OF QUESTION I SHOULD DISCUSS IN A FORUM LIKE THIS.201JAMES ELIAS: SEVERAL PEOPLE FROM STATE DEPARTMENTS OF EDUCATION HAVESAID THAT IF YOU RECEIVED AN INVESTIGATION RESPONSE OR TIP THAT ISVAGUE OR DOES NOT ADDRESS A VIOLATION, WHAT IS THE APPROPRIATEFOLLOW-UP ACTION?JACK BUCKLEY: I HEARD IT TO BE, FOLKS FROM S.E.A.S ARE WONDERING, IFWE RECEIVE A TIP THAT IS VAGUE AND DOESN'T SORT OF PROVIDE ENOUGHINFORMATION, WHAT IS THE APPROPRIATE FOLLOW-UP ACTION?SO WHAT DO WE DO WITH A VAGUE TIP?LOU FABRIZIO: THAT I CAN SPEAK TO.JACK BUCKLEY: SURE. PLEASE.LOU FABRIZIO: ANYTIME WE'VE EVER GOTTEN ANY KIND OF ANONYMOUS TIP ORSOME LETTER WITH NO RETURN ADDRESS THAT SAYS SOMETHING IS HAPPENINGIN SUCH AND SUCH A SCHOOL SYSTEM, WE IMMEDIATELY CALL THE CENTRALOFFICE TEST COORDINATOR OF THAT SCHOOL DISTRICT AND SAY, "HEY,LOOK. WE'VE JUST GOTTEN THIS LETTER. WE DON'T KNOW IF IT MEANSANYTHING, BUT WE NEED FOR YOU TO CHECK AROUND, SEE IF THERE ISSOMETHING GOING ON OR IF YOU ARE HEARING ANYTHING," AND THAT KINDOF THING, AND THEN WE LET THEM GO AND FOLLOW IT.202STEVE FERRARA: YEAH. BOB SAID IT. YOU'VE HEARD IT A COUPLE OF TIMESNOW. I THINK YOU HAVE TO FOLLOW UP ON EVERYTHING.BOB WILSON: BUT IF YOUR QUESTION IS, CAN IT BE SO VAGUE THAT YOUCAN'T HARDLY FOLLOW UP, CERTAINLY IT CAN BE. YOU DO THE BEST YOUCAN. THAT IS ALL YOU CAN DO. IF YOU HAVE NO RETURN ADDRESS, NORETURN PHONE NUMBER, NO NAME, AND IT IS SO AMBIGUOUS SO YOU PASS ITAROUND AMONG A BUNCH OF OTHER FOLKS TO BE SURE YOU ARE NOT THE ONLYIDIOT IN THE ROOM WHO CAN'T UNDERSTAND IT AND EVERYBODY SAYS, "IDON'T KNOW WHAT THAT MEANS. I DON'T KNOW WHAT YOU CAN DO. WELL,WHAT CAN WE DO WITH IT?" THEN I THINK YOU'VE PROBABLY DONE ALL YOUCAN. YOU TRIED, BUT SOMETIMES THERE WILL BE ENOUGH OF A CRACK INTHERE THAT YOU CAN FIGURE OUT, JUST LIKE YOU SAID, IT JUST SAYSSOMETHING IS GOING ON. THAT IS WHAT LOU SAYS.WELL, OK. YOU AT LEAST NEED TO TALK TO THE DIRECTOR OF THATDIVISION OR WHATEVER AND SAY, "LOOK. WE GOT THIS LETTER, AND IDON'T KNOW WHAT IS SUPPOSED TO BE GOING ON, BUT DO YOU HAVE SOMEREASON?" AND THEY MAY SAY, "WELL, WE HAVE HAD A LITTLE PROBLEMRECENTLY THAT I HAVEN'T TOLD ANYBODY ABOUT."WELL, MAYBE THAT'S IT. YOU NEVER KNOW. SO YOU JUST DO THE BEST YOUCAN. I THINK THAT'S ALL YOU CAN DO. I DON'T THINK YOU IGNORE IT.YOU JUST DO THE BEST YOU CAN.203JACK BUCKLEY: TONY, DO YOU HAVE A QUESTION FROM BEFORE?TONY ALPERT: YEAH. LOU, YOU MIGHT BE ABLE TO TALK A LITTLE BIT ABOUTTHIS. THE DIFFERENT STRATEGIES THAT YOU MIGHT NEED TO IMPLEMENT FORA SMALL, RURAL SCHOOL VERSUS A LARGE URBAN.LOU FABRIZIO: WELL, IF YOU ARE TALKING ABOUT THINGS LIKE HAVINGPROCTORS AND--TONY ALPERT: NO, IN TERMS OF INVESTIGATING IMPROPRIETIES IN WHAT ISOTHERWISE A SMALL, CLOSE-KNIT COMMUNITY.LOU FABRIZIO: WELL, I DON'T THINK WE HAVE AS SMALL A SET OF SCHOOLSAS YOU MAY HAVE IN OREGON, BUT WE ALWAYS HAVE THE ASSISTANCE THATTHE STATE WOULD PROVIDE. OUR SMALLEST SCHOOL DISTRICT IN OUR STATEIS, LIKE, 700 STUDENTS, SO--TISHA EDWARDS: BUT I THINK ONE OF THE THINGS THAT WAS ALLUDED TO ISONE OF THE THINGS YOU MIGHT CONSIDER, IS PARTNERING WITH ANOTHERSCHOOL DISTRICT IN THE STATE AND HAVING SOME TYPE OF RECIPROCALRELATIONSHIP AROUND. YOU COULD HAVE MULTIPLE SMALL L.E.A.S FORMINGA CONSORTIUM OR CLUSTER OF SORTS THAT, WHEN ISSUES ARRIVE, THERE IS204A RELATIONSHIP THAT ALLOWS YOU TO DO INVESTIGATIONS OF ONEANOTHER'S SCHOOLS, IF NECESSARY.JACK BUCKLEY: MAKES SENSE. JOHN?JOHN FREMER: NOT A QUESTION, BUT AN OBSERVATION THAT GOES BACK TO TIPLINES. SO I HAVE BEEN TALKING TO AGENCIES THAT ARE RESPONSIBLE FORTESTS--STATES, DISTRICTS, BUT OTHER PLACES, TOO--AND I HAVEPROBABLY HAD 50 DIFFERENT CONVERSATIONS WITH THE PERSONSRESPONSIBLE FOR THE PROGRAM IN CONVERSATIONS, THEIR REVIEWS OFTHEIR PROCESSES AND RECOMMENDATIONS, AND WE ALWAYS RECOMMEND A TIPLINE IF THEY DON'T HAVE A TIP LINE, AND THEN WE GET FOLLOW-UP LATERON, AND IF THEY HAVE A TIP LINE, THEY SHARE THEIR EXPERIENCES, ANDTHE THING THAT WAYNE CAMARA SAID ABOUT GETTING A LOT OF FALSEPOSITIVES, IN GENERAL, THAT IS NOT WHAT IS HAPPENING.PLACES THAT HAVE HAD TIP LINES ALL ALONG OR HAVE INTRODUCED THEM,AT LEAST THEY ARE NOT GETTING CRANK INQUIRIES. THEY ARE GETTINGINQUIRIES THAT, UPON INVESTIGATION, DON'T LEAD TO A CONCLUSION OFCHEATING, BUT ALMOST ALWAYS WHEN SOMEONE TAKES THE TROUBLE TO MAKEA TIP--WHETHER IT'S BY E-MAIL OR PHONE, WHATEVER--THE E-MAIL YOUGIVE YOURSELF AWAY, BUT BY PHONE IT IS WORTH GETTING, AND SO I'M ASTRONG ADVOCATE OF TIP LINES. I DON'T THINK YOU HAVE TO WORRY ABOUTWASTING A LOT OF YOUR TIME IF YOU INTRODUCE THEM.205STEVE FERRARA: I HAVE A NEW TOPIC. DAVE IN THE EARLIER PANEL MADE THEPOINT THAT MUCH OF WHAT WE DO AROUND SECURITY, INTEGRITY,DETECTION, AND SO FORTH IS IN ITS INFANCY, AND I AGREE WITH THAT INLARGE MEASURE. I HAVE TRIED TO MAKE THE POINT THAT THE DEVELOPMENTOF OUR PROCEDURES IN INVESTIGATIONS, I WOULD SAY, THAT IS VERY MUCHIN ITS INFANCY, OR MAYBE WE HAVE NOT EVEN GOTTEN TO BIRTH YET.THERE IS ANOTHER POINT RELATED TO THIS NOTION OF WE ARE IN OURINFANCY. SO WE HAVE THESE STATISTICAL DETECTION PROCEDURES FORMULTIPLE CHOICE ITEMS.NOW WE HAVE THE COMMON CORE STATE STANDARDS, THE DESIGNS OF PARCCAND SMARTER BALANCED CONSORTIUM, AND WE ARE NOW GOING TO SEE MORECONSTRUCTED RESPONSE ITEMS AND PERFORMANCE TASKS. I'M SO HAPPY TOSEE ALL THAT STUFF. I WORK IN THE CENTER FOR PERFORMANCEASSESSMENT. I DID THAT IN THE NINETIES. I'M GLAD THAT IT IS BACK.SO WE REALLY NEED TO START THINKING ABOUT WHAT ARE THE DETECTIONPROCEDURES WE ARE GOING TO USE FOR THE VARIOUS FORMS OF ANOMALIES,CHEATING AND OTHER ANOMALIES THAT CAN GO ON IN PERFORMANCEASSESSMENTS, AND THEN WHAT DO WE DO ABOUT INVESTIGATION.SO ONE AREA FOR CONSIDERATION IS--SOMEONE REFERRED THIS MORNING TOTHESE ONLINE PROGRAMS SO TEACHERS CAN CHECK TO SEE IF KIDS AREPLAGIARIZING WHEN THEY TURN IN WRITTEN ASSIGNMENTS. THERE MAY BE A206WAY TO ADAPT THOSE PROGRAMS TO LOOK FOR THINGS LIKE THE STORY THATI TOLD WHERE MULTIPLE KIDS IN THE SAME TESTING GROUP HAVE HIGHLYSIMILAR RESPONSES TO THEIR PERFORMANCE TASKS OR ESSAYS,CONSTRUCTED RESPONSE ITEMS, AND SO FORTH. SO I THINK WE REALLY NEEDTO START DEVELOPMENTAL WORK IN THAT DIRECTION.BOB WILSON: STEVE, THE REASON, IN MY OPINION, THAT WE'RE SOMEWHAT INTHE INFANCY IN INVESTIGATIVE TECHNIQUES WITH SCHOOLS, I'M NOTGOING TO SAY TEACHERS DIDN'T CHEAT BEFORE 15 YEARS AGO, BUT UPUNTIL ABOUT 15 YEARS AGO WHEN WE WERE TALKING ABOUT CHEATING INSCHOOL, WE WERE ALMOST ALWAYS, ALMOST ALWAYS, TALKING ABOUTSTUDENTS. IN THE LAST 15 YEARS, THE EQUATION HAS CHANGED BECAUSETHE EQUATION ON THE TEST HAVE CHANGED. UP UNTIL ABOUT 15 YEARS AGO,THE TESTS WERE ALL ABOUT THE STUDENTS.NOW, I'M NOT GOING TO SIT HERE AND BASH NO CHILD LEFT BEHIND SIMPLYBECAUSE IT SHIFTED THE EQUATION. TEACHERS HAVE TO BE RESPONSIBLEAND HAVE TO BE ACCOUNTABLE, BUT THE SHIFT IS SO GREAT THAT NOW THETEST, IN THIS INVESTIGATOR'S HUMBLE OPINION, BECAME MORE ABOUT THETEACHERS AND THE SCHOOLS AND THE PRINCIPALS THAN IT DID ABOUT THESTUDENTS.SO YOU BROUGHT IN ANOTHER WHOLE UNIT OF FOLKS, AND NOW WE ARE NOTTALKING ABOUT ADULTS TO STUDENTS. WE ARE TALKING ABOUT ADULTS TO207ADULTS. THAT REQUIRES REAL INVESTIGATION, AND THE EQUATION HASCHANGED.JACK BUCKLEY: GOOD POINT. CORNELIA?CORNELIA ORR: IT HAS BEEN ALLUDED TO THROUGHOUT DISCUSSIONS ALL DAYTODAY ABOUT RESOURCES, AND I THINK IT IS A HUGE ISSUE FOR STATESTHAT ARE TRYING TO OPERATE TWO PROGRAMS--A STATE ASSESSMENT, MAYBEA CONSORTIUM-LED ASSESSMENT—TO THINK THEY HAVE ENOUGH RESOURCES,EITHER FISCAL RESOURCES TO CONTRACT OR STAFF RESOURCES, TO DO THIS.COULD SOME OF YOU ADDRESS WHAT YOU THINK IT TAKES TO BE REASONABLYVIGILANT ON THIS IN TERMS OF RESOURCES?BOB WILSON: PREVENTION IS CHEAPER.CORNELIA ORR: RIGHT. YEAH.BOB WILSON: I CAN TELL YOU THAT BECAUSE I HAVE BEEN ON THE OTHER ENDOF CAUSING THE BILL TO GO UP. SO THE MORE MONEY YOU CAN PUT INTOPREVENTION AND CREATING THAT CULTURE THAT SAYS, "WE ARE GOING TOPLAY BY THE RULES, AND WE ARE GOING TO DO IT RIGHT," IS A LOT LESSEXPENSIVE THAN AFTER THE FACT. IT IS ALWAYS MORE EXPENSIVE AFTERTHE FACT.208TISHA EDWARDS: RIGHT. I WOULD JUST REITERATE FROM BALTIMORE CITY'SPERSPECTIVE, IN THE LAST TWO YEARS, WE'VE MADE A SIZABLE INVESTMENTIN PREVENTION AND THIS ISSUE OF THE CULTURE, CHANGING THE CULTURE.I'M A FORMER PRINCIPAL, AND WHEN MEETING WITH OUR SUPERINTENDENTABOUT OUR STRATEGY AFTER THIS PARTICULAR INCIDENT--WHICH WAS VERY,VERY HARD FOR THE ORGANIZATION--ONE OF THE THINGS I SAID TO OURTEAM, OUR LEGAL TEAM, AND TO OUR SUPERINTENDENT IS, "WE SHOULD NOTBE TALKING ABOUT THIS FROM THE PERSPECTIVE OF CATCHING CHEATERS. ITHINK THAT THE ORGANIZATION IS NOT GOING TO FIND THAT A PALPABLEMISSION. WE SHOULD BE TALKING ABOUT PROTECTING EDUCATORS ANDPROTECTING CHILDREN," AND SO IT SHOULD BE THE FRAME AROUND TESTINGINTEGRITY, AND THE CONVERSATION IN OUR DISTRICT HAS SHIFTED FROM,"WE ARE LOOKING FOR THE CHEATER," TO, "WE ARE PROTECTING OUREDUCATORS, AND WE'RE PROTECTING THE RESULTS OF CHILDREN,"AND SO I THINK IT MAKES A HUGE DIFFERENCE ABOUT HOW PEOPLE EMBRACEPROCTORS. IT MAKES A HUGE DIFFERENCE IN TERMS OF THE WILLINGNESS OFTHE ORGANIZATION TO EVEN MAKE THE INVESTMENT WHEN IT IS NOT ABOUTGOING AFTER THE BAD PEOPLE BUT IT IS ACTUALLY ABOUT PROTECTINGPEOPLE WHO COME AND WORK HARD FOR CHILDREN EVERY DAY. SO THATCULTURAL SHIFT IN MAKING THAT ORGANIZATIONAL KIND OF PARADIGM209CHANGE WAS HUGE FOR OUR SCHOOL SYSTEM, AND SO WE DON'T TALK ABOUTIT.YES, WE HAVE CHEATING INCIDENTS, AND WE ARE GOING TO GO AFTER THOSEINDIVIDUALS, BUT WE BELIEVE THAT THE MAJORITY OF OUR TEACHERS ANDTHE MAJORITY OF OUR EDUCATORS ARE TRYING TO DO THE RIGHT THING.THEY MAKE BAD DECISIONS, BUT THEY ARE TRYING TO DO THE RIGHT THING,AND SO OUR TESTING INTEGRITY SYSTEM IS ABOUT PROTECTING THEINTEGRITY OF THE SYSTEM AND PROTECTING PEOPLE SO THAT THERE ARE NOTALLEGATIONS, SO THAT WE CAN STAND BEHIND THE RESULTS OF CHILDREN.SO I THINK THAT CULTURAL SHIFT THAT STEVE TALKED ABOUT IN HISPRESENTATION IS SOMETHING THAT MADE A HUGE DIFFERENCE IN OURORGANIZATION.STEVE FERRARA: CORNELIA, I THINK THERE ARE AT LEAST TWO WAYS OFTHINKING ABOUT RESOURCES. SORRY ABOUT THAT. ONE IS JUST PERSONNELRESOURCES. THE OTHER IS EXPERTISE RESOURCES. THAT WAS PART OF MYCOMMENTS. WE'VE HEARD TERMS LIKE "LOCAL CONSORTIA," "SHAREDRESOURCES," "SHARED RESPONSIBILITY." I THINK, FIRST OF ALL, STATESAND LOCAL SCHOOL SYSTEMS HAVE TO WORK TOGETHER IN THE STATEASSESSMENT PROGRAMS. IF THERE ARE OTHER WAYS OF SHARING EXPERTISE-IT WAS AN IDEA THAT LOU PROPOSED--THAT IS ANOTHER WAY OFADDRESSING THE LIMITED-RESOURCES ISSUE, I THINK.210JACK BUCKLEY: QUESTION?AUDIENCEMEMBER:GOOD AFTERNOON. MY NAME IS CHRISTINA SAMUELS. I'M AREPORTER WITH "EDUCATION WEEK."I HAD A QUESTION THAT REFLECTED SOMETHING THAT CHANCELLORHENDERSON WAS TALKING ABOUT IN THE MORNING. IT SEEMED LIKE ONE OFHER CONCERNS THAT SHE MENTIONED WAS ABOUT, I GUESS, A LACK OFSTANDARDIZATION INTO WHAT WOULD TRIGGER A INVESTIGATION AND THENALSO HOW AN INVESTIGATION SHOULD BE DONE SO THAT IT DIDN'T FACEPEOPLE FROM THE PRESS OR FROM THE PUBLIC SAYING THAT THE DISTRICTDIDN'T DO ENOUGH OR THAT THE SCHOOL SYSTEM DIDN'T DO ENOUGH TOINVESTIGATE THE ALLEGATIONS, AND I WONDER, AS PEOPLE WHO HAVE BEENINVOLVED IN INVESTIGATIONS, WHETHER YOU THINK THAT IT IS NECESSARYTO HAVE SOME KIND OF STANDARD SET OF TRIGGERS FOR, "HEY, THIS LOOKSSTRANGE," AND THEN ALSO WHAT SHOULD HAPPEN NEXT BECAUSE--WHEN IS ITOK FOR IT TO STAY AT THE SCHOOL? WHEN DOES IT HAVE TO GO OUTSIDE THESCHOOL, ISSUES LIKE THAT? THANKS.STEVE FERRARA: I HAVE A COMMENT THERE. YOU ASKED, IS IT NECESSARY TOHAVE STANDARDS? I'M NOT SURE WHETHER IT IS NECESSARY OR NOT. ITHINK IT WOULD BE EXTREMELY HELPFUL.211THE THING THAT WORRIES ME IS--I'M REALLY PLEASED THAT NCES TOOK ONTHIS COMPLICATED AND VERY SENSITIVE ISSUE AND BROUGHT US TOGETHER.WHAT MY WORRY IS, WE ARE GOING TO WALK AWAY FROM HERE WITH A LOT OFNEW WAYS OF THINKING ABOUT THIS, AND I DON'T THINK THERE'S ONENATURAL, AUTOMATIC CONSTITUENCY THAT IS RESPONSIBLE FOR ALL THEPIECES OF ALL THIS.SO, FOR EXAMPLE, WE ARE ON THE INVESTIGATION PANEL. I HAVE A LITTLETROUBLE THINKING ABOUT WHAT IS THE EDUCATIONAL CONSTITUENCY OUTTHERE WHO WOULD FEEL COMPELLED PROFESSIONALLY TO DEVELOP OTHERMEETINGS LIKE THIS TO DEVELOP STANDARDS FOR INVESTIGATIONS. MAYBEIT IS THE LEGAL COMMUNITY, BUT, I MEAN, I DON'T KNOW ABOUT THAT.I'M NOT SURE THE RESEARCH COMMUNITY THAT MANY OF US ARE IN IS GOINGTO DO THAT, MEMBERS OF NCME, BECAUSE IT IS NOT PART OF RESEARCH.SO SOME OVERARCHING GROUP--AND I GUESS I WON'T NAME NAMES BECAUSE IDON'T WANT TO PUT PEOPLE ON THE SPOT--BUT THERE ARE ORGANIZATIONSLIKE NCES WHO ARE WORKING ACROSS STATES, ACROSS ASSESSMENTPROGRAMS WHO SHOULD CONTINUE TO PUSH FORWARD, IF IT IS NOTSTANDARDS, AT LEAST GUIDELINES THAT WOULD BENEFIT STATES AND LOCALSCHOOLS AND SCHOOL SYSTEMS AND SO FORTH.LOU FABRIZIO: ONE OF THE THOUGHTS THAT COMES TO MY MIND BASED ONHEARING ABOUT REPORTERS, YOU KNOW, WHEN YOU ARE IN YOUR OFFICE AND212YOU GET SOMETHING IN AN E-MAIL AND IT BLURTS OUT SOMETHING ABOUTSOMETHING HAPPENING IN ATLANTA OR SOMETHING HAPPENING INBALTIMORE, IT REALLY MAKES YOU STOP AND SAY, "YOU KNOW, THAT MAY BEHAPPENING SOMEWHERE IN OUR STATE," AND I KNOW SECRETARY DUNCAN SENTOUT A LETTER TO ALL THE STATE SUPERINTENDENTS TELLING THEM THAT WENEEDED TO BE THINKING ABOUT TESTING INTEGRITY AND EVERYTHING ELSE.I IMMEDIATELY DRAFTED SOMETHING FOR OUR STATE SUPERINTENDENT TOSEND TO ALL OF THE LOCAL SCHOOL SUPERINTENDENTS, COPIED IT TO ALLOF THE L.E.A. TEST COORDINATORS, SAYING, "FOLKS, THIS IS WHY WEHAVE THE CODE OF ETHICS. THIS IS WHY WE HAVE THE PROCTORS IN THECLASSROOM," TRYING TO MAKE SURE THAT WHEN THEY GET CALLS FROMREPORTERS--BECAUSE EVERY TIME ONE OF THOSE STORIES HITS, OUR PHONESTARTS TO RING OFF THE HOOK BECAUSE THEY WANT TO KNOW WHAT ISHAPPENING AND, "WHAT ARE YOU DOING?" AND SO WE USE ALL OF THOSEINSTANCES AS TEACHABLE MOMENTS, AND I LIKE TO ALWAYS START OFFEVERY CONVERSATION WITH A REPORTER WHO IS ASKING ABOUT IT FOR ME TOSAY, "YOU KNOW, LOOK. THERE ARE LAWYERS THAT SOMETIMES DON'T DO THERIGHT THING. THERE ARE DOCTORS THAT SOMETIMES DON'T DO THE RIGHTTHING, AND THERE ARE TEACHERS WHO SOMETIMES MAY NOT DO THE RIGHTTHING, BUT IT IS A SMALL NUMBER, AND WE DON'T WANT THAT TO PUT ASHADOW OVER THE GREAT WORK THAT TEACHERS AND EDUCATORS ARE DOING,"AND SO I'M ALWAYS TRYING TO REFRAME IT BECAUSE THE STORY IS ALWAYSTHESE HORRIBLE THINGS ARE HAPPENING AND IT IS PROBABLY HAPPENING213EVERYWHERE, AND SO YOU JUST HAVE TO DO THAT, OR ELSE YOU END UPGETTING MORE HATE MAIL BECAUSE THEY SAY, "WELL, HOW COME YOU DIDN'TTALK ABOUT THE GOOD TEACHERS?" AND SO I THINK IT IS JUST SOMETHINGWE ALL NEED TO BE VERY SENSITIVE ABOUT.BOB WILSON: THE STANDARD OF WHETHER IT IS SOMETHING THAT SHOULD STAYINSIDE THE SYSTEM OR YOU SHOULD GET OUTSIDE COUNSEL ORINVESTIGATORS TO HELP YOU IS A LITTLE TOUGHER LINE TO DRAW, BUT YOUARE GOING TO HAVE A TEST PROTOCOL. YOU ARE GOING TO HAVE STATELAWS. YOU ARE GOING TO HAVE CODES OF ETHICS. IF WHAT IS REPORTED ORSUSPECTED MIGHT POSSIBLY BE A VIOLATION OF ANY OF THOSE, THEN IT ISSUBJECT TO INVESTIGATION.NOW, YOUR CODE OF ETHICS IN ONE STATE MAY BE A LITTLE DIFFERENTTHAN SOMEBODY ELSE'S IN ANOTHER STATE. YOUR STATE STATUTES MAY BEDIFFERENT. YOUR TEST PROTOCOLS MAY BE A LITTLE DIFFERENT, BUT IFYOU COULD FIND, WITHIN REASON, THAT WHAT HAS BEEN REPORTED MIGHTVIOLATE ANY OF THOSE, THEN THAT IS SOMETHING TO BE INVESTIGATED,AND I WILL TELL YOU--I WANT TO GO TO SOMETHING THAT LOU SAID--ONEOF THE SAD THINGS ABOUT ATLANTA, EVEN THOUGH WE SAID THAT IT WASMAYBE TWO TO 3 TIMES THE NUMBER WE REPORTED, THERE WERE THOUSANDSOF ADMINISTRATORS AND TEACHERS WHO WERE DOING THEIR JOB, DOING IT214RIGHT, DOING IT HONESTLY, AND THEY HAVE SUFFERED GREATLY BECAUSE OFTHIS, AND THEY DIDN'T DO A DADBURN THING WRONG.SO IF THE SYSTEM IS VIGILANT, YOU ARE GOING TO HAVE WHAT LOU SAYS.YOU ARE ALWAYS GOING TO HAVE LAWYERS AND DOCTORS AND FOLKS WHO DOSOME THINGS WRONG, AND THE PUBLIC UNDERSTANDS THAT. WHAT THEY DON'TUNDERSTAND IS WHEN YOU--AT A LEADERSHIP LEVEL, WHETHER YOU ARE THEBOARD OR THE SUPERINTENDENT--ALLOW IT TO GET TO SUCH A MASSIVENUMBER.YOU ARE FAILING TO LEAD THE RIGHT WAY SOMEHOW. THEY WILL UNDERSTANDIF YOU HAVE A TEACHER OR TWO MAKE MISTAKES AND CHEAT. YOU GO IN.YOU DO THE BEST YOU CAN WITH IT, AND ROOT IT OUT. THAT IS NOT GOINGTO CONDEMN THE WHOLE SYSTEM, BUT WHEN YOU END UP WITH WHAT HAPPENEDIN ATLANTA, THE MAJORITY OF THE TEACHERS DID NOTHING WRONG, ANDTHEY ARE SUFFERING ALONG WITH THAT MINORITY THAT CREATED THEPROBLEM.SO YOU WANT TO NIP IT IN THE BUD, AND IF IT COULD POSSIBLY BE AVIOLATION OF ANY OF THOSE THINGS, YOU HAVE GOT TO LOOK INTO IT.AUDIENCEMEMBER:HI. MY NAME IS HEATHER GARAM. I ACTUALLY WORK INCERTIFICATION, BUT I CAME TODAY BECAUSE I THINK THERE ARE A LOT OFPARALLELS YOU CAN DRAW BECAUSE THIS IS ASSESSMENT, WHETHER IT'S IN215SCHOOLS OR CERTIFICATION, AND ALL DAY, WE HAVE BEEN HEARING THISTALK OF INTEGRITY AND A CULTURE OF HONESTY.I FEEL LIKE MY QUESTION ACTUALLY GOES TO A WHOLE OTHER SYMPOSIUM,BUT JUST OFF THE TOP OF YOUR HEAD, HOW WOULD RECOMMEND SCHOOLSCREATE THAT CULTURE OF HONESTY AND INTEGRITY?STEVE FERRARA: I THINK TISHA, WHEN SHE DESCRIBED WHAT HAPPENED INBALTIMORE CITY AFTER A RATHER BLATANT VIOLATION, I THINK WHAT SHEAND ANDRES ALONSO DID IS A GOOD EXAMPLE OF DEVELOPING THAT CULTUREOF INTEGRITY. IT STARTED WITH THE SUPERINTENDENT, AND IT GOES ALLTHE WAY DOWN INTO THE SCHOOLS AND THEN BACK UP AGAIN, I HOPE.BOB WILSON: I THINK IT STARTS WITH LEADERSHIP. STARTS WITH YOURBOARD OF EDUCATION AND WHO THEY HIRE TO ADMINISTER THAT SYSTEM. ITSTARTS WITH THE ASSISTANT SUPERINTENDENTS AND THE PRINCIPALS. IFTHE PRINCIPALS, THE SUPERINTENDENT, ASSISTANT SUPERINTENDENTS, ANDTHE BOARD INSIST ON INTEGRITY, SELF-REPORTING, SELF-REPORTING,PLAY THE RULE OF LIFE LIKE GOLF IS REALLY SUPPOSED TO BE PLAYED.IT IS ONE OF THE GREAT GAMES OF ALL TIME WITH ITS RULES. LET APLAYER INADVERTENTLY TOUCH THE SAND IN A SAND TRAP BEFORE HE HITSTHE BALL WITH HIS CLUB HEAD, HE WILL SELF-REPORT. HE WILL SELFREPORT, "I GROUNDED THE CLUB," AND IT WILL COST HIM.216STEVE FERRARA: MOST OF THEM.BOB WILSON: MOST OF THEM WILL. WELL, THERE SURE WAS ONE A COUPLE OFYEARS AGO ON THE LAST HOLE THAT COST HIMSELF THE TOURNAMENT BYMOVING A TWIG INADVERTENTLY. THE REF THAT WAS ON THE SIDELINE DIDNOT SEE IT. HE REPORTED IT. THEY LOOKED AT IT ON TELEVISION, ANDYOU COULD BARELY SEE IT MOVE. COST HIM THE TOURNAMENT.THERE HAS IT BE THIS SENSE OF NOT ONLY ACCOUNTABILITY FOR THESYSTEM, BUT ACCOUNTABILITY FOR ME, AND IF I'M ABOUT TO READ A STORYTO MY STUDENTS BEFORE THE TEST THAT MIGHT ACTUALLY SHADE THE TEST ALITTLE BIT, MAYBE I NEED TO STEP TO MY PRINCIPAL AND SAY, "CAN IREAD THIS?" OR, "SHOULD I HAVE READ THIS?" SO IT IS A CULTURE THATIS DEVELOPED BY LEADERSHIP.I ABSOLUTELY AM CONVINCED THAT LEADERS OF ANY ORGANIZATION HAVE APERMEATING EFFECT THROUGHOUT THAT SYSTEM, AND IT STARTS THERE, ANDIT ENDS THERE. THEY HAVE TO DO THAT.TISHA EDWARDS: YEAH. I WOULD REITERATE THAT TWO YEARS AGO, WE ONLYHAD MONITORS IN 13 SCHOOLS, AND THESE WERE SCHOOLS THAT WE HADALLEGATIONS OR SUSPICIONS OF CHEATING.217LAST YEAR, WE HAD MONITORS IN EVERY SINGLE SCHOOL IN THE DISTRICT,AND WE DID THE WIDESPREAD TRAINING AND LOTS OF TALKING WITH PEOPLEABOUT WHY THIS ISSUE WAS SO IMPORTANT IN TERMS OF PROTECTINGEDUCATORS AND PROTECTING THE INTEGRITY OF OUR SCHOOL SYSTEM.I WAS ONE OF THOSE PEOPLE WHO WAS ON CALL THE DAY THAT WE DIDTESTING IN OUR SCHOOL SYSTEM LAST YEAR WHEN WE IMPLEMENTED THISMORE RIGOROUS TEST MONITORING PROCESS, AND WE WERE VERY, VERYEXCITED THAT OUR PHONES STARTED RINGING OFF THE HOOK DURING TESTINGADMINISTRATION WHEN PEOPLE WERE SAYING, "I THINK I HAVE MADE AMISTAKE. I THINK I HAVE DONE SOMETHING WRONG. CAN YOU DEPLOYSOMEONE HERE SO THAT WE CAN INITIATE CORRECTIVE ACTION AROUND HOWTO ADDRESS THIS BEFORE THE END OF THE TESTING ADMINISTRATION?" ANDWE SAW THAT AS EVIDENCE OF A HUGE LEAP FOR OUR DISTRICT AROUND THEFACT THAT PEOPLE WANT TO DO THE RIGHT THING.YOU HAVE TO HAVE PEOPLE IN PLACE TO MONITOR AND TO SEE WHAT ISHAPPENING, BUT THIS WHOLE ISSUE OF PREVENTION ALLOWS YOU TO CREATETHAT CULTURE OF HONESTY, AND IT CAN BE A SITUATION THAT MOVES FROMBEING VERY TRAUMATIC FOR AN ORGANIZATION TO BEING VERY EMPOWERINGFOR TEACHERS TO BE ABLE TO SAY, "THERE IS SOMETHING HAPPENINGACROSS THE HALL." THEY WON'T GO AS FAR AS TO SAY IT IS CHEATING,BUT THEY WILL SAY, "IT'S SOMETHING I THINK SOMEONE NEEDS TO CHECK,"OR, "SOMEONE NEEDS HELP, OR, "I NEED HELP."218SO WE SAW THAT HAPPEN VERY QUICKLY WITH ONLY ONE YEAR OFMONITORING. SO I AGREE WITH YOU. THERE IS A LOT OF SELF-REPORTINGTHAT WILL HAPPEN IF YOU HAVE THE PEOPLE AND THE PROCESSES IN PLACEFOR THAT.BOB WILSON: CONSTANTLY TALK ABOUT IT. IT IS NOT TO BE SWEPT UNDER THERUG. IT IS NOT TO BE SAID ONE TIME. INTEGRITY IS SOMETHING TO BEDEALT WITH DAY IN AND DAY OUT BY EXAMPLE AND BY DEEDS AND BY WORDS.JACK BUCKLEY: RIGHT. ALL RIGHT. WELL, WITH THAT, WE NEED TO WRAP UP.I'M SORRY. I'M RUNNING OUT OF TIME.QUICKLY, THEN, TO SUMMARIZE, WHAT DID WE HEAR OF THIS PANEL? WELL,I THINK THERE IS CONSENSUS THAT SAYS INVESTIGATE EVERYTHING,RIGHT? IF YOU GET A TIP, IF YOU GET A CALL, YOU NEED TO DO THE BESTYOU CAN TO INVESTIGATE IT, NO MATTER HOW VAGUE IT IS.I THINK THERE IS PERHAPS LESS CONSENSUS ABOUT WHAT LEVEL OFDIFFERENT INVESTIGATION REQUIRES OUTSIDE PERSONNEL AND HOW FAROUTSIDE IS OUTSIDE, AND I THINK THERE IS A SEVERE RESOURCECONSTRAINT THERE, TOO, DEPENDING ON YOUR DISTRICT AND YOUR STATE,AND THAT IS AN IMPORTANT QUESTION, I THINK, FOR SDAS AND L.E.A.S TO219CONSIDER AS THEY MOVE TO THE NEXT POINT, WHICH IS TO CODIFY THINGS,RIGHT?SO WHEN YOU ARE WRITING YOUR PROCEDURES, WHETHER OR NOT THEY LOOKLIKE MARYLAND'S OR NORTH CAROLINA'S OR SOMEONE ELSE'S, YOU NEED TOLAY THESE THINGS OUT AND BE VERY CLEAR ABOUT WHAT FOLKS CAN EXPECTWITH RESPECT TO WHO WILL CONDUCT AN INVESTIGATION WHEN AND OF WHAT,AND THEN, I GUESS IF I HAD TO PULL ONE OTHER PROBABLY FACT THAT IHEARD, IS THAT PREVENTION IS CHEAPER.SO IF WE HAD TO PICK A PANEL TO BE ON, THE FIRST PANEL WOULDPROBABLY BE MORE COST-EFFECTIVE THAN LEADING US HERE, WHERE, BYTHIS POINT, WE PROBABLY SPENT SOME MONEY ON PREVENTION AND NOTENOUGH AND NOW A GREAT DEAL ON RESOURCES ON INVESTIGATION.WE HAVE GOT ONE PANEL LEFT, BUT WE'RE ACTUALLY GOING TO TAKE OURBREAK. THIS BREAK IS A BIT LONGER THAN THE OTHER BREAKS. IT ISACTUALLY UNTIL 3:15. THIS IS WHAT, IN WASHINGTON, WE CALL SIESTATIME. SO PLEASE JOIN US BACK HERE AT 3:15 PROMPTLY, AND WE WILLCONDUCT OUR LAST PANEL.220PANEL IV: TESTING INTEGRITY PRACTICES AND PROCEDURES FOR ONLINEAND COMPUTER-BASED ASSESSMENTSJACK BUCKLEY: WELCOME BACK.OUR FINAL PANEL DISCUSSION OF THE DAY FOCUSES ON TESTING INTEGRITYPRACTICES AND PROCEDURES FOR THE FUTURE--OR IN SOME PLACES THEPRESENT--FOR ONLINE AND COMPUTER-BASED ASSESSMENTS.SO MANY ASSESSMENTS, AS WE ALL KNOW, ARE TRANSITIONING FROMTRADITIONAL PAPER AND PENCIL TO ONLINE AND COMPUTER-BASED. AND SOWE EXPECT THIS TO BE A PARTICULARLY IMPORTANT AND SALIENTDISCUSSION, ALL THE MORE SO BECAUSE WE ARE JOINED BYREPRESENTATIVES FROM THE TWO RACE TO THE TOP ASSESSMENT CONSORTIAWHO CAN REALLY TALK TO US ABOUT THE FUTURE TODAY.SO AS WE HAVE HAD ALL DAY, WE HAVE A WONDERFUL AND DYNAMIC PANELASSEMBLED. TODAY WE ARE VERY PLEASED ON THIS FINAL PANEL TO BEJOINED FIRST BY WAYNE CAMARA, WHO IS THE VICE PRESIDENT FORRESEARCH AND DEVELOPMENT AT THE COLLEGE BOARD. RESPONSIBLE FORMANAGING RESEARCH AND ASSESSMENT DEVELOPMENT FOR PROGRAMSINCLUDING THE S.A.T. AND THE A.P. HE IS THE IMMEDIATE PASTPRESIDENT OF THE NATIONAL221COUNCIL ON MEASUREMENT IN EDUCATION, AND PRESIDENT-ELECT OF AERA'SDIVISION D, MEASUREMENT RESEARCH METHODOLOGY. HIS RESEARCH ANDPUBLICATIONS FOCUS ON COLLEGE ADMISSIONS, COLLEGE READINESS,LARGE-SCALE ASSESSMENT, ETHICAL AND PROFESSIONAL ISSUES INASSESSMENT AND TEST VALIDATION.WE'RE ALSO PLEASED TO BE JOINED BY JOHN FREMER, THE FOUNDER OFCAVEON TEST SECURITY--A COMPANY THAT HELPS IMPROVE SECURITY INTEST DEVELOPMENT, TEST ADMINISTRATION, REPORTING AND SCORE USE,WHICH ASSISTS STATES, SCHOOL DISTRICTS, AND OTHER ENTITIESINVESTIGATING TESTING IRREGULARITIES. JOHN IS A PAST PRESIDENT OFTHE ASSOCIATION OF TEST PUBLISHERS AS WELL AS NCME AND THEASSOCIATION FOR ASSESSMENT IN COUNSELING. IN 2007 HE RECEIVED THEATP AWARD FOR CONTRIBUTIONS TO MEASUREMENT. AND HE SERVED AS THEEDITOR FOR THE NCME JOURNAL, "EDUCATIONAL MEASUREMENT ISSUES ANDPRACTICE" AND IS CO-EDITOR OF "THE HANDBOOK OF TEST SECURITY."WES BRUCE WAS NAMED CHIEF ASSESSMENT OFFICER FOR THE INDIANADEPARTMENT OF EDUCATION IN JANUARY OF 2009, AND BEGAN WORK WITHTHE DEPARTMENT IN 1999 AS THE DIRECTOR OF THE DIVISION OF SCHOOLASSESSMENT. HE WAS ALSO APPOINTED BY U.S. SECRETARY OF EDUCATIONMARGARET SPELLINGS TO THE FIRST NATIONAL TECHNICAL ADVISORYCOUNCIL, AND CURRENTLY SERVES ON THE PARCC LEADERSHIP TEAM ANDCHAIRS PARCC'S TECHNOLOGY OPERATIONAL WORKING GROUP.222AND TONY ALPERT, WHO SERVES AS THE CHIEF OPERATING OFFICER FOR THESMARTER BALANCED ASSESSMENT CONSORTIUM, WHERE HE OVERSEES THEFISCAL OPERATIONS OF SMARTER BALANCED, COLLABORATING WITHWASHINGTON AS THE LEAD FISCAL STATE, AND PROVIDES EXPERT GUIDANCEON ASSESSMENT DESIGN AND TECHNOLOGY. PRIOR TO JOINING SMARTERBALANCED, TONY SERVED AS DIRECTOR OF ASSESSMENT FOR THE OREGONDEPARTMENT OF EDUCATION, WHERE HE DIRECTED THE ADMINISTRATION OFOREGON'S COMPUTER ADAPTIVE ASSESSMENT SYSTEM. AND, LIKE WES, TONYSERVED ON THE U.S. DEPARTMENT'S NATIONAL TECHNICAL ADVISORYCOMMITTEE.WAYNE, COULD YOU PLEASE START US OFF?WAYNE CAMARA: THANK YOU VERY MUCH, JACK AND EUNICE, FOR INVITING ME.I WANTED TO COMMEND THE NCES AND THE DEPARTMENT FOR HAVING THIS.I HAVE CERTAINLY LEARNED A LOT TODAY AND HOPE I CAN PROVIDE SOMECOMMENTS THAT ARE SOMEWHAT USEFUL. WHAT I WANTED TO DO IS APOLOGIZEIN ADVANCE.I'M FROM THE BOSTON AREA. I TALK FAIRLY FAST, BUT NOT TO WORRYBECAUSE I USE AN EXCEEDINGLY SMALL FONT ON MANY OF MY SLIDES. SO IF223YOU DON'T GET CATCH THE WORDS, YOU CERTAINLY WON'T CATCH THEWRITING.I WANTED TO CAST WHAT WE ARE TALKING ABOUT HERE--CHEATING ANDINTEGRITY--WITHIN A VALIDITY FRAMEWORK FIRST OF ALL, BEFORE IBEGIN THE REMARKS. I THINK IT IS IMPORTANT TO THINK OF INTEGRITYAND CHEATING OR THE LACK THEREOF AS AN EXTENSION OF VALIDITYARGUMENT. CERTAINLY, IT RELATES TO THE APPROPRIATE USE OF TESTS ANDTO THE EXTENT THE TESTS ARE USED FOR PURPOSES FOR WHICH THEY HAVENOT BEEN INTENDED OR HAVE NOT DEVELOPED VALIDATION EVIDENCE. THOSEISSUES OF INTEGRITY AND OPPORTUNITIES TO CHEAT INCREASE.THE VALIDATION ARGUMENT REQUIRES US TO SUPPORT INTENDEDINTERPRETATIONS OF TEST SCORES. AND SO IF WE'RE ATTEMPTING TO USETEST SCORES TO EVALUATE EDUCATORS OR SCHOOL PERSONNEL, THEN AS MUCHAS THE INTEGRITY AND POTENTIAL THEFT ARE RISKS, GREATER RISKS TOTHAT PROGRAM ARE THE LACK OF VALIDATION EVIDENCE. SO, WITHOUTVALIDATION EVIDENCE, I THINK INTEGRITY AND CHEATING ARE MINORPROBLEMS. AND WHERE INTEGRITY AND CHEATING ARE MAJOR PROBLEMS,THERE IS PROBABLY INSUFFICIENT VALIDATION EVIDENCE BECAUSE THEYTHREATEN THE USE OF THE TEST. SO I THINK THAT'S THE FRAMEWORKI WOULD LIKE US TO BEGIN THINKING OF.224IN TERMS OF THE VALIDATION ARGUMENT, I THINK IT IS ALSO IMPORTANTWHEN YOU HAVE CHEATING, MISUSE, LAX PROCEDURES, WHEN YOU HAVEDIFFERENCES IN ADMINISTRATIVE CONDITIONS, EXTREME AMOUNTS OFFLEXIBILITY BY SCHOOL OR BY TEACHER OR BY ADMINISTRATIVE SETTING,THOSE ARE SIGNIFICANT THREATS TO THE VALIDITY. THEY INTRODUCECONSTRUCT IRRELEVANT VARIANCE TO THE SITUATION. AND I THINK THOSEARE ISSUES THAT WE NEED TO GUARD AGAINST. IN TERMS OF PAPER VERSUSONLINE, MOST OF WHAT I'M GONNA TALK ABOUT, CERTAINLY ONLINE TESTINGHAS NUMEROUS ADVANTAGES OVER PAPER AND PENCIL, AND THAT EXTENDS TOTEST SECURITY.BUT WITH ALL ASSESSMENTS, I THINK WHAT WE REALLY NEED TO FOCUS ONIS THE INTENDED PURPOSES AND THE POTENTIAL CONSEQUENCES. AND ONCEWE NAIL THOSE DOWN, THEN WE HAVE TO HONE IN INTO THE SPECIFIC TYPESOF INTEGRITY AND THREATS THAT EXIST. I CAST THEM IN 5 AREAS—ITEMEXPOSURE, CANDIDATE AUTHENTICITY, DATA TRANSMISSION AND STORAGE,PROCTOR AND PERSONNEL INTEGRITY, AND SYSTEMS INTEGRITY. AND I WANTTO REITERATE WHAT I'VE HEARD EARLIER, THAT THIS REALLY SHOULD BEABOUT PREVENTION AND MORE PREVENTION. AND DETECTION, TO ME, WHILEIT DOES SERVE TO IDENTIFY IRREGULARITIES, THE PRIMARY PURPOSE OFDETECTION IS TO MITIGATE FUTURE RISKS. SO I THINK PREVENTION IS 85%OF THE BATTLE AND DETECTION IS SOMETHING--AND INVESTIGATION ISSOMETHING WE NEED TO DO.225BUT I THINK WHEN YOU FIND SITUATIONS WHERE CHEATING IN INTEGRITYHAVE GONE ON, I THINK WE NEED TO LOOK AT OURSELVES AND WHATCONDITIONS AND WHAT SYSTEMS HAVE WE CREATED THAT ALLOWED THIS TOHAPPEN.BECAUSE WE KNOW FROM SOCIAL PSYCHOLOGY LITERATURE THAT THEREARE A LOT OF PRECURSORS TO THEFT--OR COUNTERPRODUCTIVE BEHAVIOR INWORKPLACES JUST GENERALLY IN THE LITERATURE. AND THOSE INSTANCESCOME ABOUT IN TERMS OF THE SYSTEMS. THE GREATER THE OPPORTUNITY,THE MORE LIKELY YOU WILL TAKE INDIVIDUALS WHO HAVE NO PROCLIVITY TOCHEAT OR TO BE DISHONEST. AND YOU'LL INCREASE THEIR PROBABILITY OFDOING SO.SO, IT REALLY IS A SYSTEMS ISSUE. I WANTED TO TALK ABOUT CHEATING,WHICH INCREASES WITH THE AGE OF THE STUDENT; WITH BANDWIDTH, WHICHIS JUST SIMPLY THE INFORMATION PER SECOND; AND ALSO THE DISTANCE.WE'RE NOT HERE TALKING ABOUT DISTANCE LEARNING OR PROCTORLESSEXAMS, BUT IF WE WERE, THOSE WOULD PRESENT SIGNIFICANT RISKS.SIMILARLY, THE SUMMATIVE ASSESSMENTS AND THE DIFFERENT PURPOSESRESULT IN DIFFERENT INTENDED USES OF SCORES AND DIFFERENT THREATS.SCHOOL AND DISTRICT ACCOUNTABILITY PROBABLY PROVIDE LOW THREATSFOR STUDENT CHEATING BUT MODERATE THREATS TO TEACHER. STUDENT226REWARDS ENDORSE DIPLOMAS, COLLEGE CREDIT, SUCH AS AN A.P.EXAMINATION, PROBABLY INCREASE THE STUDENT CHANCES OF CHEATING.PROBABLY HIGH RISK FOR STUDENT BUT LOW RISK FOR TEACHERS ANDEDUCATORS.SIMILARLY, TEACHER AND EDUCATIONAL ACCOUNTABILITY, AGAIN, PROBABLYA LOW RISK FOR—STUDENT BUT HIGH RISK FOR TEACHER, AND SO WITHSTUDENT BARRIERS, GRADUATION, RETENTION, DEVELOPMENTAL COURSES,PROBABLY A HIGHER RISK FOR STUDENT CHEATING, LOWER FOR TEACHERS.SO I THINK THOSE ISSUES WE NEED TO--WHEN WE TALK ABOUT INTEGRITYAND PREVENTION, WE NEED TO REALLY LOOK AT THE INTENDED USE OF THETEST AND FOCUS ON OUR PREVENTION IN THOSE AREAS.I'M NOT GOING TO READ ALL OF THIS CHART, BUT I DID WANT TO JUSTHIGHLIGHT TWO OF THE AREAS, EXTENDED TESTING WINDOW. AGAIN, THERISKS TO THE STUDENT ARE A LITTLE BIT DIFFERENT THAN THE TEACHER.WHEN YOU HAVE AN EXTENDED TESTING WINDOW, WHICH WE TEND TODO WITH A LOT OF THE COMPUTER-BASED TESTING, I THINK YOU'REINCREASING THE RISK THAT STUDENTS WILL DISCLOSE ITEMS AND TESTS TOOTHER STUDENTS. CAN YOU IMAGINE A SCENARIO WHERE WE HAVE THE SAMES.A.T. FORM ADMINISTERED IN YOUR SCHOOL FOR A FOUR-WEEK PERIOD?227WE MIGHT SPIRAL THE ITEMS. BUT EVERY ITEM WAS THE SAME. AND WESPIRALED BETWEEN THREE ESSAY TOPICS. THAT TEST WAS LIVE AND INCOLOR ACROSS THE STATE OF NEW YORK AND MISSOURI FOR THREE WEEKS.WOULD THAT BE TOLERATED? NO. NO. I THINK WITH THE AGE OF COMPUTERBASED TESTING--AND ALSO PAPER-BASED TESTING AS RACE TO THE TOP ISREQUIRING TEACHER AND EDUCATOR ACCOUNTABILITY, THE IDEA THAT WECAN HAVE TEACHERS PROCTOR THEIR OWN EXAM IS LOST. IT HAS ABSOLUTELYNO CREDIBILITY IN THIS DAY AND AGE.IN TERMS OF REUSE OF ITEMS, I THINK THERE IS A GREATER CHANCE OFINTENTIONAL OR NON-INTENTIONAL DISCLOSURE OF ITEMS IF YOU DO NOTHAVE ADEQUATE POOLS BUT ALSO FOR TEACHERS AND EDUCATORS. IF YOUREUSE THE ITEMS OVER TIME, PRETESTING AND EQUATING COULD BECOMPROMISED, BUT THERE'S GREATER RISK THAT THE TEACHER WILL TEACHTO THE TEST OR DEVELOP TEMPLATES THAT WILL, AS WE HEARD THISMORNING, WILL SOMEHOW ADVANTAGE ONE GROUP OVER ANOTHER.LET ME TALK A LITTLE BIT ABOUT THE RECOMMENDATIONS THAT IWANTED--TO LEAVE YOU WITH IN TERMS OF PROCESSES AND POLICIESTHAT COULD MITIGATE RISKS TO THE INTEGRITY OF C.B.T. TESTS.FIRST OF ALL, I THINK WE WANT TO LOOK AT THIS AS A THREE-PHASEDAPPROACH.228ONE--WE WANT TO ELIMINATE THE OPPORTUNITY TO CHEAT TO THE EXTENTPOSSIBLE. WE WANT TO REDUCE THAT OPPORTUNITY.SECONDLY, WE WANT TO SEND THE MESSAGE THROUGH THE CULTURE, ASWE HAVE TALKED ABOUT, THAT THIS IS NOT TOLERATED.BUT, THIRD, WE NEED TO EDUCATE. WE NEED TO EDUCATE THE STUDENT ANDWE NEED TO EDUCATE THE SCHOOL PERSONNEL, WHAT CONSTITUTESAPPROPRIATE AND INAPPROPRIATE ADMINISTRATION CONDITION. WHAT TYPEOF ASSISTANCE IS APPROPRIATE AND INAPPROPRIATE?AND TO THE EXTENT THAT THERE'S ANY CONFUSION OR DOUBT ABOUT WHAT ISAND WHAT IS NOT APPROPRIATE, YOU ARE INCREASING THE OPPORTUNITYFOR CHEATING. AND SO FROM A FAIRNESS PERSPECTIVE AND FROM AVALIDITY ARGUMENT, I WOULD ARGUE THAT ALL THREE OF THOSE HAVE TO BEDONE.IN TERMS OF ADMINISTRATION OF SCORING, I WOULD RECOMMEND THAT WEREDUCE THE OPPORTUNITY FOR CHEATING AS MUCH AS POSSIBLE. AS I SAID,I WANT TO REITERATE, CLASSROOM TEACHERS SHOULD NOT BEADMINISTERING THE TESTS TO THEIR STUDENTS. OTHER CLASSROOMTEACHERS CAN DO THAT. BUT WHEN RESULTS ARE GOING TO HAVE ANIMPLICATION ON THE PERFORMANCE OF AN EDUCATOR OR EVEN THE SCHOOL, I229DON'T THINK WE CAN TOLERATE THAT ANY LONGER. AND IT OPENS THE DOOR.PROCTORS SHOULD HAVE NO STAKE IN THE OUTCOME.THE ENVIRONMENT SHOULD PRECLUDE COPYING RESPONSES FROM OTHERSTUDENTS SEATED ADJACENTLY. MANDATORY TRAINING IS IMPORTANT. EVENMORE SO WITH COMPUTERS. STUDENTS SHOULD READ AND SIGN A STATEMENTLIKE AN HONOR CODE, AND POSSIBLY EDUCATORS AS WELL. YOU NEED TOPROHIBIT ALL HAND-HELD ELECTRONIC DEVICES, SMARTPHONE,CALCULATORS, YOU NEED TO EMPLOY A VARIETY OF ITEM FORMATS,INCLUDING CONSTRUCTED RESPONSE ITEMS. STEVE FERRARA MENTIONED THISEARLIER. BECAUSE IT REDUCES AND CONFUSES THE OPPORTUNITY TO CHEAT.YOU NEED TO IMPOSE CONDITIONS ON RETESTING OPPORTUNITIES. YOUCANNOT HAVE RETESTING UNLESS YOU HAVE AN UNLIMITED ITEM BANK.THERE ARE STUDENTS AND THERE ARE NON-STUDENTS WHO WILL ATTEMPT TORETEST AS MUCH AS POSSIBLE TO GAIN AN ADVANTAGE TO GET ACCESS TOTHE ITEMS.TECHNOLOGY--PREPARE FOR THE UNEXPECTED. IT WILL OCCUR. HOW ABOUTTHE STUDENT WHO GOES IN THERE PURPOSELY, UNPLUGS THE COMPUTER, ORDOES SOMETHING SO THAT THEIR TEST SETTING WILL BE DISRUPTED, SOTHEY CAN TEST BACK TOMORROW, EVEN THOUGH THEY HAVE BEEN EXPOSED TO2303/4 OF THE ITEM POOL. THAT IS A VERY COMMON PLOY. AND IT WILLBECOME MORE COMMON WITH COLLEGE AND CAREER READINESS TESTS, IFTHESE ARE HIGH STAKES TESTS.ENSURE STUDENTS CANNOT ACCESS WEB RESOURCES. IF YOU WANT THEM TOUSE SIMULATIONS OR INNOVATIVE ITEM POOLS, YOU CAN HAVE SOME OFTHOSE WEB RESOURCES IN AN ISOLATED POOL. BUT YOU DO NOT ALLOW THEMTO GET TO THE OUTSIDE ENVIRONMENT. ITEM AND DATA HAVE TO BEENCRYPTED AND STORED ON A SECURE SERVER.IF YOU HAVE PAPER FORMS--AND WE ALL WILL FOR STATE ACCOUNTABILITY-PAPER FORMS USE DIFFERENT ITEM BANKS, AND HAVE THE CHAIN OF CUSTODYESTABLISHED. YOU NEED TO ALSO AUDIT SOCIAL NETWORKS. YOU NEED TOAUDIT SCHOOL PREPARATION OR TEST PREPARATION ACTIVITIES AND BLOGSTO ENSURE THAT SOME OF THE ITEMS, ESPECIALLY THE CONSTRUCTEDRESPONSE ITEMS, ARE NOT BEING EXPOSED.ENSURE HIGH SYSTEM RELIABILITY SO THAT YOU CANNOT EASILY START ANDSTOP. GUARD AGAINST SNIFFERS AND ATTEMPTS TO GET THE PROCTOR TODISCLOSE THEIR PASSWORD. DISABLE NETWORK CAPABILITIES TO THEEXTENT POSSIBLE, USE DETECTION SOFTWARE, AND BACK UP ANY GRADE ORROSTER THAT KEEPS GRADES OR STUDENT PERFORMANCE.231FINALLY--I WON'T GET INTO ALL THE STATISTICS--BUT THERE'S A LOTWRITTEN IN THIS AREA. I THINK THE NATIONAL COUNCIL ON MEASUREMENTAND EDUCATION, THEIR CONFERENCE. THE ASSOCIATION OF TESTPUBLISHERS ARE PLACES TO LOOK AT. CERTAINLY SOME OF THE JOURNALS.BUT YOU WANT TO CHECK ON ABERRANCE RATES; YOU WANT TO LOOK ATRETEST AND SCORE VOLATILITY STATISTICS. YOU WANT TO CHECK ONIRREGULAR LATENCIES AND RESPONSE PATTERNS. YOU WANT TO LOOK AT BOTHHIGH AND LOW ABERRANT SCORES, CHEATING INDEXES AND THRESHOLDS,AND--ALSO FOR DISTANCE ASSESSMENTS, WE NEED TO REALLY MONITOR THEMMORE CLOSELY BECAUSE THE OPPORTUNITY TO CHEAT AND THE MOTIVATIONDIFFERS IN THOSE SETTINGS. THANK YOU.JOHN FREMER: I'M PRETTY EXCITED TO BE HERE. I'VE BEEN WORKING IN THEFIELD FOR 50 YEARS AND I'VE BEEN TO A LOT OF CONFERENCES, BUTRARELY HAVE I BEEN TO ONE THAT HAS SO MUCH POTENTIAL TO MAKE ADIFFERENCE.I WORK FULL-TIME IN TEST SECURITY. I HAVE BEEN NOW FOR OVER NINEYEARS. AND I KNOW THAT PEOPLE ARE READY FOR STANDARDS, FORGUIDELINES, FOR MESSAGES, AND I THINK WE'RE PREPARING THEM,NOT THAT IT FEELS SO COMFORTABLE.MY METAPHOR OF THE PERFECT TEST SECURITY STORM IS INTENDED TO232PICK UP THE BOOK IN THE MOVIE "THE PERFECT STORM."WE'RE IN THE MIDDLE OF IT, AND IT'S NOT GOING TO GET BETTER. AND INMY MIND, IT COMES FROM MANY SOURCES. AND ANY TIME I READ SOMETHINGTHAT SAYS ALL OF THESE PROBLEMS HAVE TO DO WITH TEACHER BONUSES ORALL OF THESE PROBLEMS HAVE TO DO WITH ONE THING, I KNOW THAT IT'SNOT TRUE.THE SITUATION WE'RE IN IS A FUNCTION OF MANY DIFFERENT ISSUES.IT'S A FUNCTION OF THE KINDS OF REQUIREMENTS THAT CAME WITH NOCHILD LEFT BEHIND. IT'S A FUNCTION OF THE FACT THAT WE ARE ON ATRAIN THAT IS GOING TO USE TEST SCORES AS PART OF TEACHEREVALUATION, AND IT SEEMS LIKE THERE'S NO TURNING THAT TRAIN. ATLEAST EVERYTHING THAT I READ SAYS THAT.WE HAVE OUR STATE TESTS AT THE HIGH SCHOOL LEVEL USED AS GRADUATIONREQUIREMENTS.WE HAVE INCREASING EVIDENCE THAT CHEATING IS GOING UP. REPORTERSASK ME--I GET ABOUT ONE INTERVIEW A WEEK FROM THE MEDIA--"IS ITREALLY GOING UP OR IS IT JUST MORE REPORTING?" I THINK IT'S REALLYGOING UP. I THINK THE EVIDENCE IS QUITE CLEAR.233I KNOW IT'S GOING TO STOP AT 100%, BUT IT MAY NOT STOP UNTIL IT'STHERE. I MEAN, IT'S UP TO 2/3, 3/4 NOW FOR YOUR AVERAGE HIGH SCHOOLSTUDENTS.THERE'S ALSO AN INCREASE IN SOPHISTICATION IN CHEATING. JUST LOOKON THE INTERNET IF YOU HAVE ANY DOUBT ABOUT THAT.AND THERE'S ALL KINDS OF ADVICE ON HOW TO CHEAT. SO I WON'T GIVEYOU ANY ADVICE OF HOW TO CHEAT. I TRY NOT TO DO THAT IN ANY SETTING.AND THE CONSORTIA'S PLAN TO USE COMPUTER-BASED TESTING IS GOING TOADD SOME FASCINATION TO THAT STORM.SO, AS OTHERS HAVE POINTED OUT, WHAT WILL BE THE EFFECT OF C.B.T.?WELL, IT WILL REDUCE SOME RISKS. OTHERS WILL REMAIN. AND SOME WILLGO UP.WHAT RISKS WILL BE REDUCED? YOU'RE NOT SO LIKELY NOW TO LOSEA TEST BOOK BECAUSE THERE AREN'T ANY TEST BOOKS. OR THEY'RE NOTSO LIKELY TO BE STOLEN WHEN THE TRUCK DELIVERING THE TESTS STOPS SOTHE DRIVER CAN GET A COFFEE OR SOMETHING, WHICH IS SOMETHING THATHAPPENS NOW. AND COPYING DURING TESTING, IF YOU DO C.B.T. RIGHT ANDI THINK OUR CONSORTIUM FOLK WILL--THEY HAVE VERY GOOD PEOPLE234WORKING WITH THEM IN ALL KINDS OF ROLES—BUT THAT STILL LEAVES MOSTOF THE OTHER PROBLEMS WE'VE HAD WITH PAPER AND PENCIL.THEY DON'T GO AWAY. THEY'RE STILL THERE. PRE-KNOWLEDGE IS THERE,ASSISTING DURING THE EXAM IN VARIOUS WAYS, STEALING, MEMORIZINGQUESTIONS, COLLUDING. AND BECAUSE OF TECHNOLOGY-ASSISTED CHEATING,THE PEOPLE COLLUDING DON'T EVEN HAVE TO BE IN THE SAME ROOM. THEYJUST NEED TO HAVE A WAY OF COMMUNICATING WITH EACH OTHER.I HEARD A STORY ABOUT THAT YEARS AGO. IT WAS A WEST AFRICANCOUNTRY, AND THEY WERE DETERMINED TO AVOID CHEATING. AND THEY KEPTHAVING THIS PROBLEM OF ANSWERS BEING TRANSFERRED INTO THE TESTINGCENTER FROM OUTSIDE. SO THEY RING THE TEST CENTER WITH ARMED GUARDSSO THERE COULD BE NO POSSIBILITY OF PEOPLE MOVING IN AND OUT. BUTSHORTLY AFTER THE TEST WAS ADMINISTRATED, WAS GIVEN AND HANDED OUT,FROM ACROSS THE RIVER, WHICH WAS ANOTHER COUNTRY, THE ANSWERSSTARTED TO BE BROADCAST TO THE TESTING SITE. SO THERE ARE WAYSAROUND ALMOST ANYTHING WE WOULD TRY TO DO.SOME RISKS WILL GO UP.WE HAVE TO DISTRIBUTE ALL OF THOSE ELECTRONIC TESTS TO THE SITES.THAT'S NOT A PERFECT PROCESS. WE HAVE SOME VERY TECH-SAVVY FOLKSTHAT ARE GOING TO TRY TO UNDERMINE IT. AS MANY HAVE MENTIONED,235WE'RE GOING TO HAVE THESE EXTENDED PERIODS. AND THERE'S JUST NO WAYTHAT YOU'RE GOING TO HAVE TESTING PERIODS OF FOUR, FIVE, SIX WEEKSWITHOUT KNOWLEDGE GETTING OUT ACROSS THE TESTING AREAS,PARTICULARLY ABOUT ESSAY AND KNOWLEDGE-BASED QUESTIONS. I THINKWE'RE GOING TO HAVE A NEW EXPANDED UNDERGROUND MARKET FOR TESTSBECAUSE OF THE MULTIPLE STATES.MAYBE IT WASN'T WORTH IT TO STEAL THEM AND TRY TO SELL THEM WHENTHEY WERE JUST FOR ONE STATE, BUT IF THEY'RE FROM 20 DIFFERENTSTATES, I THINK THAT MARKET IS GOING TO BUILD. AND WE DO A LOT OFWORK WITH CERTIFICATION PROGRAMS--HUNDREDS OF MILLIONS OF DOLLARSARE BEING MADE COMPROMISING CERTIFICATION PROGRAMS, NOT SO MUCHYET IN EDUCATION, BUT IT WOULD ONLY HAVE TO BE A FEW MILLIONS FORIT TO GET TO BE A SERIOUS PROBLEM.AND I ALSO WORRY THAT C.B.T., IN GENERAL, HAS INCREASED THE COST OFTEST DEVELOPMENT. I THINK YOU CAN MAKE ARGUMENTS THAT IT'S WORTHIT. I WOULD TEND TO GO ALONG WITH THAT, BUT IT DOES COST MOREMONEY. AND THAT MAY MEAN OVER TIME, WE'RE UNWILLING TO INVEST INSECURITY THE WAY WE NEED TO DO.FORTUNATELY, THERE ARE SOLUTIONS THAT WE CAN DRAW ON, INCLUDING,WE'RE GETTING BETTER AT USING VARIOUS TYPES OF DATA FORENSICS.SINCE WE WERE STARTING AT A VERY LOW LEVEL, WE STILL HAVE QUITE A236WAYS TO GO, BUT WE'RE GETTING BETTER AT HOW TO ANALYZE PERFORMANCEIN ORDER TO STAVE OFF CHEATING AND COMPUTER-BASED TESTING. AND ITHINK DOWN THE ROAD, WE'LL BE ABLE TO DETECT IT WHILE IT'SOCCURRING AND DO SOMETHING TO INTERFERE WITH THE CHEATING ONLINE,WHICH IS WHERE WE WANT TO END UP.IF WE GET TO HAVE WIDESPREAD USE, WHICH IS WHAT I WOULD PREDICT--SOIT'S ROUTINE THAT YOU HAVE DETECTION BUILT INTO YOUR C.B.T.-- IPREDICT THIS OUTCOME--IT WILL BECOME MORE COST EFFECTIVE, LESSEXPENSIVE. AND BECAUSE WE'LL HAVE A LOT OF EXPERIENCE EXPLAININGIT, WE'LL GET BETTER OUT OF IT. AND ONE OF THE PROBLEMS WE HAVE NOWIS WE RUN DATA FORENSICS ANALYSES, BUT THE PEOPLE THAT WE'REEXPLAINING IT TO ARE LEARNING IT FOR THE FIRST TIME. THEY KNOW OUROTHER ANALYTIC METHODS. THEY DON'T KNOW DATA FORENSICS, SO WE HAVETO GET BETTER EXPLAINING.IT'S NOT THAT THEY HAVE TO GET BETTER AT UNDERSTANDING. IF YOUEXPLAIN AND THEY DON'T UNDERSTAND, IT'S YOUR FAULT AS THEEXPLAINER. IT'S NOT THE FAULT OF THE PEOPLE RECEIVING THE DATA.SO WHAT TECHNIQUES DO WE HAVE IN C.B.T.? SOME OF THEM ARE EXACTLYTHE SAME TECHNIQUES WE HAD BEFORE. WE CAN DO UNUSUAL GAINS ANALYSESOR COUNTS OF PERFECT OR NEAR-PERFECT SCORES. ONE THING I WANT TO237COMMENT ON THOSE TWO METHODS--THEY CAN BE DONE TYPICALLY BY THEMEDIA OR ANYBODY WHO HAS ACCESS TO A SITE WHERE YOU HAVETHE INFORMATION POSTED. YOU DON'T NEED THE COOPERATION OF THEVENDOR OR THE DEPARTMENT. ALL OF THE OTHER METHODS DO REQUIRE THAT.YOU CAN'T DO IT ON YOUR OWN, SO YOU CAN'T ANALYZE THE SIMILARITY OFRESPONSES OF INDIVIDUAL TEST-TAKERS UNLESS YOU HAVE ACCESS TOTHOSE RESPONSES. YOU CAN'T ANALYZE AN INDIVIDUAL'S RESPONSEPATTERNS, WHAT, IN I.R.T. IS PERSON FIT. IS WHAT THEY'RE RESPONDINGLOGICAL? AND THE EASIEST ILLOGICAL EXAMPLE I CAN GIVE IS THEY GETTHE HARD QUESTIONS RIGHT AND THEY GET THE EASY QUESTIONS WRONG.THAT DOESN'T HAPPEN IN ANY DEGREE UNLESS THEY'RE PRE-KNOWLEDGE.WE CAN STILL DO ANSWER CHANGING ANALYSES, THE COMPUTER ANALOG OFERASURE ANALYSES, EXCEPT NOW WE KNOW EXACTLY WHAT ANSWERS THEY TOOKAND IN WHAT ORDER. AND I THINK WE'RE STILL WORKING ON HOW TO DOTHAT. WE HAVE ONE ENTIRELY NEW THING WE CAN DO WHICH IS RESPONSETIME ANALYSES. HOW MUCH TIME DID THEY SPEND ON INDIVIDUALQUESTIONS?WE CAN WORK OUT THEIR PATTERNS. WE CAN LOOK FOR ANOMALIES. ANDIN THE THEFT DEPARTMENT, THAT'S A REAL CLUE, IF THEIR TIMING ISVERY DIFFERENT FROM ANYONE ELSE. FAST, FAST, FAST, SLOW, SLOW,238SLOW. IT'S BECAUSE THEY'RE MEMORIZING OR MAKING A RECORD OF THOSEOTHER QUESTIONS. AND ALSO, WE CAN DO WEB MONITORING.SO I'M GOING TO CLOSE WITH SOME RECOMMENDATIONS. I HOPE THEY SOUNDFAMILIAR WITH THINGS YOU'VE HEARD BEFORE.ACKNOWLEDGE HOW SERIOUS IT IS. EXPECT CHEATING.USE MULTIPLE DETECTION METHODS.SOMEHOW KEEP THE WINDOWS AS SMALL AS YOU CAN.THEY CAN'T BE EVERYBODY ON THE SAME DAY, THE SAME TEST THE WAY INMANY INSTANCES WE HAVE NOW, BUT MAKE IT AS SHORT AS POSSIBLE.EVALUATE THE CHAIN OF CUSTODY FOR THE TEST BANKS CAREFULLY.INCREASE EMPHASIS ON TRAINING, WHICH IS WHAT EVERYONE HAS SAID, BUTWE NEED TO WORK TOGETHER ON THAT.HAVE ENOUGH RESOURCES FOR TEST SECURITY. WHEN YOU'RE PILOTINGEVERYTHING ELSE IN THE CONSORTIA, FOR EXAMPLE, PILOT YOUR CHEATINGDETECTION METHODS AT THE SAME TIME SO YOU'RE READY WHEN YOU GOOPERATIONAL.239LEARN FROM EACH OTHER. THE CONSORTIA HAVE FOLKS ASSOCIATED WITH IT,LIKE TONY AND OTHERS, WHO HAVE C.B.T. I GUESS INDIANA IS GOING INTOIT, TOO. AND LEARN FROM OTHERS WHO ARE NOT IN EDUCATION.AND FINALLY, TRACK EVERYTHING IN THE WAY THAT THE CENTER FORDISEASE CONTROL TRACKS DISEASES. COLLECT ALL THAT INFORMATION;PUT IT TOGETHER; AND TRY TO FIGURE OUT EXACTLY WHAT'S HAPPENING,BECAUSE I DON'T THINK ANY OF US COMPLETELY UNDERSTAND THE PROCESSYET. AND C.B.T. IS NEW ENOUGH THAT I THINK THERE'S A LOT TO LEARN.THANK YOU.JACK BUCKLEY: THANK YOU VERY MUCH. WES?WES BRUCE: THANKS. SOME OF THE THINGS I'M GOING TO SAY ARE GOING TOHOPEFULLY EXTEND SOME OF THE COMMENTS FROM BOTH WAYNE AND JOHN, BUTFIRST, I NEED TO REPEAT THE DISCLAIMER YOU HEARD EARLIER.MY VIEWS DO NOT REPRESENT THOSE OF THE DEPARTMENT OF EDUCATION,NCES, SECRETARY OF EDUCATION, THE PRESIDENT, CONGRESS, SUPREMECOURT, OR THE U.N.AND, OFTEN, THEY DON'T REFLECT MY OWN THOUGHTS.240[LAUGHTER]BUT I WANT TO TALK TO YOU SOME ABOUT THE TRANSITION, BECAUSE, WHILEIN '14, '15, THE CONSORTIA ANTICIPATE THAT THE TESTS, WITH THEEXCEPTION OF ACCOMMODATIONS, ALL BE ONLINE, MANY OF US IN THEMEANTIME ARE MOVING, ARE MAKING TRANSITIONS, AND TRYING TO THINKABOUT HOW YOU TRANSITION YOUR SECURITY, YOUR TESTING INTEGRITYPROGRAMS FROM PAPER TO COMPUTER. AND SOME OF THE PROMISES ANDCONCERNS THAT IT BRINGS.SO, FOR MOST OF US, INDIANA IS ONE OF THOSE PLACES WHERE WE'RETRANSITIONING BY GRADE LEVEL, BY CONTENT AREA, SOMETIMES BYSCHOOL, FROM PAPER TO ONLINE AS CAPACITY IS AVAILABLE. WE DON'T YETHAVE ENOUGH COMPUTERS IN EVERY SCHOOL IN THE STATE OF INDIANA FORALL STUDENTS TO TAKE THE TEST ONLINE. SO I'VE GOT SOME PLACES WHERETHE SAME GRADE LEVEL, IN THE SAME SCHOOL DISTRICT, ONE SCHOOL'STESTING ONLINE, ONE SCHOOL'S TESTING ON PAPER. AND SO YOU NEED TOTHINK ABOUT, AS YOU'RE MAKING THOSE TRANSITIONS AND SPENDING LOTSOF TIME THINKING ABOUT HOW YOU TRANSITION YOUR PROGRAM FROM PAPERTO ONLINE, YOU NEED TO THINK ABOUT HOW YOU'RE GOING TO TRANSITIONTO SOME OF THESE TOOLS THAT LOTS OF PEOPLE HAVE TALKED ABOUT,BECAUSE YOU WANT TO LEVERAGE THOSE STRATEGIES THAT YOU CAN FORONLINE, THAT ARE SPECIFIC TO ONLINE, SOME OF WHICH ARE241COMPLEMENTARY, SOME OF WHICH ARE THE SAME, SOME OF WHICH ARE UNIQUETO ONLINE.BUT IN TERMS OF THE FIELD, WHEN I'VE GOT A PRINCIPAL WHO'S GOT HALFOF HIS GRADE LEVELS ONLINE AND HALF OF HIS GRADE LEVELS ON PAPER,THEY DON'T WANT TWICE AS MANY REPORTS ABOUT TESTING INTEGRITY.YOU HAVE TO THINK ABOUT HOW YOU'RE GOING TO BLEND THOSE THINGSTOGETHER SO THAT THEY CAN ACT ON THAT.WE'VE TALKED A LOT TODAY ABOUT THE FACT THAT THIS IS A SMALLPERCENTAGE OF FOLKS, WHILE IT MAY BE INCREASING, IT'S STILL, YOUKNOW, KIND OF THE TIP OF THE ICEBERG, ARE THE PEOPLE WHO ARECOMMITTING THESE KINDS OF ACTS. AND WE WANT TO MAKE SURE THAT WE'VEMADE IT AS SIMPLE AS POSSIBLE FOR THOSE WHO NEED TO INVESTIGATE TOBE ABLE TO USE THE INFORMATION.SO, DO LEVERAGE THE DIFFERENCES THAT COMPUTER-BASED TESTING BRINGSTO YOU.ONE OF THE PROBLEMS IS GOING TO BE DECIDING WHAT YOU DON'T NEEDBECAUSE NOW YOU'VE GOT INFINITELY MORE DATA ON EVERY SINGLESTUDENT. YOU ACTUALLY, IF YOU CHOOSE TO, COULD RECORD EVERY SINGLEKEYSTROKE THAT A STUDENT MAKES. YOU HAVE TO DECIDE HOW MUCH242INFORMATION YOU WANT TO MAINTAIN AND USE, OR HOW MUCH DATA YOU WANTTO KEEP AND TURN INTO INFORMATION AS A PART OF YOUR REGULARPROCESS, AND WHAT WILL YOU SYSTEMATICALLY KEEP AS YOUR BACKUP, ORKEEP IN YOUR BACK POCKET.I THINK OF IT AS, YOU KNOW, IF YOU WANT TO USE AN ANALOGY HERE, ANERASURE ANALYSIS THAT YOU DO ON ALL STUDENTS, BUT WHEN PUSH COMESTO SHOVE, YOU PULL STUDENT DOCUMENTS IN CERTAIN CASES.YOU DON'T PULL EVERY SINGLE STUDENT DOCUMENT AND LOOK THROUGHTHOSE, BUT YOU'VE GOT A SET OF INFORMATION THAT'S GOING TO BE PARTOF YOUR REGULAR PROCESS. AND YOU'VE GOT INFORMATION THAT YOU WANTTO HAVE IN YOUR BACK POCKET, BECAUSE YOU'VE GOT, AS JOHN SAID,YOU'VE GOT INFORMATION ON TIME AND HOW MUCH TIME EACH STUDENT SPENTON EACH ITEM.YOU ACTUALLY HAVE WHAT CLOCK TIME EACH STUDENT ACTUALLY ANSWERED ANITEM. AND IF ALL THE STUDENTS ARE ANSWERING THE ITEM WITHIN ACOUPLE OF SECONDS, MAYBE YOU HAVE A REAL-TIME WAY TO DETECT SOMEPOTENTIAL FRAUD.YOU HAVE THE AMOUNT OF TIME STUDENTS SPEND ON--EACH SESSION ANDWHAT WE'RE CONSISTENTLY FINDING IN INDIANA, ON AVERAGE, IS THAT243STUDENTS TAKE LESS TIME WITH THE ONLINE TESTS THAN THEIR REGULARTIME.WE CONSISTENTLY HAVE--WHEN THE TEST IS ADMINISTERED ON PAPER, WEGIVE THE SAME TIME LIMITS, WHERE OUR TESTS ARE TIMED. BUT WHEN THEYTAKE IT ONLINE, CONSISTENTLY, ALL THE STUDENTS IN THE CLASSROOM AREDONE WELL BEFORE THE END OF THE TIME.BUT THAT'S AN IMPORTANT PIECE, NOT JUST TO JOHN'S POINT, THAT YOUCAN USE AS YOU'RE DEVELOPING TESTS TO MAKE SURE THAT YOU UNDERSTANDTHE AMOUNT OF TIME THAT IT REALLY TAKES FOR A TASK. WE'RE EXPLORINGSOME NEW ITEM TYPES. AND I THINK WE DO NEED TO LEVERAGE OUR PILOTTESTING THAT WE'RE GOING TO BE DOING AND CAPTURE THAT INFORMATIONUP FRONT.AND WE DON'T REALLY KNOW WHAT THE TIME REQUIREMENTS ARE. WE DO WANTTHOSE TIMES--THOSE TESTS TO BE FAIR. BUT YOU'VE GOT, YOU KNOW-THIS, TO SOME EXTENT, MINUTE LEVEL OF INFORMATION.YOU HAVE THE ACTUAL ORDER IN WHICH STUDENTS ANSWER TEST ITEMS.ANOTHER THING THAT WE'VE FOUND IS MORE JUMPING AROUND, TO USE AHIGHLY TECHNICAL TERM, IN THE TEST THAN PEOPLE NORMALLY WOULDANTICIPATE. IF YOU ASK TEACHERS HOW MUCH DO STUDENTS JUMP AROUND ON244THE TEST, THE TYPICAL RESPONSE IS, "WELL, IF THEY CAN'T GET ONE,THEY MIGHT MARK IT TO COME BACK LATER."WHAT WE SEE IN ONLINE IS ACTUALLY STUDENTS MOVING AROUND MUCH MORETHAN WE WOULD HAVE EXPECTED, WHEN WE START ANALYZING THE ACTUALPATTERN OF RESPONSES IS VERY DIFFERENT. AND YOU HAVE TO THINK ABOUTHOW YOU WANT TO USE THAT.AND TO A POINT, YOU ACTUALLY HAVE THE REAL PATTERN OF ANSWER CHOICECHANGES. AND I'M GOING TO--CIRCLE BACK TO THAT IN AN EXAMPLE.SO MANY OF THE METRICS--AND JOHN'S TALKED A LITTLE BIT ABOUT THIS-THAT WE USE WITH PAPER ARE EQUALLY VALID FOR COMPUTER-BASEDTESTING--SCORE CHANGE METRICS BOTH AT THE SCHOOL AND THE STUDENTLEVEL; PART TO WHOLE, HOW A STUDENT IS PERFORMING ON INDIVIDUALITEMS VERSUS HOW THEY PERFORM ON THE TEST OVERALL, BACK TO THE EASYITEMS VERSUS DIFFICULT ITEMS, AND LOOKING AT THAT AT BOTH SCHOOL,STUDENT AND CLASS LEVEL; PERFECT SCORE REPORTS, WHICH IS, AS JOHNPOINTED OUT, SOMETHING THAT'S EASILY OBTAINABLE.THAT'S ONE OF THE EARLY INDICATORS THAT WE USE. STILL--CAN NEVERGET OUT OF MY MIND THE OPPORTUNITY I HAD WHEN THERE WERE 24 PERFECTSCORES IN THE STATE OF INDIANA, AND 23 OF THOSE CAME FROM THE SAMEHIGH SCHOOL.245THAT WAS A GREAT DAY. AND JUST A SIDE NOTE. ONE OF THE MOSTDIFFICULT THINGS THAT THOSE OF US IN THE ASSESSMENT BUSINESS HAVETO DO IS GO THROUGH THAT PROCESS OF PUNISHING STUDENTS FOR THE SINSOF ADULTS. SO, YOU KNOW, BACK TO THE LITANY THAT YOU'VE HEARD ABOUTPREVENTION BEING REALLY CRITICAL.SO I WANT TO WALK YOU THROUGH AN ILLUSTRATION ABOUT THE TRANSITIONAND TALK ABOUT THE ERASURE ANALYSIS.SO YOU'VE HEARD A LOT ABOUT IT TODAY. I WANT TO DRILL INTO IT. JACKWAS UPSET THAT WE HADN'T REALLY GOTTEN TO THE WEEDS A LOT TODAY. SOI WANT TO MAKE SURE THAT WE DO GET DEEP DOWN IN.SO WHEN WE DO AN ERASURE ANALYSIS--TRADITIONALLY WITH PAPER--WHATWE'RE DOING IS LOOKING AT LIGHTER AND DARKER MARKS ON AN ANSWERDOCUMENT.AND THE GENERAL LOGIC IS THAT THE LIGHTER RESPONSE IS CONSIDERED TOBE AN ERASURE. AND THE DARKER MARK, IF THERE ARE TWO MARKS FOR ANYPARTICULAR ITEM, IS CONSIDERED TO BE THE FINAL ANSWER, OK?AND SO THEN WE APPLY LOGIC AND SAY, IF THE LIGHTER ONE WAS THEWRONG ANSWER, WAS A WRONG ANSWER FOR THAT ITEM, AND THE DARKER MARK246IS THE RIGHT ANSWER, THEN WE FLAG THAT ITEM AS A WRONG TO RIGHT,OK?THE PROBLEM IS--AND, OF COURSE, THEN YOU'VE HEARD ALL ABOUT THECRITERIA--HOW MANY STANDARD DEVIATIONS DO YOU USE—CLASSES ANDSCHOOLS GET FLAGGED FOR EXCEEDING. LET'S SAY, IN INDIANA WE USE AFOUR-STANDARD DEVIATION FLAG, AND SOMETIMES THEY GET FLAGGED.NOW, THE FIRST ISSUE HERE IS THAT, OF COURSE, WHENEVER YOU SET ABAR THAT'S BASED ON SOME KIND OF DISTRIBUTION OF SCORES, WHAT ARETHE CHANCES THAT IF NO ONE CHEATED ON THE TEST THAT YOU'D HAVE ANERASURE PATTERN AT SOME CLASSROOM AND SOME SCHOOL THAT WAS GREATERTHAN FOUR STANDARD DEVIATIONS ABOVE EITHER THE MEAN OR THE MEDIAN?IT WILL HAPPEN.SO, YOU KNOW, THERE'S ALWAYS THAT ISSUE ABOUT--STARTING THESEINVESTIGATIONS AND DECIDING WHEN, IN FACT, THERE HAS BEEN WHENSOMETHING'S GONE BEYOND A CONCERN, AND REALLY GOTTEN INTO AN ISSUEOF CHEATING OR WHERE TEST INTEGRITY HAS REALLY BEEN COMPROMISED.SO IF YOU TOOK STATISTICS IN COLLEGE, YOU KNOW THAT THERE OFTEN AREPEOPLE BEYOND THE FOURTH STANDARD DEVIATION. SO, THE BIGGESTCONCERN THAT I HAVE--AND I USE THESE ANALYSES ALL THE TIME--IS THAT247WE DON'T KNOW WHAT THE ACTUAL PATTERN OF A STUDENT'S RESPONSES WEREON A PAPER DOCUMENT.WE HAVE NO IDEA ON EARTH. WE MAKE A HUGE ASSUMPTION ABOUT IT ANDINFER FROM THAT BECAUSE WE DON'T KNOW WHETHER OR NOT THE PATTERNWAS WRONG TO RIGHT OR WHETHER IT WAS RIGHT TO WRONG TO RIGHT. NOIDEA.BUT WITH COMPUTER-BASED TESTING, WE DO KNOW THAT; WE CAN TELL, ANDPOTENTIALLY REDUCE THE NUMBER OF FALSE POSITIVES BECAUSE WE KNOWTHE PATTERN IN WHICH THE STUDENTS ANSWERED. WE KNOW WHEN THEY MADETHEIR FIRST ANSWER, IF THEY CHANGED IT, WHETHER THEY WENT BACK TWOOR THREE OR FOUR TIMES. ALL THAT GETS CAPTURED.WE CAN FACTOR IN OTHER DIMENSIONS, LIKE WHEN THOSE THINGS WERECHANGED. BUT IN THE END, THIS IS STILL GOING TO BE A STATISTICALANALYSIS, AND WILL BE SUBJECT TO THE SAME KINDS OF LIMITATIONS.EVEN THOUGH THERE MIGHT BE FEWER FALSE POSITIVES, WE STILL HAVEN'TPROVEN ANYTHING. ALL WE'VE DONE IS CREATE AN INFERENCE. IN INDIANA,WE COMBINE THE TWO ERASURE ANALYSES INTO ONE DOCUMENT FOR SCHOOLSWHO ARE TESTING. WE USE THE PATTERN ANALYSIS FROM THE COMPUTERBASED VERSION AND COMBINE IT IN A REPORT. WE FLAG CLEARLY WHICHCLASSROOMS TOOK THE TESTS BY COMPUTER AND WHICH BY PAPER,248BUT WE TRY TO PRODUCE ONE REPORT THAT HAS THE SAME KIND OFSTATISTICS FOR SCHOOLS TO USE, SO THAT WE USE THE SAME FLAGGINGCRITERIA AND HAVE THE SAME EXPECTATIONS IN TERMS OF INVESTIGATINGAND REPORTING SO THAT IT'S EASY TO COMMUNICATE.WE'VE TRIED TO THINK ABOUT THE CONTEXT AND THE COGNITIVE LOAD ONFOLKS--TO A POINT EARLIER ABOUT TRYING TO FIND BETTER WAYS TOCOMMUNICATE ABOUT THESE METRICS, AND TO THINK ABOUT HOW YOU MAKETHAT TRANSITION IN THIS WORLD.SO COMPUTER-BASED SECURITY ISN'T GOING TO BE OMNIPOTENT. IT'SSTILL INFERENTIAL. AND WE MAY BE ABLE TO GET STRONGER OR ADDITIONALEVIDENCE THAT WE COULDN'T HAVE BEFORE. IT STILL PROVES NOTHING. ITTAKES THE INVESTIGATION TO DO THAT. AND AS YOU HEARD FROMBALTIMORE, SOMETIMES EVEN AN INVESTIGATION WON'T PROVE EXACTLYWHAT HAPPENED. SO THE INVESTIGATION IS STILL CRITICAL.YOU KNOW WHAT HAPPENED; YOU DON'T KNOW WHO OR WHEN OR HOW. THEPRESS LOVES A SCANDAL. AND COMPUTER-BASED TESTING WILL ALLOW YOU TOCREATE AN EVEN BIGGER ONE.JACK BUCKLEY: WES, THANKS VERY MUCH. TONY, I'M AFRAID WES TOOK ALLYOUR TIME. SO...249TONY ALPERT: IT'S NOT THE FIRST TIME.JACK BUCKLEY: [LAUGHING] PLEASE GO AHEAD.TONY ALPERT: SO THANK YOU VERY MUCH FOR THE OPPORTUNITY TO TALK WITHYOU. INTERESTING. WHEN WE WERE TOLD WHAT THE PURPOSE OF THE SEMINARWAS, WE FOCUSED ON HOW TO HELP STATES AND DISTRICTS IMPLEMENTBETTER PROCEDURES. IT DIDN'T OCCUR TO US THAT PEOPLE WOULD BEASKING THE QUESTION, "WELL, WHAT ARE THE CONSORTIA GOING TO DO TOADDRESS THESE?"SO I'M GOING TO GO OFF--I KNOW. WE'VE JUST BEEN BUILDING THETEST...SO I'M GOING TO GO A LITTLE BIT OFF-SCRIPT.SOME OF THESE POINTS ARE GOING TO BE DIRECTLY RELATED TO SBAC,BUT MANY OF THEM ARE CONSISTENT WITH THE APPROACH PARCC IS USING,I'M SURE.BUT I WILL START OUT TALKING A LITTLE BIT ABOUT SHAREDRESPONSIBILITY IN A SLIGHTLY DIFFERENT WAY THAN GREG TALKED ABOUTEARLIER. AND PROBABLY THE MOST CRITICAL IN MY PERSPECTIVE IS THATWHEN WE EXPECT--WE SHOULD MODEL, AS AN ACTIVITY, RIGOROUS250IMPLEMENTATION AT THE STATE LEVEL. AND I WOULD SAY THE SAME THINGAT THE CONSORTIUM LEVEL.SO WHEN WE PUT OUT AN ASSESSMENT, IT SHOULD BE PERFORMING IN THEMANNER IN WHICH WE DESCRIBED IT, OR SHOULD, SO THAT THERE'S ANEXPECTATION OF PROFESSIONALISM ACROSS THE ENTIRE SYSTEM. AND WHENSOMETHING GOES WRONG, IT JUST OPENS UP THE DOOR FOR IDIOSYNCRATICBEHAVIOR. AND SO TO THE EXTENT THAT EVERYTHING WORKS AS IT SHOULD,PEOPLE BEGIN TO EXPECT MORE OF THEMSELVES, AS WELL.SO, SIMILAR TO THAT, WE SHOULD ESTABLISH A CULTURE OF SECURITY. ANDTHAT'S BEEN TALKED ABOUT PREVIOUSLY, BUT I THINK IT GOES BEYONDASSESSMENT SECURITY AND WELL INTO DATA SECURITY BECAUSE THE LINEBETWEEN COMPUTER-BASED ASSESSMENT AND THE DATA IS MUCHMORE BLURRY THAN IT IS WHEN YOU HAVE A PAPER-BASED ADMINISTEREDFORM AND THE DATA COMES BACK MONTHS LATER. WITH COMPUTER-BASEDTESTS, IT IS POSSIBLE AND LIKELY THAT THE DATA WILL BE RETURNEDIMMEDIATELY, AND THERE'S TRANSFERS OF STUDENT IDENTIFIERS. SO YOUREALLY HAVE TO THINK BROADER ABOUT HOW YOU'RE GOING TO ADDRESS MANYDIFFERENT TYPES OF BREACHES.AND THESE POLICIES SHOULD INCLUDE THE LARGER NETWORK OF ADULTS THATNEED TO BE INVOLVED IN ADMINISTERING A COMPUTER-BASED TEST.251SO NO LONGER IS IT A TEACHER IN ISOLATION OR A PROCTOR INISOLATION.YOU HAVE TO GET THE I.T. FOLKS INVOLVED, POSSIBLY THE SECRETARIESINVOLVED IN LOG--IN HELPING THE TICKETING PROCESS FOR THESTUDENTS. SO THERE'S MANY MORE PEOPLE, AND TO THE EXTENT THAT THOSEPEOPLE ARE GOING TO BE INTERACTING WITH--EITHER SECURE DATA ORSECURE MATERIALS IN THE FORM OF THE COMPUTER-BASED TEST, THEPOLICIES HAVE TO ADDRESS THAT BREADTH.THERE ARE ADDITIONAL COMPLEXITIES IN TERMS OF LOGISTICS, WHICHINCLUDES BRINGING STUDENTS DOWN TO A COMPUTER LAB AND GIVING GOODINSTRUCTIONS ABOUT WHAT SHOULD HAPPEN AND WHAT SHOULDN'T HAPPEN.THERE'S COLLECTING OF NOTES THAT STUDENTS MIGHT HAVE BEEN TAKINGDURING THE SESSION. WHICH, EVEN IN A COMPUTER-BASED TEST, THERE'SPAPER INVOLVED AND THAT COULD VIOLATE THE SECURITY.THERE'S STUDENTS LOGGING OFF OR OTHERWISE SECURING THE COMPUTERWHEN THE STUDENT GOES TO THE RESTROOM.ALL THESE PROCEDURES NEED TO BE DELINEATED. OTHERWISE, IT COULDINADVERTENTLY AFFECT THE VALIDITY OF THE ASSESSMENT. AND THENTHERE'S COMPLEXITIES OF NEW ITEM TYPES. SO TO THE EXTENT THAT252STUDENTS ARE LISTENING TO AN AUDIO PORTION OF THE TEST, IFHEADPHONES AREN'T USED OR THE ROOM ISN'T OTHERWISE SECURED, THENMOST LIKELY THE SECURITY OF THE TEST IS AT-RISK.AND THEN TO THE EXTENT THAT STUDENTS ARE ACTUALLY SPEAKING, LIKETHEY DO IN OREGON'S ENGLISH LANGUAGE PROFICIENCY ASSESSMENT, THATCREATES A WHOLE NEW CATEGORY OF POTENTIAL PROBLEMS INIRREGULARITIES IN ADMINISTRATION. SO THE REASON WHY I TALK ABOUTTHOSE, IS AS A CAVEAT HERE, I THINK THE VAST MAJORITY OFIRREGULARITIES ARE MISTAKES THAT ARE OFTEN DUE TO A LACK OF CLEARDELINEATION BY THE STATE THAT'S PROVIDING THE ASSESSMENT.THERE ARE CERTAINLY SOME SALACIOUS INSTANCES WHERE PEOPLEINTENTIONALLY CHEAT. BUT, BY FAR, I THINK MOST PEOPLE ARE INVESTEDIN ADMINISTERING THE SYSTEM TO THE BEST OF THEIR ABILITY.AND SO I THINK WE HAVE AN OBLIGATION TO MAKE THAT POSSIBLE BYCREATING GOOD SUPPORTS AND STRUCTURES. WE SHOULD DELINEATETRAINING REQUIREMENTS BASED ON THOSE ROLES AND RESPONSIBILITIES,SO THAT IF THERE ARE SPECIAL REQUIREMENTS FOR MATH, WE DON'T FORCETHE READING TEACHERS TO GO THROUGH THOSE TRAINING REQUIREMENTS,ALTHOUGH TYPICALLY THEY ARE THE SAME, AND PROVIDE PRACTICEVERSIONS OF THE APPLICATIONS TO MINIMIZE ERRORS IN ADMINISTRATION,BOTH ON THE PARTS OF STUDENTS AND TEACHERS.253WE SHOULD ESTABLISH HELP DESK SUPPORTS THAT ARE CONSISTENT WITH THELONGER TEST WINDOWS THAT ARE REQUIRED FOR COMPUTER-BASED TESTINGSO THAT QUESTIONS CAN BE RESOLVED QUICKLY AND ERRORS AREN'T MADEINADVERTENTLY. AND WE SHOULD CONDUCT USER ACCEPTANCE TESTING INTHE SCHOOLS BECAUSE SOMETIMES WHAT APPEARS TO BE AN IRREGULARITYCAUSED BY THE SCHOOL IS AN IRREGULARITY IN THE WAY THE APPLICATIONFUNCTIONS. AND THAT WAS POTENTIALLY DUE TO A LACK OF QUALITYCONTROL.WE SHOULD BE AWARE OF WHICH ADULTS CAN BE IN THE SECURED TESTS. SOTHINKING ABOUT THEM, THE LOCAL SYSTEM AS A PARTNER, BE AWARE OFWHICH ADULTS CAN BE IN THE SECURE TESTING ENVIRONMENT. AND WESHOULD USE THE OPPORTUNITIES FOR SAMPLE TESTS AND APPLICATIONSWHEN THEY ARE AVAILABLE; PROVIDE CLEAR EXPECTATIONS FOR WHICHINDIVIDUALS MUST ATTEND THOSE TEST TRAININGS; AND PROVIDE A CLEARPATH FOR IDENTIFICATION AND RESOLUTION OF PROBLEMS.MOST IMPORTANTLY, BE AWARE THAT COMPUTER-BASED TESTING COULD BEOVERWHELMING FOR NEW TEACHERS AND SUBSTITUTES. BY FAR, IN MYEXPERIENCE, THAT'S WHERE MANY OF THE ERRORS OCCUR. AND THAT'S DUETO A LACK OF TRAINING.254DON'T EXPOSE STUDENT IDENTIFIERS INADVERTENTLY. AND THEN BECAUSEIT'S SO IMPORTANT, I REPEAT IT AGAIN. PROVIDE A CLEAR PATH FORIDENTIFICATION AND RESOLUTION OF PROBLEMS.IN TERMS OF SYSTEM DESIGN--I'LL TAKE THE LAST FEW MINUTES TO TALKABOUT HOW THE NEXT GENERATION OF ASSESSMENTS ARE GOING TO ATTEND TOSOME OF THESE ISSUES--IN SBAC, WE'RE GOING TO USE A COMPUTERADAPTIVE TEST WITH LARGE ITEM POOLS BASED ON ITEM TEMPLATES THATWILL ALLOW US TO RAPIDLY CREATE A WIDE VARIETY OF ITEMS THAT WILLMITIGATE THE RISKS, BUT, AS HAS BEEN STATED BEFORE, WON'TNECESSARILY SOLVE IT.WE'RE GOING TO MINIMIZE THE NEED FOR ADULT SUPPORTS BY BUILDINGESSENTIAL FUNCTIONS INTO THE TESTING SYSTEM. ONE OF THE BIGGESTPROBLEMS WE HAVE IN MATHEMATICS IS WE ASK TEACHERS TO READ ITEMSALOUD TO SOME KIDS OFTEN AND THEN WE TELL THEM NOT TO REMEMBER THEITEMS. AND SO THAT'S REALLY NOT PARTICULARLY FAIR. AND ITDEFINITELY GIVES A MIXED MESSAGE. SO TO THE EXTENT THAT THECOMPUTER CAN TAKE THAT ROLE, IT ALLOWS HUMANS TO BE HUMANS AND ACTACCORDINGLY BECAUSE ASKING THEM NOT TO REMEMBER SOMETHING THATTHEY READ IS REALLY DIFFICULT TO DO.WE'RE GOING TO CONTROL ITEM EXPOSURE BASED, IN PART--BASED ON THEITEM TYPE. SO THOSE ITEMS THAT ARE MORE MEMORABLE ARE PROBABLY255GOING TO BE SEEN BY FEWER KIDS AND ROTATED OUT OF THE ITEM BANKFASTER.TO GREG'S EARLIER POINT, AS WELL, WE'RE GOING TO BE PROVIDINGASSESSMENTS THAT SERVE SPECIFIC PURPOSES. SO IN THE SBAC THEORY OFACTION, WE HAVE A FORMATIVE AND AN INTERIM ASSESSMENT, WHICH AREOPEN, NONSECURE, INTENDED TO HELP TEACHERS UNDERSTAND WHAT THEASSESSMENT WILL LOOK LIKE, ALLOW THEM TO ASSESS STUDENTS IN ADVANCESO THAT THEY'RE NOT SURPRISED BY THE RESULTS. AND THEN THESUMMATIVE CAN REMAIN SECURE AND SERVE THE PURPOSE OFACCOUNTABILITY, SO THAT THERE ISN'T AS MUCH INCENTIVE FORCURIOSITY ABOUT WHAT THE QUESTIONS ARE.WE'RE GOING TO BUILD A SYSTEM ARCHITECTURE THAT HAS A PREMISE OFENTERPRISE-LEVEL SECURITY. SO IN SBAC'S CASE, WE'VE POSTED OURFIRST DRAFT OF OUR SYSTEM ARCHITECTURE, WHICH WAS BUILT ON THEASSUMPTION OF SECURITY AT THE LEVEL THAT WE WOULD EXPECT OF THEBANKING INDUSTRY. AND WE WILL EXPECT THAT ALL OF OUR APPLICATIONSWILL ALSO MAINTAIN THAT PREMISE.AND THEN AS MENTIONED EARLIER, WE'RE GOING TO PROVIDE SECUREMETHODS OF ACCESSING THE TEST TO MINIMIZE THE CHANCES FORINADVERTENT IRREGULARITIES.256JACK BUCKLEY: THANK YOU VERY MUCH. ALL RIGHT. SO, AGAIN...I'MHEARING SOME COMMONALITIES BUT ALSO MAYBE SOME POINTS THAT THEPANEL DOESN'T ENTIRELY AGREE ON. I THINK I WANT TO START WITHSOMETHING THAT'S COME UP AT LEAST IN THREE OF YOUR REMARKS, WHICHIS THE ISSUE OF THE TESTING WINDOW. AND SO ONE THING WE KNOW,CERTAINLY WE ALL SEEM TO BE ASSUMING TO BE TRUE, IS THAT WHAT WESEE RIGHT NOW WHEN WE MOVE TO COMPUTER-BASED TESTING AND WHAT WE'RELIKELY TO SEE IN THE FUTURE IN '14, '15 AND A FEW YEARS BEYOND THATWOULD BE A GREATLY INCREASED TESTING WINDOW CERTAINLY WITH RESPECTTO WHAT JIM WAS TALKING ABOUT IN NEW YORK, WHERE WE COULDADMINISTER PAPER AND PENCIL ALL IN THE SAME DAY. HOW FEASIBLE IS ITTHAT THE TECHNOLOGY CAN CATCH UP IN TIME TO REDUCE THIS PROBLEM? IMEAN, THIS IS A PROBLEM THAT IS NOT INHERENT IN COMPUTER-BASEDTESTING NECESSARILY IF YOU HAD SUFFICIENT COMPUTER RESOURCES. ORIS THAT NOT TRUE? OR IS THAT JUST NOT PRACTICAL?WES BRUCE: THE OBVIOUS SOLUTION IS ONE-TO-ONE, YOU KNOW.PRACTICALLY, THOUGH, IN SCHOOLS, ESPECIALLY AS WE RAMP UP TOONLINE, THERE IS GOING TO BE THAT COMPROMISE THAT WE HAVE TO THINKABOUT HOW WE MITIGATE AND PROVIDE LOTS OF SECURITY, MAYBE LOTS OFFORMS, MORE FORMS INITIALLY, BECAUSE IN MOST CASES, THE WINDOW ISGOING TO BE LONGER. FOR SOME OF US WHO HAVE WALKED ON THIS PATHALREADY, WE'VE GIVEN UP ONE OF THE BASIC TENETS OF SECURITY THATYOU HEARD ABOUT EARLIER, IS THAT IN THE PAPER WORLD, GIVING THE257TEST TO ALL OF THE FOURTH GRADERS IN A SCHOOL AT THE SAME TIME ISVERY, VERY SIMPLE. IF YOU HAVE, HOWEVER, 4 FOURTH GRADES AND ONECOMPUTER LAB, GIVING ALL OF THOSE STUDENTS THE SAME TEST ONLINE ATTHE SAME TIME IS IMPOSSIBLE. AND SO YOU HAVE TO THINK ABOUT--ASTONY TALKED ABOUT-- THE LOGISTICS OF THAT CHANGE. AS PEOPLE HAVEMOVED FROM THE PAPER WORLD TO ONLINE WORLD, THEY'VE NEVER HAD TOWORRY ABOUT SCHEDULING MULTIPLE GROUPS OF STUDENTS IN THE SAMEGRADE. AND THEY GET CONCERNED ABOUT THE POTENTIAL FOR EXPOSURE. ANDSO, YOU KNOW, I THINK FROM A PRACTICAL STANDPOINT, IT IS GOING TOBE AN ISSUE THAT WE HAVE TO CONFRONT UNTIL TECHNOLOGY BECOMES MUCHMORE WIDELY PROLIFERATED, MOST IMPORTANTLY FOR INSTRUCTION. BUTTHEN WHEN IT COMES TIME TO ASSESS, THAT WE CAN LEVERAGE THATTECHNOLOGY FOR ASSESSMENT.TONY ALPERT: AND I AGREE WITH WES. I WOULD ALSO ADD, THOUGH, THATEVEN IN A ONE-TO-ONE SITUATION, THERE ARE BANDWIDTH LIMITATIONS.SO THAT ISN'T THE SOLE SOLUTION. AND I THINK TO THE EARLIER POINTS,THERE'S ALWAYS A WAY AROUND SECURITY. SO IT'S REALLY CREATING ALL-TO MITIGATE THE RISKS, WE CAN HOPE TO MITIGATE THE RISK ANDCONTINUE TO ADD MORE CONTROLS AND MITIGATE THE RISK MORE, BUTPEOPLE ARE VERY INVENTIVE. SO IF THEY WANT TO GET AROUND THESYSTEM, THEY'LL FIGURE OUT A WAY.258WAYNE CAMARA: I WOULD SAY THAT-- I PERSONALLY BELIEVE THAT, ASSUMINGYOU RUN INTO A REALLY GOOD TESTING PROGRAM AND YOU'VE GOT THESECURITY AND YOU'VE DONE THE RIGHT THINGS, THAT THE ISSUE OF THETESTING WINDOW COULD PRESENT THE SINGLE LARGEST RISK TO C.B.T.,ESPECIALLY TO HIGH SCHOOL. AND IF THOSE STAKES ARE VERY HIGH TO ASTUDENT, MEANING THAT YOU DO NOT NEED TO TAKE ANY DEVELOPMENTALCOURSE OR YOUR HIGH SCHOOL DIPLOMA DEMANDS YOU PASS THE EXAM, ITHINK IT'S AN UNACCEPTABLY HIGH RISK. I THINK YOU DO NOT NEED AONE-TO-ONE, BUT AT LEAST A ONE-TO-FOUR, MEANING THAT IN A HIGHSCHOOL, YOU NEED ENOUGH COMPUTERS TO OUTFIT THE NINTH-GRADE CLASS.YOU DON'T NEED ONE COMPUTER FOR EVERY STUDENT, BUT YOU NEED ONE FORFOUR, ONE FOR THREE. AND I THINK UNTIL YOU HAVE THAT, YOU'LL BEGINCOMPROMISING OTHER ASPECTS OF THE TEST. SO FOR EXAMPLE, JUSTTHOUGHT LOGIC, WE HAVE GOT THE ADVANCED PLACEMENT EXAM THAT WEADMINISTER NATIONALLY IN HIGH SCHOOLS EVERY YEAR. AND MONDAY, MAY3RD, IN THE MORNING, MAYBE CALCULUS A-B, AND THAT SAME DAY, IN THEAFTERNOON IS BIOLOGY. AND IF WE WERE TO BE REQUIRED TO OFFER THATTEST ON THREE OR FOUR DAYS, NOW WE MUST USE DIFFERENT CONSTRUCTEDRESPONSE TASKS. WE ALSO KNOW THAT THEY MAY NOT BE PARALLEL. SO WEMAY BE GIVING YOU A FORM THAT THE MULTIPLE-CHOICE ITEMS ARE EQUATEDQUITE WELL BUT THE CONSTRUCTED RESPONSE TASKS ARE A LITTLE BITEASIER ON FORM "B" THAN THEY ARE ON FORM "A." WE COULD ALWAYSEQUATE THE TASKS, BUT THEN WE'D BE EXPOSING THEM. SO, IN ORDER TOMAINTAIN SECURITY, YOU BEGIN TO DIMINISH THE PSYCHOMETRIC259QUALITIES OF THE TESTING PROGRAM TO SUCH A LEVEL THAT THE PROGRAMREALLY DOES NOT HAVE THE RIGOR OF A GOOD PAPER PROGRAM. AND I THINKTHAT'S SOMETHING WE WANT TO GUARD AGAINST.JOHN FREMER: WE'RE SUPPOSED TO BE SOLVING PROBLEMS, NOT SIMPLYAMPLIFYING THEM, BUT I SPENT LAST WEEK DOING A SECURITY AUDIT WITHA COLLEAGUE WHO USED TO BE EXECUTIVE DIRECTOR FOR COMPUTER-BASEDTESTING AT E.T.S. AND I WAS DESCRIBING THIS SYMPOSIUM AND C.B.T.AND THE CONSORTIA. AND HE SAID, "WELL, THERE ARE THREE PROBLEMSTHAT THEY HAVE TO SOLVE," TWO OF WHICH HAVE ALREADY BEEN MENTIONED-COMPUTERS AND BANDWIDTH. HE SAID, "YOU REALIZE THAT IN A LOT OFSCHOOLS AROUND THE COUNTRY, THERE JUST ISN'T ENOUGH ELECTRICALHOOKUPS TO DO IT." SO NOT THAT WE--I HAVE A FEELING THAT MIGHT BETHE EASIEST OF THE THREE TO SOLVE. BUT RIGHT NOW WITH HAVING TOREACH ALL THOSE STATES AND ALL THOSE DISTRICTS WITHIN STATES, SOMEOF WHICH HAVEN'T HAD MONEY TO BUY EQUIPMENT, I UNDERSTAND, FORYEARS AND YEARS, IT'S A CHALLENGE.JACK BUCKLEY: TONY, YOU HAD A RESPONSE TO WAYNE'S COMMENTS EARLIER?TONY ALPERT: SO I THINK THERE ARE RISKS. IN MY EXPERIENCE, THOUGH, WEARE ABLE TO EQUATE CONSTRUCTIVE RESPONSE IN SUCH A WAY USING ACOMPUTER ADAPTIVE TEST AT LEAST AND PRE-EQUATING. AS LONG AS THEFIELD TEST IS COMPREHENSIVE ENOUGH AND WIDESPREAD ENOUGH, I THINK260THE EXPOSURE RISKS ARE MITIGATED SIGNIFICANTLY. AND I HAVE NEVEREXPERIENCED A PAPER TEST THAT PROVIDED THE ADVANTAGES OF COMPUTERBASED TESTS. SO IN MY EXPERIENCE, A BAD COMPUTER TEST IS STILLBETTER THAN A GOOD PAPER TEST, ESPECIALLY A COMPUTER-ADAPTIVE TESTIN VIRTUALLY EVERY CIRCUMSTANCE.JACK BUCKLEY: SO MOVING ON TO A SECOND ISSUE THAT CAME OUT OF THEDISCUSSION. SO LET'S ASSUME THAT THE TESTING WINDOW PROBLEM ISMINIMIZABLE. WE LISTEN TO YOU, WAYNE, AND WE SAY, "YOU KNOW WHAT?THIS IS THE BIGGEST PROBLEM," SO WE THROW THE RESOURCES AT IT ANDWE ACTUALLY GET ELECTRICAL POWER WHERE WE NEED IT AND GET THEMACHINES THERE. WHAT ABOUT THE INNOVATIVE ITEM FORMAT? SO WHENWE'RE LOOKING AT THIS NEXT GENERATION OF ASSESSMENT, ONE OF THEPROBLEMS THAT I THINK SEVERAL OF YOU, ESPECIALLY, JOHN, YOUMENTIONED, WE'RE NOW MOVING BEYOND SORT OF SIMPLE, AUTOMATED--YOUCOULD IMAGINE--A.I.-DEVELOPED MULTIPLE-CHOICE ITEMS TO SOME REALLYCOMPLEX HANDS-ON COMPUTER TASKS OR REALLY INTERESTING SIMULATIONS.AND THERE ARE SORT OF TWO OBVIOUS PROBLEMS THERE. THE FIRST IS THATTHEY'RE MEMORABLE, AND THE SECOND IS THAT THEY'RE REAL EXPENSIVE TODEVELOP. SO YOU PROBABLY DON'T HAVE A DEEP POOL OF THEM. WHAT DO WEDO? HOW DO WE FIX THAT PROBLEM? BECAUSE, I MEAN, THE DEPARTMENT,CERTAINLY, WE ARE INTERESTED IN ENSURINGTHAT WE CONTINUE TO DRIVE INNOVATION, AND THAT INCLUDES IN NAEP.AND SO WE HAVE THIS PROBLEM, AS WELL.261TONY ALPERT: SO, ONE OF THE ADVANTAGES OF THOSE INTERACTIVE ITEMTYPES IS THAT THE CONSTRUCT THAT YOU'RE PROBABLY MEASURING ISPROCESS. SO, YOU KNOW, AT THE LEAST, IF KIDS ARE LEARNING THAT ANDTHEY'RE ABLE TO RESPOND TO THAT, THEY'VE PROBABLY LEARNED IT, WHICHISN'T NECESSARILY A BAD THING. YOU CAN VARY, IF YOU ORGANIZE THESEMEMORABLE TESTS WELL AND YOU VARY SOME OF THE CONTENT WITHIN THETEMPLATE, SUCH THAT THERE ARE SOME UNIQUE VARIABLES THAT AREINCLUDED THAT ARE ADJUSTED ENOUGH SO THEY CAN'T MEMORIZE THE WHOLETASK, THEN THERE STILL IS—YOU STILL ARE ABLE TO GENERALIZE THESKILL TO THE LARGER BODY OF KNOWLEDGE. AND THAT ITEM STILL IS ABLETO EVALUATE THE DEGREE TO WHICH THE STUDENT HAS MASTERED THATCONTENT. BUT, CERTAINLY, IT IS A RISK. CERTAINLY, WE WILL HAVE TOCONTROL THE EXPOSURE OF THOSE ITEM TYPES A LITTLE BIT MORECAREFULLY. AND THE RISK IS GREATER EARLY ON IN THE ASSESSMENTPROGRAM. IT WILL BE MITIGATED LATER AS WE ADD MORE AND MORE ITEMSTO THE POOL.WAYNE CAMARA: I WOULD JUST SAY THAT THE--IT DEPENDS ON THE TYPE OFITEM. I THINK THERE ARE A LOT OF REALLY INNOVATIVE ITEMS THAT THECOMPUTER CAN DELIVER THAT ARE OBJECTIVE ITEMS AND THAT ARE VERYSHORT, CONSTRUCTED RESPONSE ITEMS. AND I THINK IN THOSE INSTANCES,YOU DON'T HAVE TO SACRIFICE HAVING MULTIPLE ITEMS. YOU CAN DEVELOPA LOT. I THINK WHEN YOU'RE TALKING ABOUT SCENARIOS, WHEN YOU'RE262VIDEOTAPING, FOR EXAMPLE, A SIMULATION--WE DO A LOT OF THAT INCERTIFICATION, THE F.A.A. TESTS. SOME REALLY GOOD SIMULATIONS. IDON'T THINK YOU CAN AFFORD TO DEVELOP A LOT OF THESE. AND SOULTIMATELY, I WOULD SAY, IF YOU ARE SOLVING THE BANDWIDTH SOLUTIONAND THE COMPUTER SOLUTION, I REALLY AM A FAN OF ONE ANDDONE...BECAUSE IF YOU'RE GOING TO BE USING EXTENDED CONSTRUCTEDRESPONSE TASKS THAT REQUIRE ME TO DO WHAT I DO IN COLLEGE, WHICH ISTO GO ON A LIMITED WEBSITE; TO LOOK AT ORIGINAL SOURCE DOCUMENTS;TO BE EVALUATING THEM AND WRITING AN ESSAY AND YOU'RE GIVING ME 60TO 90 MINUTES TO DO THAT TASK, IT'S REALISTIC; IT HAS HIGHFIDELITY; IT'S WHAT I'M GOING TO DO IN COLLEGE. AND WE CANNOTAFFORD TO DEVELOP 50 OF THOSE TASKS. AND IF WE ARE, I DON'T BELIEVETHEY WILL BE STATISTICALLY PARALLEL NO MATTER HOW MUCH EQUATING ANDPRE-EQUATING WE DO. BUT I DO BELIEVE WITH ONE AND DONE, YOU DON'THAVE TO WORRY ABOUT THAT. YOU HAVE TO HAVE A MAIN FORM, MAYBE TWOMAIN FORMS AND A MAKEUP. SO, AGAIN, I PUT IT BACK TO, FOR THE REALHIGH FIDELITY AND INNOVATIVE ITEMS THAT REQUIRE A LOT OFCONSTRUCTED RESPONSE, I DO BELIEVE THAT IF WE CAN MINIMIZE THATWINDOW, WE WILL HAVE MITIGATED THESE OTHER ISSUES DRAMATICALLY,AND IT WON'T BE A CONCERN.JOHN FREMER: IT STRIKES ME AS POSSIBLE-- AND I HAVEN'T HAD TO WRESTLEWITH THESE PROBLEMS THE WAY MY COLLEAGUES HAVE--THAT THIS ISSUE OFTHE LARGE WINDOW, THERE MAY BE WAYS TO HAVE A QUITE SHRUNKEN WINDOW263FOR CERTAIN TYPES OF THINGS AND A LONGER WINDOW FOR OTHERS, MAYBESOME COMBINATION OF THE KINDS OF THINGS WAYNE WAS TALKING ABOUT-DIFFERENT SIZE WINDOWS. MY SENSE IS THAT THIS IS A SOLVABLEPROBLEM. IT'S JUST I WISH...IT DIDN'T HAVE TO BE SOLVED SO SOON. IFWE HAD, YOU KNOW, FIVE OR SIX YEARS--I'M SURE THEY THINK THAT ALLTHE TIME, BUT I'M ALLOWED TO THINK IT BECAUSE I DON'T HAVE ANYCONTRACT THAT HAS ANYTHING TO DO WITH IT.JACK BUCKLEY: LET ME OPEN UP NOW AGAIN TO THE PANEL AND THE BROADERAUDIENCE. STEVE?STEVE FERRARA: SORRY, FOLKS. I HAVE A QUESTION ON THIS. WHAT ABOUTOTHER DEVICES? I THINK IT'S A POSSIBLE SOLUTION FOR ONE OF THEPROBLEMS. YOU DON'T HAVE ENOUGH DEVICES, YOU DON'T HAVE ENOUGHELECTRICAL OUTLETS, CORNELIA ORR HAS A GREAT STORY ABOUTELECTRICAL OUTLETS IN FLORIDA SCHOOLS THAT SHE TOLD ME A COUPLEYEARS AGO. ANYWAY, YOU DON'T HAVE ENOUGH ELECTRICAL OUTLETS. I KNOWBANDWIDTH IS AN ISSUE. TONY MENTIONED THAT. BUT YOU CAN SOLVE THEDEVICES ISSUES MUCH MORE CHEAPLY THAN DESKTOPS. SO WHAT DO YOU WANTTO SAY ABOUT THE USAGE, OTHER DEVICES, INCLUDING STUDENT-OWNEDDEVICES?WES BRUCE: THEY'RE EVIL.264[LAUGHTER]YEAH...I THINK... ONE OF THE ISSUES THAT BOTH CONSORTIA AREWRESTLING WITH AND NOT JUST THE CONSORTIA, BUT STATES ARE WRESTLINGWITH IS THE EXPLOSION OF THE ADOPTION OF DEVICES THAT DIDN'T EXIST,WHAT, 24 MONTHS AGO. YOU KNOW, THE WHOLE REALM OF TABLET DEVICESAND SMART PHONES. BUT I THINK TABLET DEVICES ARE PUSHING US MORE.AND WHILE IT'S POSSIBLE, ONE OF THE THINGS THAT WE'RE MOSTCONCERNED ABOUT IS THE VALIDITY IN ANSWERING THE COMPARABILITYQUESTIONS.A TABLET DEVICE OR A DEVICE THAT DOESN'T HAVE A KEYBOARD BRINGS UPA SET OF COMPARABILITY ISSUES THAT HAS TO BE ANSWERED. I DON'T KNOWIF IT HAS TO BE ANSWERED A COUPLE TIMES, BUT IT HAS TO BE ANSWEREDAT LEAST ONCE--ABOUT WHETHER OR NOT INPUTTING ON AN ON-SCREENKEYBOARD PUTS A STUDENT AT AN ADVANTAGE OR DISADVANTAGE TO ALLTHOSE OTHER KIDS WHO ARE USING FOUR- AND FIVE-YEAR-OLD TECHNOLOGYWHERE THEY HAVE GOT A PHYSICAL KEYBOARD. I'VE SEEN SOME RESEARCHTHAT TALKS ABOUT THAT THERE IS AN IMPACT CURRENTLY FOR STUDENTPERFORMANCE BASED ON HOW MUCH THEY HAVE TO SCROLL. NOW, I DON'TBELIEVE THAT THAT IS NECESSARILY GOING TO BE THE CASE THREE OR FOUROR FIVE YEARS FROM NOW. BUT AS LONG AS I'VE GOT EVIDENCE AS SOMEONEWHO'S HELPING DESIGN A TEST PROGRAM THAT IF YOU SCROLL MORE, YOURSCORE IS LOWER, I'M GOING TO HAVE TO SAY, "I WANT A STANDARD SIZE265OF A DEVICE. AND I WANT TO BE ABLE TO DISPLAY EACH ITEM IN A SIMILARWAY ACROSS THOSE DEVICES. YOU MAY HAVE A LOT OF EXTRA REAL ESTATEIF YOU'RE LOOKING AT A SCREEN LIKE ONE OF THESE AND YOU'RE TAKINGYOUR TEST. BUT I THINK, YOU KNOW, WE'RE LOOKING AT SETTING AMINIMUM SCREEN SIZE AT LEAST INITIALLY. I CAN'T PREDICT WHERE WEWILL BE IN EIGHT OR 10 YEARS. AS JOHN'S POINTED OUT, WE HAVE GOTALL THIS TIME BETWEEN NOW AND '14 AND '15 TO SOLVE THESE ISSUES,BUT I THINK WE ALSO CAN'T HIDE OUR HEADS IN THE SAND, THAT WE HAVETO THINK ABOUT THE USE OF THOSE DEVICES. AND I KNOW WE'RE BOTHACTIVELY-- BOTH CONSORTIA ARE ACTIVELY LOOKING AT, HOW ARE WE GOINGTO DELIVER ON TABLETS BECAUSE CURRENTLY THERE AREN'T MANY PROGRAMSTHAT ARE ACTUALLY USING THE TABLETS? MY TESTING PROGRAM RIGHT NOW,NO TABLET IS SUPPORTED ON IT.WE ARE LOOKING AT HOW TO DO THAT. BUT AS YOU POINTED OUT, THERE AREA HOST OF ISSUES, NOT ONLY THE SECURITY ISSUE YOU POINTED OUT, BUTALSO, HOW MUCH OF THE REAL ESTATE YOU CAN ACTUALLY ACCESS ANDCONTROL VERSUS THOSE KIND OF IN-BUILT NATIVE CONTROLS THAT ARETHERE.TONY ALPERT: JUST TO ADD ON TO WHAT WES WAS SAYING, ONE OF OURCHARGES AS CONSORTIA IS TO ASSES THE FULL BREADTH AND DEPTH OF THECOMMON CORE AND GET AT DEEP KNOWLEDGE AND SKILLS ASSOCIATED WITHCAREER AND COLLEGE READINESS. AND TO THE EXTENT THAT CAN BE266ACCOMPLISHED ON A SMALLER SCREEN SIZE, WE'RE INTERESTED ININVESTIGATING THAT, BUT VALIDITY IS PARAMOUNT. AND IT GOES BEYONDJUST EVEN TYPING. SO AN EXAMPLE FROM A SCHOOL DISTRICT IN OREGON,THEY TRIED IMPLEMENTING NET BOOKS. AND IT WAS RELATIVELYSUCCESSFUL FOR THE THIRD-GRADERS. BUT HIGH SCHOOL FOOTBALL PLAYERSAND THEIR BIGGER FINGERS DIDN'T REALLY WORK WITH THE KEYBOARD, SOTHERE'S AN INTERACTION BETWEEN THE STUDENT AND THE DEVICE THAT HASTO BE CONSIDERED. AS PART OF THE APPROACH TO EVALUATING MULTIPLEDEVICES, WE'RE GOING TO BE CONDUCTING COGNITIVE LABS TOINVESTIGATE, WHAT ARE THE IMPLICATIONS FOR DIFFERENT ITEM TYPESAND THE MEDIUM THAT STUDENTS WILL USE TO INTERACT WITH THOSE, ANDFIND OUT FROM THE STUDENTS HOW WELL IS IT WORKING. ARE WE REALLYGETTING THE ANSWER FROM YOU THAT YOU TRIED TO GIVE US, WHICH IS ANIMPORTANT PART OF THE DESIGN PROCESS.JACK BUCKLEY: OTHER QUESTIONS? SIR.AUDIENCEMEMBER:WITH PAPER AND PENCIL TESTS, AS THEY ROLL IN, SORTOF, TO THE VENDOR, YOU HAVE EITHER DOING EQUATING OR RUN STATANALYSIS, IS REALLY SCORING THE ACTUAL FORMS, YOU HAVE THEOPPORTUNITY TO INVESTIGATE THE INDICES FOR POSSIBLE COLLABORATIVEEFFORTS OR ANSWER COPYING. AND I'M SURE DR. WHITEHEAD CAN ATTEST TOTHIS, THAT IT GIVES YOU SORT OF THIS WINDOW OF OPPORTUNITY WHEREYOU CAN REALLY INVESTIGATE SOME OF THESE BREACHES IN TEST SECURITY.267WHEN YOU'RE GOING TO COMPUTERIZED-BASED TESTING OR C.A.T., THEPRESSURE IS PROBABLY GOING TO EXIST TO HAVE SCORES IMMEDIATELYREPORTED TO THE USER EITHER FOR REMEDIATION OR FOR WHATEVERPURPOSES. SO YOU'RE CUTTING OUT THAT POTENTIAL PREPROCESSINGWINDOW IN WHICH YOU COULD HOLD SCORES AND NOT RELEASE THEMNECESSARILY TO THE EXAMINEE, SO TO THE STATE. AND SO NOW YOU'RELEFT WITH POTENTIALLY HAVING TO CANCEL SCORES WHEN YOU'RE THENDOING-- THE ANALYSES LATER RATHER THAN HOLDING THEM FOR RELEASE. SOI WAS JUST WONDERING ABOUT YOUR OPINIONS--HOW YOU ENVISION THIS-WHETHER YOUR ADVICE WOULD BE TO NOT EVER RECOMMEND FOR RELEASINGSCORES RIGHT AWAY. I DON'T KNOW HOW YOU'RE SEEING, SORT OF, TESTSECURITY IN TERMS OF PREPROCESSING RATHER THAN THEN HAVING TO PAYFOR REISSUING SCORES.WAYNE CAMARA: IT'S A LITTLE BIT DIFFERENT. WHAT WE KNOW IN SOCIALPSYCHOLOGY IN TERMS OF THE IMPLICIT PRESSURE. AND FOR A STATEDEPARTMENT OR FOR A PRINCIPAL OR SUPERINTENDENT, I THINK THERE IS ADIFFERENCE IN TERMS OF PROCESSING A COMPLAINT OR A SUSPICIOUS--OR ACHARGE THAT COMES TO ME AFTER THE TEST WAS ADMINISTERED TWO DAYSAGO AND TWO MONTHS BEFORE THE SCORES ARE GOING TO BE RELEASED. ANDWHAT WILL I DO WITH THAT? WHAT IS MY INTENTION TO DO VERSUS AFTERTHE FACT WHEN I'VE REPORTED THE DATA TO MY SCHOOL BOARD WHEN THESCORES HAVE GONE OUT AND ACTIONS HAVE TAKEN? AND, UNINTENTIONALLY-WE'RE HUMAN BEINGS AND WE PROBABLY HAVE DIFFERENT MOTIVATIONS268BASED ON THOSE TWO SCENARIOS-- SO MY CONCERN WOULD BE NOT WITHRELEASING SCORES BECAUSE I THINK THERE'S A WAY TO RELEASE THEM ASDRAFT SCORES OR TENTATIVE SCORES--BUT MY CONCERN WOULD BE, THEN,WHAT WOULD THAT SAY ABOUT OUR PROCLIVITY OR OUR WILLINGNESS TOINVESTIGATE CERTAIN ACTIVITIES? I THINK IT WOULD BE DIFFERENTIAL.AND THAT'S WHY I THINK IT WOULD BE MUCH MORE IMPORTANT IN THATSCENARIO TO HAVE INDEPENDENT GROUPS APART FROM THE SCHOOLADMINISTRATORS AND CERTAINLY APART FROM THE ASSESSMENT FOLKS IN ASTATE DEPARTMENT OF EDUCATION, AS WE'VE SEEN TODAY. BUT SOMEINDEPENDENT GROUP WHO PROCESSES THOSE COMPLAINTS ORINVESTIGATIONS, BECAUSE OTHERWISE I'M WEARING TWO HATS. I'MWORRIED ABOUT THE INTEGRITY OF MY PROGRAM OR MY SCHOOL, AND NOW I'MHEARING THESE RUMORS. THEY SOUND UNFOUNDED. AND I WANT THEM TO GOAWAY. AND I WANT THEM TO GO AWAY MUCH MORE AFTER THE SCORES HAVEBEEN REPORTED IN THE MEDIA AND TO PARENTS THAN I WOULD IF THEY WERENOT RELEASED.JOHN FREMER: ...MY PERSPECTIVE, IT'S BETTER NOT TO REPORT SCORES ASOF THE TIME OF TAKING THE TEST. AND IF YOUR MODEL IS, WELL, WE'RETRANSITIONING FROM A PAPER AND PENCIL STATE ASSESSMENT PROGRAM,THEY'RE NOT USED TO GETTING THEIR SCORES RIGHT AWAY. THEY'RE USEDTO WAITING. THE PROBLEM OCCURS IF YOU HAVE A PROGRAM AND IT ALREADYIS REPORTING RIGHT AWAY. THEN YOU THINK THAT'S THE ONLY WAY IT CANBE. WE DO A LOT OF WORK AT CAVEON WITH THE I.T. INDUSTRY. AND THEY269REPORT RIGHT AWAY. AND THEY PROBABLY HAVE THE MOST TROUBLE OF ANYINDUSTRY WITH PROBLEMS WITH THE CORRUPTION OF SCORES. WE DON'T HAVETHAT MUCH OF A TROUBLE IN EDUCATION. I ESTIMATE IT'S ONLY MAYBE 1%OR 2%. SOMETIMES YOU SEE LARGER ESTIMATES, BUT YOU DON'T HEAR,OTHER THAN AUDREY'S SUMMARY, YOU DON'T HEAR 10% OR 20%. IN THE I.T.INDUSTRY, SOME OF THEM, 50% OF THEIR SCORES SHOW SOME EVIDENCE OFCOMPROMISE. LET'S NOT IMITATE THEM AND THEIR IMMEDIATE SCOREREPORTING.DAVID FOSTER: BUT I JUST WANT TO ADD I DON'T THINK REPORTING SCORESIMMEDIATELY IN THAT PARTICULAR EXAMPLE CONTRIBUTEDTO THE PROBLEMSTHAT THAT INDUSTRY HAS. IT WAS REALLY POOR TEST ADMINISTRATIONPROCEDURES. WHEN YOU HAVE COMPUTERIZED TESTS, PEOPLE ARE GOING TOEXPECT YOU TO BE A LITTLE FAST WITH YOUR RESPONSES. THEY'RE NOTGOING TO, "WELL, WE'RE GOING TO WAIT THREE WEEKS" OR SIX WEEKS ORWHATEVER. NAH. YOU KNOW, IT'S NOT GOING TO FLY. THIS IS ANIMMEDIATE WORLD WE LIVE IN. SO I THINK WE NEED TO--I LIKE THE IDEAOF PROVISIONAL SCORES, WHICH ALLOWS YOU IN A CERTAIN PERIOD OFTIME--PROBABLY DAYS-- TO MAKE A DECISION THAT THAT TEST IS VERIFIEDOR NOT AND MAKE A CHANGE, BUT IT'S GOING TO HAVE TO BE DONE FAIRLYQUICKLY.JACK BUCKLEY: WELL, I'LL ASK ONE MORE, WHICH ACTUALLY GETS BACK TO APOINT THAT, JOHN, YOU MADE, WHICH IS THE ISSUE ABOUT, AGAIN, NOW270LOOKING FOR IT--I GUESS IT'S FAIR TO SAY, SO, REALLY, WE'RE TALKINGABOUT TWO THINGS HERE. AND WE'VE KIND OF CONFLATED THEM, OR ATLEAST I HAVE. I'M GUILTY OF THIS. COMPUTER-BASED TESTING ORASSESSMENT BUT ALSO THE RACE TO THE TOP ASSESSMENTS OR THE COMMONCORE OR STATE STANDARDS ASSESSMENTS. AND, YES, BECAUSE THEY'REINTERTWINED, IT'S EASY TO SORT OF CONFLATE THE ISSUES. JOHN, YOURAISED THE QUESTION ABOUT PILOTING THE CHEATING DETECTION ORWHATEVER GOES INTO THE TECHNOLOGY IN THAT ARCHITECTURE AT THE SAMETIME AS ACTUALLY PILOTING THE ASSESSMENTS. AND I JUST WANTED TO ASKWES AND TONY IF YOU THOUGHT THAT WAS FEASIBLE BECAUSE IT SOUNDSLIKE A GOOD IDEA FROM WHERE I SIT.TONY ALPERT: SO THE SMARTER BALANCED APPROACH, WE'RE GOING TO HAVE APILOT WHICH IS RELATIVELY SMALL SCALE. IT'LL ONLY BE ABOUT 30,000STUDENTS. I KNOW. THAT'S A WHOLE DIFFERENT WORLD. AND THEN THEREWILL BE A FIELD TEST. THE PILOT, BY ITS NATURE, SINCE THE ITEMSWON'T BE SCALED, IT WOULD BE HARD TO DETECT IRREGULARITIES. THEFIELD TEST, WE'RE PROBABLY GOING TO HAVE SOME PRELIMINARY SCALINGTHAT MIGHT BE USED AS A BASIS FOR EVALUATING CHANGES IN SCORES, BUTWE'RE NOT GOING TO HAVE THE BREADTH OF DATA THAT WE MIGHT OTHERWISEEXPECT IN AN OPERATIONAL ADMINISTRATION. SO I DON'T THINK WE COULDPILOT THE FULL BREADTH OF SECURITY MEASURES, BUT IT'S POSSIBLE TODO SOME OF THEM.271WES BRUCE: YEAH. I THINK I WOULD AGREE THAT, YOU KNOW, PART OF THISIS TO TRY TO START THINKING IN DIFFERENT WAYS THAN WE DO WITHPAPER, TO THINK ABOUT WHAT DATA IS IT POSSIBLE TO COLLECT IN "REALTIME,"WHETHER USING PROXY SERVERS OR WHETHER YOU'RE ACTUALLYDELIVERING THE TEST OVER THE INTERNET. CAN YOU COLLECT DATA THATSHOWS EVIDENCE OF SOME KIND OF COLLUSION OR SOME KIND OF ADULTINTERVENTION WHILE THE TEST IS GOING ON. BECAUSE YOU'RE CAPTURINGEVERY ITEM, YOU'RE DOING SOMETHING WITH IT. AND AT THE VERY LEAST,YOU'RE ENCRYPTING IT AND STORING IT. BUT YOU DO HAVE--YOU KNOW, YOUDON'T HAVE THAT BUILT-IN DELAY WHERE YOU'RE WAITING FOR EVERYTHINGTO RUN THROUGH A SCANNER. SO I THINK TO TONY'S POINT, I DON'T THINKWE CAN DO THE FULL RANGE, BUT CERTAINLY SOME OF THOSE THINGS THATWE HAVE NOT BEEN DOING, WE CAN START THINKING ABOUT AND DEVELOPING.AND IT ONLY MAKES SENSE TO TEST THOSE OUT IN THAT FIELD TESTINGTIME TO SEE WHETHER OR NOT IT LOOKS LIKE THOSE ARE GOING TO WORK ATSCALE. OR SUDDENLY YOU FIND OUT THAT THOSE ARE HOGGING UP A BUNCHOF YOUR BANDWIDTH AND THEY'RE NOT GOING TO BE PRACTICAL TO DO INREAL TIME. YOU MAY HAVE TO DO THEM OFF-LINE. BUT I THINK JOHN'SSUGGESTION IS A GOOD ONE AND CERTAINLY ONE THAT WE'LL WANT TOCONSIDER.JACK BUCKLEY: ALL RIGHT. WELL, IF THERE ARE NO OTHER QUESTIONS, IGUESS WE WILL WRAP THIS PANEL UP. I THINK, AGAIN, LOOKING BACK ATWHAT I'VE HEARD, CERTAINLY THERE'S CONCERNS IN COMPUTER-BASED272TESTING ABOUT NOT ONLY WILL SOME THREATS TO TESTING INTEGRITY BEMITIGATED BUT OTHERS WILL BE CONSTANT OR INCREASED. AND I THINKTHAT SEEMS TO BE WHAT'S EMERGED AS THE LARGEST AREA OF CONCERN ISTHE TESTING WINDOW OR THE SIMPLE SIZE OF THE TESTING WINDOW, WHICHIS LARGELY DRIVEN BY LIMITATIONS OF TECHNOLOGY AND AVAILABILITY.AND TO THE POINT OF WHETHER OR NOT WE WOULD BROADEN THE POOL OFACCEPTED TECHNOLOGY BY ALLOWING DIFFERENT DEVICES, NON-STANDARDDEVICES, OR STUDENTS' OWN DEVICES, WE WERE TOLD THAT WAS EVIL.[LAUGHTER] BEYOND JUST THE TESTING WINDOW, CERTAINLY, THERE'SPOTENTIAL PROBLEMS WITH INNOVATIVE ITEM TYPES, ALTHOUGH I THINKTONY HAD A VERY GOOD EXPLANATION FOR HOW WE COULD MITIGATE APROBLEM THERE. AND, ALSO, SOMETHING ELSE THAT JOHN BROUGHT UP THATWE DIDN'T REALLY GET BACK TO, BUT JUST THE SIMPLE THREAT WITH NOTJUST COMPUTER-BASED TESTING BUT WITH THESE PARTICULAR COMMON CORESTATE STANDARDS ASSESSMENTS, THAT THE MULTISTATE NATURE OF THEM ISLIKELY TO LEAD TO MUCH GREATER THEFT INCENTIVE. AND I THINK THAT'SA VERY REAL RISK BOTH ON THE PART OF STUDENTS SHARING ITEMS THROUGHSOCIAL NETWORKING AND ALSO ON THE PART OF ADULTS WHERE THERE COULDBE A REAL FINANCIAL GAIN. AND THAT'S NOT JUST FOLKS IN SCHOOLS.WE'RE TALKING ABOUT EVERYWHERE-- EVERYWHERE WITH ACCESS—WHICHMEANS CHAIN OF CUSTODY ISSUES WILL ALSO BE VERY IMPORTANT ANDHARDER TO TRACK. I ALSO RECALL THAT WE TOUCHED ON SOME OTHER VERYINTERESTING POINTS, WHICH INCLUDE THE FACT THAT THESE NEW METHODS273OF TEST ADMINISTRATION ALSO PROVIDE POTENTIALLY NEW STATISTICSTHAT WILL HELP US MONITOR AND POTENTIALLY EVEN IN REAL TIME, BUTCERTAINLY LATER, TEST TAKING AND POTENTIALLY ALLOW US NOT ONLY TOFIND--MORE USEFUL INFORMATION BUT ALSO FOR CHECKING THE INTEGRITYOF THE TESTS. AND, WES, THANK YOU FOR THAT EXAMPLE, LOOKING AT SORTOF HOW WE WOULD MOVE BEYOND WRONG-TO-RIGHT ERASURE ANALYSIS, WHICHWE SPENT A LOT OF TIME ON EARLIER, TO REALLY GETTING A LOT MOREINFORMATION OUT OF THAT.I THINK AT THIS POINT WHAT I'D LIKE TO DO IS JUST SUMMARIZE--WELL,ACTUALLY, JOANNE, IF I COULD ASK YOU TO PROVIDE SOME CLOSINGREMARKS, I THINK AT THIS POINT THAT WOULD BE A GOOD TIME TO DO IT.SO, FIRST OF ALL, I'M VERY PLEASED, THEN, TO WELCOME OUR LASTSPEAKER OF THE DAY, JOANNE WEISS, THE CHIEF OF STAFF TO THE UNITEDSTATES SECRETARY OF EDUCATION, ARNE DUNCAN.JOANNE JOINED THE DEPARTMENT IN 2009 TO SERVE AS SENIOR ADVISER TOTHE SECRETARY AND THE DIRECTOR OF THE RACE TO THE TOP FUND. AND INTHIS CAPACITY, SHE LED THE DEPARTMENT'S RACE TO THE TOP PROGRAM,WHICH WAS DESIGNED TO ENCOURAGE, AS YOU KNOW, AND REWARD STATESMAKING SYSTEM-WIDE, COMPREHENSIVE, AND COHERENT EDUCATION REFORM.274PRIOR TO JOINING THE ADMINISTRATION, JOANNE WAS THE PARTNER ANDCHIEF OPERATING OFFICER AT THE NEW SCHOOLS VENTURE FUND, A VENTUREPHILANTHROPY FIRM WORKING TO TRANSFORM PUBLIC EDUCATION BYSUPPORTING EDUCATION ENTREPRENEURS AND CONNECTING THEIR WORK TOSYSTEMIC CHANGE.WE'RE VERY PLEASED THAT JOANNE IS ABLE TO JOIN US TODAY. AND I'DLIKE TO THANK HER FOR PARTICIPATING. AND I TURN THINGS OVER TOJOANNE.JOANNE WEISS: THANKS, JACK. UM, CAN YOU HEAR ME? AM I TURNED ON UPHERE? YES, NO? IT'S NOT, RIGHT? OK, NOW? NO.A LITTLE BATTERY SWITCHING GOING ON NOW.AH, GREAT. THANK YOU.SO, THANK YOU, JACK, AND THANK YOU, JOHN, FOR HOSTING THIS EVENTTODAY ON BEHALF OF THE SECRETARY. THANK YOU FOR MODERATING THESEPANELS, JACK. YOUR SUMMARIES WERE TERRIFIC RIGHT UP UNTIL THE LASTONE. I'M IMPRESSED.I'M GOING TO KEEP MY REMARKS VERY BRIEF BECAUSE I REALIZE THAT I'MALL THAT'S STANDING BETWEEN MANY OF YOU AND THE AIRPORT. BUT I DO275WANT TO REALLY THANK ALL OF YOU FOR PARTICIPATING IN TODAY'STESTING INTEGRITY SYMPOSIUM. WE'RE GOING TO SHARE WHAT WE LEARNEDTODAY WITH THE SECRETARY, OF COURSE, BUT MUCH MORE IMPORTANTLY,WE'RE GOING TO SHARE IT WITH THE PUBLIC.WE'VE ALREADY PUBLISHED ONLINE ALL OF THE RESPONSES THAT WE'VERECEIVED TO THE REQUEST FOR INFORMATION THAT WE'VE PUT OUT.ALL THE WEBINAR TODAY, INCLUDING EVERYBODY'S POWERPOINTS ANDREMARKS, ARE ONLINE AND WILL REMAIN UP THERE SO THAT PEOPLE CANLOOK AT THIS AND LEARN FROM IT. AND WE'RE GOING TO BE DRAFTING ASUMMARY OF THE FINDINGS SO THAT WE CAN HELP STATES AND SCHOOLDISTRICTS IDENTIFY AND IMPLEMENT BEST PRACTICES IN THIS AREA.THE GENESIS OF THIS EFFORT, AS I THINK YOU HEARD THIS MORNING, CAMEFROM A CONVERSATION THAT WE HAD WITH D.C. PUBLIC SCHOOLS CHANCELLORKAYA HENDERSON. LAST SUMMER WHEN SHE TOLD ME THAT AS D.C. WASPURSUING ITS INVESTIGATIONS INTO ALLEGATIONS OF CHEATING,THERE WAS VIRTUALLY NO LIBRARY OF BEST PRACTICES TO RELY ON AND NOSTANDARDS OF TESTING INTEGRITY FOR THEM TO RELY ON. SO THEY WEREREALLY FLYING BLIND. AND SHE ASKED WHETHER THE DEPARTMENT WOULDSTEP IN TO HELP STATES AND DISTRICTS BY PULLING TOGETHER A GROUP OFEXPERTS, POOLING THEIR KNOWLEDGE AND PUBLISHING THEIR276RECOMMENDATIONS. AND TODAY IS THE CULMINATING EVENT OF THAT,SHORT, OF COURSE, OF US PUBLISHING ALL OF THE INFORMATION THATWE'VE LEARNED.BUT THE GOAL, REALLY, OF THIS WHOLE ENDEAVOR HAS BEEN TO HELPSTATES AND DISTRICTS AND, OF COURSE, THE DEPARTMENT, AS WE ALSOTHROUGH NAEP AND OTHER VEHICLES ARE LARGE-SCALE ASSESSMENTPROVIDERS, BUT REALLY TO HELP STATES AND DISTRICTS HAVE THE BENEFITOF THE COLLECTIVE WISDOM OF THE EXPERTS IN THIS FIELD, WHO HAVEBEEN SO HARD TO TAP INTO FOR THE 14,000 SCHOOL DISTRICTS ACROSSAMERICA THAT ARE ADMINISTERING THESE KINDS OF ASSESSMENTS EVERYYEAR.THE CHANCELLOR TOLD ME WHEN SHE CALLED ME LAST SUMMER THAT THEREWERE THREE THINGS THAT SHE NEEDED TO BETTER UNDERSTAND.FIRST, SHE ASKED US TO HELP UNDERSTAND HOW TO PREVENT CHEATING FROMHAPPENING IN THE FIRST PLACE.SECOND, SHE WANTED MORE INFORMATION ON WHAT BEST PRACTICES WERE FORIDENTIFYING OR FLAGGING THE PLACES WHERE CHEATING MIGHT HAVEOCCURRED, TO REALLY MAKE THE AUDITING PROCESS MORE EFFECTIVE ANDMORE COST-EFFECTIVE.277AND THE THIRD THING WAS WHAT BEST PRACTICES LOOKED LIKE FORINVESTIGATING ALLEGATIONS OF CHEATING, WHAT THE PROPER PROCEDURESWOULD BE FOR--CONDUCTING SUCH INVESTIGATIONS.SO--I WANT TO ALSO JUST REALLY THANK KAYA FOR RAISING THIS ISSUE TOUS, AND SPEAKING PERSONALLY, I HOPE, ON BEHALF OF MANY OF YOU, ASWELL, HERE, THOUGH, TODAY WAS REALLY A VALUABLE LEARNINGEXPERIENCE.I WANT TO JUST THANK ALL OF YOU FOR SHARING SO MUCH WISDOM WITH USTODAY. THERE WAS A LOT TO DIGEST. AND I THINK THAT WE'LL BE KIND OFGOING BACK OVER THESE TRANSCRIPTS AND THE SUMMARY THAT WE'LLPUBLISH FOR A LONG TIME JUST TO RETURN TO IT AND REMIND OURSELVESOF THE KIND OF PRACTICES AND PROCEDURES THAT ARE GOING TO HELP USTHROUGH THIS.WE REALLY, AS A COUNTRY, RELY ON ACCURATE, RELIABLE, AND TIMELYINFORMATION TO UNDERSTAND WHAT STUDENTS KNOW AND CAN DO, AND WEHAVE TO HAVE INFORMATION THAT'S ROBUST AND RELIABLE SO THATPARENTS, SO THAT TEACHERS, SO THAT COMMUNITIES, SO THAT THE PRESSHAVE INFORMATION THAT THEY CAN TRUST.HIGH-QUALITY DATA IS CRITICAL TO MAKING SURE THAT WE CAN EVALUATETHE EFFECTIVENESS OF OUR POLICIES AT THE FEDERAL, STATE, AND LOCAL278LEVELS. WE NEED TO USE IT TO EVALUATE THE EFFECTIVENESS OF TEACHINGIN OUR CLASSROOMS, OF THE CURRICULUM MATERIALS THAT ARE PROVIDED TOOUR KIDS, AND OF THE SCHOOLS AND THE DISTRICTS THAT ARE PROVIDINGSERVICES TO OUR STUDENTS.IN SHORT, IT REALLY OFFERS AN INVALUABLE LENS ON HOW WE CAN DO ABETTER JOB OF EDUCATING EACH CHILD.SO WE NEED TO DO EVERYTHING THAT WE CAN TO DEFEND AGAINST SECURITYBREACHES AND THREATS TO DATA QUALITY. WE REALLY HAVE TO BE ABLE TOTRUST THE INFORMATION THAT WE RECEIVE.YOU KNOW, THERE'S BEEN A LOT OF TALK TODAY ABOUT SORT OF TESTINGVERSUS TEACHING. AND TESTING, AS WE ALL KNOW, IS REALLY AN INTEGRALPART OF EDUCATION. TESTING AND TEACHING ARE NOT AT ODDS. THEY'VEGONE HAND-IN-HAND SINCE FORMAL EDUCATION BEGAN.THE ANSWER TO THE EXISTENCE OF TEACHING, AS MANY OF YOU SAID TODAY,IS NOT TO REJECT TESTING, IT'S TO DEAL WITH THE CHEATING.SECRETARY DUNCAN SAID NOT TOO LONG AGO, "THE EXISTENCE OF CHEATINGSAYS NOTHING ABOUT THE MERITS OF TESTING. INSTEAD, CHEATINGREFLECTS A WILLINGNESS TO LIE AT CHILDREN'S EXPENSE TO AVOIDACCOUNTABILITY, AN APPROACH THAT I REJECT ENTIRELY."279ARNE IS NOT ALONE.IT'S ALSO AN APPROACH THAT'S REJECTED BY THE VAST MAJORITY OFEDUCATORS, AS MANY OF YOU TODAY POINTED OUT--PEOPLE WHO WOULD NEVERPARTICIPATE IN OR EXCUSE TEACHING. CURRENT DATA SUGGESTS THATTEACHING IS AN ISSUE IN ABOUT 4% TO 5% OF ELEMENTARY SCHOOLCLASSROOMS. THE VAST MAJORITY OF EDUCATORS ARE BEHAVING IN ETHICALWAYS. THEY'RE ACTING WITH INTEGRITY. AND, MOST IMPORTANTLY,THEY'RE SERVING AS STRONG ROLE MODELS FOR OUR STUDENTS. BUT PUTTINGOUR HEADS IN THE SAND AND PRETENDING THAT CHEATING DOESN'T EXISTCLEARLY DOESN'T SERVE OUR CHILDREN WELL.CHEATING DOES OCCUR IN OUR SCHOOLS. AND WE NEED TO BE MORE VIGILANTAND MORE PREPARED. AND TODAY'S DISCUSSION HAS HELPED US START TOBUILD A ROAD MAP FOR WHAT TO DO AND HOW TO ADDRESS THIS. SO WHILE WEMAY NOT BE ABLE TO ELIMINATE CHEATING ENTIRELY, BY EMPLOYING ABUNCH OF THE STRATEGIES THAT WE HEARD TODAY, WE CAN HELP TO MAKESURE THAT IT DOESN'T OCCUR ON A SYSTEMIC LEVEL.SO I WANT TO JUST THANK YOU AGAIN FOR SHARING YOUR EXPERTISE WITHUS TODAY AND FOR YOUR LEADERSHIP IN THIS AREA. WE ALL LEARNED A LOTFROM YOU. AND WE AT THE DEPARTMENT PLEDGE OVER THE COMING WEEKS280TO PACKAGE THE EXPERTISE YOU'VE SHARED WITH US IN WAYS THAT AREUSEFUL TO PRACTITIONERS AND ACCESSIBLE TO PRACTITIONERS SO THATSTATES AND DISTRICTS HAVE THE BENEFIT OF THIS INFORMATION GOINGFORWARD.I'M NOT SURE IF JACK'S GOING TO WRAP UP, BUT LET ME WISH YOU SAFETRAVELS HOME. AND THANK YOU, AGAIN, FOR COMING, AND SHARING YOUREXPERTISE WITH US TODAY.WE GREATLY APPRECIATE IT.JACK BUCKLEY: THANKS VERY MUCH, JOANNE.[APPLAUSE]AT THIS POINT, I WILL NOT INFLICT ANOTHER SUMMARY ON YOU. SO JUST,AGAIN, LET ME ECHO JOANNE'S REMARKS THAT ON BEHALF OF THEDEPARTMENT, THE INSTITUTE OF EDUCATION SCIENCES, AND THE NATIONALCENTER FOR EDUCATION STATISTICS, THANK YOU VERY MUCH, PANELISTS,FOR YOUR PARTICIPATION AND YOUR OUTSTANDING REMARKS ANDASSISTANCE, AND AUDIENCE FOR YOUR THOUGHTFUL ATTENTION ANDWONDERFUL QUESTIONS.AND HAVE A GREAT DAY.281282