EVALUATION OF CONNECTICUT’S INTERDISTRICT MAGNET SCHOOLS Prepared by: Casey D. Cobb Robert Bifulco Courtney Bell The Center for Education Policy Analysis University of Connecticut Submitted: Connecticut State Department of Education January 2009 TABLE OF CONTENTS EXECUTIVE SUMMARY ES-1 SECTION I: INTRODUCTION SECTION II: 1-6 THE LEARNING ENVIRONMENTS IN INTERDISTRICT MAGNET SCHOOLS 2-9 SECTION III: THE PERCEPTIONS, ATTITUDES AND BEHAVIORS OF INTERDISTRICT MAGNET SCHOOL STUDENTS 3-20 SECTION IV: THE EFFECTS OF INTERDISTRICT MAGNET SCHOOLS ON STUDENT ACHIEVEMENT 4-44 APPENDIX A: HIGH SCHOOL STUDENT SURVEY A-1 APPENDIX B: SURVEY SAMPLING FRAME B-1 APPENDIX C: SCALE CONSTRUCTION AND INTERNAL CONSISTENCY C-1 APPENDIX D: DESCRIPTIVE SUMMARIES FOR MAGNET SCHOOLS AND STUDENTS D-1 APPENDIX E: STUDENT ACHIEVEMENT ANALYSIS E-1 i LIST OF TABLES TABLE 1.1 TABLE 2.1 TABLE 2.2 TABLE 2.3 TABLE 2.4 TABLE 2.5 TABLE 2.6 TABLE 2.7 TABLE 2.8 TABLE 2.9 TABLE 2.10 TABLE 2.11 TABLE 2.12 TABLE 2.13 TABLE 3.1 TABLE 3.2 TABLE 3.3 TABLE 3.4 TABLE 3.5 TABLE 3.6 TABLE 4.1 TABLE 4.2 TABLE 4.3 TABLE 4.4 Number of Interdistrict Magnet Schools by Location and Grade Level, 2007-08 Exposure of Black Students to White Students, 2007-08 Percent of Black Students from Urban Districts in Interdistrict Magnet Schools, 2007-08 Exposure of Hispanic Students to White Students, 2007-08 Percent of Hispanic Students from Urban Districts in Interdistrict Magnet Schools, 2007-08 Poverty Concentration, 2007-08 Percent of Minority Students from Urban Districts in Interdistrict Magnet Schools, 2007-08 Grade 3 CMT Scores, 2007-08 Grade 6 CMT Scores, 2007-08 Grade 10 CAPT Scores, 2007-08 Attendance in Schools Serving Central City Minority Students, 2006-07 Access to Advanced Classes in Middle Schools Serving Central City Minority Students, 2006-07 Retention and Dropouts in High Schools Serving Central City Minority Students, 2005-06 Access to Advanced Courses in High Schools Serving Central City Minority Students, 2005-06 Sample of Schools Participating in the Survey, Spring 2008 Students' Perception of School Climate Perception of School Climate Among Students Who Reside in Central City Student Attitudes and Behaviors Attitudes and Behaviors Among Students Who Reside in Central City Differences in Attitudes and Behaviors Between Magnet and Non-Magnet Students Who Reside in the Central City, Controlling for Family Background Characteristics Estimates of the Average Effect of Interdistrict Magnet Schools on Student Achievement, High Schools and Middle Schools Average Test Scores, By Student's Residence and Racial Background Estimated Magnet School Effects on Student Achievement, By Student's Residence Variation in Estimated Magnet School Effect on City Students, By Difference in Percent White Between Magnet and Home District ii 1-7 2-11 2-12 2-12 2-13 2-14 2-14 2-15 2-16 2-16 2-17 2-18 2-18 2-19 3-23 3-31 3-35 3-37 3-39 3-41 4-46 4-47 4-48 4-49 TABLE D.1 TABLE D.2 TABLE D.3 TABLE D.4 TABLE D.5 TABLE D.6 TABLED.7 TABLE D.8 TABLE D.9 TABLE D.10 TABLE D.11 TABLE D.12 TABLE D.13 TABLE D.14 TABLE D.15 TABLE E.1 TABLE E.2 TABLE E.3 TABLE E.4 Academic Press Item Scores School Influence on College Aspirations Item Scores Peer Academic Norms Item Scores Social Sanctions for Achievement Item Scores Teacher-Student Relationship Item Scores Classroom Disruption Item Scores Academic Aspirations Item Scores College Expectations Item Scores Attendance Item Scores Safety and Belonging Item Scores Equal Status, Interaction, Interdependence, and Supportive Norms Item Scores Social Closeness Measures Racial Tension at the School Close Friends by Race/Ethnicity Item Scores Future Multicultural Interest Item Scores Treatment and Comparison Group Samples Comparison of Lotteried-in and Lotteried-Out Students for Achievement Analysis Lottery Based Estimates of the Effect of Interdistrict Magnet Schools on Achievement Regression, Propensity Score and Lottery Based-Estimates of the Average Effect of Treatment of Treated D-1 D-2 D-3 D-3 D-4 D-4 D-5 D-6 D-7 D-7 D-8 D-9 D-10 D-11 D-12 E-3 E-6 E-9 E-12 LIST OF FIGURES FIGURE ES.1 Percent White in the Average Black Student’s School, High School, 2007-08 FIGURE ES.2 Percent White in the Average Hispanic Student’s School, High School, 2007-08 FIGURE ES.3 Percent Free-Lunch Eligible in the Average Minority Student’s School, High School, 2007-08 FIGURE 3.1 Analytic model: Variables measured by the High School Student Survey iii ES-1 ES-1 ES-2 3-28 EXECUTIVE SUMMARY As of October 2007, 54 interdistrict magnet schools enrolling 18,928 students were operating in Connecticut. The bulk of these schools are located in the Hartford and New Haven areas – 21 in the Hartford area and 17 in the New Haven area. Interdistrict magnets also serve significant numbers of students in the Waterbury region. In keeping with the mandate of Sheff v. O'Neill, the express purposes of interdistrict magnet schools are “to reduce, eliminate or prevent racial, ethnic or economic isolation while offering a high-quality curriculum that supports educational improvement.” This report examines the extent to which interdistrict magnet schools provide students in the cities of Hartford, New Haven, and Waterbury access to less isolated environments. In addition, we report the results of a student survey that provides information on intergroup relations, perceptions of teachers, academic engagement and other aspects of students’ experiences in interdistrict magnet schools. Finally, we estimate the impacts of attending an interdistrict magnet school on student achievement. The Learning Environments in Interdistrict Magnet Schools Interdistrict magnet schools clearly provide students of color from Connecticut’s most racially and economically isolated central cities the opportunity to join less isolated learning environments. The interdistrict magnet schools where central city minority students tend to enroll have higher percentages of White students and lower percentages of low-income students than the non-magnet schools located in the central city. Figures ES1 and ES2 compare the average percent White in interdistrict magnet schools where central city minority students tend to enroll to the corresponding average for non-magnet schools in the same city. These comparisons are at the high school level, but similar results are found at the elementary and middle school levels. Racial and ethnic isolation in Connecticut’s central city districts is very high. The students of color from these districts who attend interdistrict magnet schools are in considerably more integrated peer environments than their counterparts in central city district schools. Figure
ES2
‐
Percent
White
in
the
 Average
Hispanic
Student's
School,
 High
School,
2007‐08
 Figure
ES1
‐
Percent
White
in
the
 Average
Black
Student's
School,
 High
School,
2007‐08
 53.6
 60
 40
 20
 0
 22.7
 1.7
 Har.ord
 Non‐Magnets
 20.6
 24.3
 40.0
 20.0
 5.2
 53.6
 60.0
 22.9
 4.0
 21.2
 8.5
 25.2
 0.0
 New
Haven
 Waterbury
 Har.ord
 Interdistrict
Magnets
 Non‐Magnets
 ES-1 New
Haven
 Waterbury
 Interdistrict
Magnets
 Figure ES3 shows that the percent free-lunch eligible in the interdistrict magnet schools attended by central city students of color is much lower than in the central city district schools these students would otherwise attend, suggesting interdistrict magnet schools reduce economic isolation for their students. Figure
ES3
‐
Percent
Free‐Lunch
Eligible
in
the
 Average
Minority
Student's
School,
High
School,
 2007‐08
 100
 50
 69.2
 43.2
 70.7
 50.5
 66.1
 37.3
 0
 Har.ord
 Non
Magnet
 New
Haven
 Waterbury
 Interdistrict
Magnets
 Although interdistrict magnet schools are more diverse than central city district schools, they provide access to less isolated learning environments for only a small percentage of students of color in the state’s central cities, limiting the overall effect of the program on racial, ethnic, and economic isolation. For instance, only 12% of Black students and 6% of Hispanic students in Hartford attend magnet schools with more than 20% White students. Similarly, only 6% of minority students from Hartford attend magnet schools in which less than 40% of the students are eligible for free-lunch. In addition to examining whether interdistrict magnet schools provide access to less isolated environments, we also look at indicators of the extent to which interdistrict magnet schools provide their students positive learning environments and opportunities to learn. Compared to nearby, central city non-magnet schools, the interdistrict magnet schools where minority students from the central city are concentrated tend to have: • lower percentages of students scoring below proficiency, higher percentages of students scoring at or above goal, and higher average scale scores on CMT and CAPT exams; • higher rates of student attendance and fewer teacher absences; • higher percentages of 8th graders taking high school math and/or a world language; and • at the high school level, lower rates of in-grade retentions and lower dropout rates. The Perceptions, Attitudes, and Behaviors of Interdistrict Magnet School Students Magnet school environments are intended to serve students in two primary ways. First, they are meant to improve academic achievement. By bringing together students and teachers with similar interests to experience specialized, thematic curricula, magnet schools are designed to improve student engagement and academic aspirations. The second way magnet schools serve ES-2 students is by providing diverse learning environments. These environments are meant to enhance multicultural understandings and broaden worldviews, helping students develop the skills and orientations to successfully engage in our diverse society. A student questionnaire was designed to examine these fundamental assumptions and better understand the experiences of students in magnet high schools. Magnet school students generally report positive academic attitudes and behaviors, suggesting that Connecticut magnet high schools provide academic environments that are particularly conducive to learning. Comparisons between magnet schools and two types of non-magnet schools (city and suburban) reveal the following: • peer support for academic achievement is stronger in magnets than in non-magnet city schools; • 12th grade magnet city students perceive more encouragement and support for college attainment than 12th grade city students in non-magnets; • magnet students have similar academic aspirations, higher college expectations, and are less likely to be absent or skip classes than non-magnet city students; • magnet students demonstrate slightly lower college expectations, but report slightly higher academic aspirations and are less likely to skip class than non-magnet suburban students; and • and overall, magnet schools provide an academic climate similar to that found in a wealthy, suburban non-magnet high school. Taken together, these findings suggest that magnet schools are succeeding in their efforts to create positive academic learning environments for their students. Alongside these successes is some indication that teacher-student relationships and students’ sense of safety and belonging might be weaker in magnet schools than in some other schools. Among 9th graders, students in both city and suburban non-magnets report stronger student teacher relations than do students in the magnet schools. However, within the magnet schools, 12th grade students report stronger relationships with teachers than do 9th grade students, and there are no statistically significant differences in reported teacher student relationships across schools for the 12th graders. Students in the magnet schools feel a stronger sense of safety and belonging than do students in the city non-magnets and a weaker sense of safety and belonging than students in the suburban non-magnet. An examination of students’ multicultural attitudes and inclinations suggests that, on average, magnet schools are meeting their goal of helping all students develop more positive orientations toward multicultural issues. Student responses to the questionnaire suggest that: • city magnet students perceive more positive intergroup relations and less racial tension among their peers than do students in non-magnet city schools; • minority students in magnet city schools report feeling significantly more close to Whites and more likely to have multiple White friends than minorities in non-magnet city schools; ES-3 • White magnet students feel significantly more close to minority students and report higher percentages of multiple minority friends than do White students from the nonmagnet suburban school; and • magnet school students expressed stronger future multicultural interests and are significantly more likely than students in the suburban non-magnet school to report that their school experience helped them understand people from other groups. In the case of both academic and multicultural attitudes, it is possible these comparisons may simply be showing that magnet schools enroll students who are systematically different from non-magnet students. However, regression analyses that control for several home and family background characteristics reveal similar patterns. While we cannot be certain that magnets are causing improved attitudes, the differences are seen across a number of dimensions and are large enough to suggest that magnet schools are having positive influences on academic and racial attitudes. The Effects of Interdistrict Magnet Schools on Student Achievement In the final section of this report, we estimate the effects of magnet schools on academic achievement. These estimates use various techniques, including those that take advantage of randomized admission lotteries and longitudinal student achievement measures, to control for pre-existing differences between magnet school and non-magnet school students. The analyses focus on the effects of attending a magnet middle or high school in the Hartford, New Haven or Waterbury areas. The analysis indicates that interdistrict magnet schools have a statistically significant positive effect on the reading and math achievement of high school students, and on the reading achievement of middle school students. For high school, the estimated effects of two years of magnet school exposure are a 0.10 standard deviation gain in reading, and a 0.09 to 0.11 standard deviation gain in math. For three years exposure to a magnet school during the middle school grades, we see a gain of 0.18-0.19 standard deviations in reading. These results describe the average effect of magnet schools. Given Sheff’s focus on improving academic achievement for city students, we are particularly interested in how these effects may vary across students and schools. Our analyses suggest that interdistrict magnet high schools are particularly effective for city students regardless of the degree to which those schools decrease racial isolation. Specifically, attending an interdistrict magnet high school increases the 10th grade math achievement of city students, on average, by 0.12 standard deviations and the 10th grade reading achievement of city students by 0.15 standard deviations. Interdistrict magnet middle school effects differ from the high school effects. Interdistrict magnet middle schools have stronger effects on suburban students than city students; a result which suggests that these schools have not had as large an effect as interdistrict magnet high schools on the achievement of students from Connecticut’s cities. Additional analyses suggests that the average effect of middle schools on city students may be influenced by the degree to which the school is able to decrease racial isolation. City students experience positive effects in middle schools that have reduced racial isolation by 40% or more from their home district. ES-4 Conclusions This report highlights many positive aspects of Connecticut’s interdistrict magnet schools. • These schools provide access to less isolated learning environments for the minority students from Connecticut’s most isolated central cities than do the non-magnet schools located in those cities. • Magnet schools, on average, appear to provide an academic climate similar to that found in a wealthy, suburban high school, and peer support for academic achievement is stronger in magnet than in non-magnet city schools. • Magnet school students report more positive intergroup relations than in either the city or suburban non-magnet schools that we surveyed. • Focusing on city students, there were very few differences in attitudes between 9th grade magnet students and 9th grade students in non-magnet schools. Twelve graders in magnet schools, however, feel significantly closer to White students, are more likely to have multiple White friends, have higher expectations for college, have fewer absences and skip classes much less frequently than students attending non-magnet city schools. • On average, interdistrict magnet high schools have positive effects on both math and reading achievement, and interdistrict magnet middle schools have positive effects on reading achievement. Taken as a whole these findings indicate that interdistrict magnets are largely meeting their mission of providing learning environments that are both more diverse and more conducive to academic achievement than would otherwise be available to students in Connecticut’s central cities. There are also reasons to believe that these more diverse and academically oriented environments are associated with more positive intergroup attitudes and relations and improved academic performance for individual students. Along with these overall positive findings, the report also reveals some areas where magnet school operators and state policy makers might focus attention moving forward. Perceptions of teacher-student relationships and sense of safety and belonging were slightly lower among magnet students compared to some non-magnet schools. We encourage teachers and administrators working in the magnet schools to consider whether or not this finding is applicable to their students, and to search for ways they might address this concern. For policy makers, an important concern is that the proportion of students from the central cities who attend diverse interdistrict magnet schools is low. This is true in part because there are some magnet schools that have had only limited success attracting diverse student bodies, but more significantly because the number of seats in diverse magnet schools is limited. We encourage policy makers to continue to search for ways to reduce racial, ethnic, and economic isolation for a larger proportion of central city students. ES-5 SECTION I: INTRODUCTION An interdistrict magnet school is a publicly funded school operated by a local school district or a regional educational service center (RESC). Each magnet has an educational theme or focus, and students choose to enroll based on interest in the school’s theme. All students in the school districts participating in the magnet are eligible to attend. The operators of an interdistrict magnet school may limit the number of seats and must hold a lottery if there are more applicants than spaces.1 Interdistrict magnet schools are a key component of the remedial measures that the state of Connecticut has undertaken in response to Sheff v. O'Neill, 238 Conn. 1, 678 A.2d 1267 (1996). In that landmark decision, the Connecticut Supreme Court held that students in the Hartford public schools were racially, ethnically and economically isolated and that, as a result, Hartford public school students had not been provided a substantially equal educational opportunity under the state constitution. In keeping with the mandate of Sheff v. O'Neill, the express purposes of interdistrict magnet schools are “to reduce, eliminate or prevent racial, ethnic or economic isolation while offering a high-quality curriculum that supports educational improvement.”2 Although the first interdistrict magnet schools in Connecticut predate the Sheff v. O’Neill decision, funding and support provided by the state in response to the ruling has led to a marked expansion from a handful of schools in the mid-1990s to 54 schools serving 18,928 students by October of 2007.3 Of the 54 interdistrict magnets operating in 2007-08, five were phasing in, meaning that the school is in the process of converting to an interdistrict magnet, and only some of the grades or classes within the school have converted. Six of the interdistrict magnet high schools are half-time programs, where students attend part of the school day at the magnet and part in their home school. Two of these six half-time programs began accepting freshmen on a full-time basis for the 2007-08 year and one additional half-time program began accepting fulltime students for the 2008-09 school year. Counts of interdistrict magnet schools and students by grade level and location are provided in Table 1.1. Thirty-eight of the 54 magnets are located in the Hartford and New Haven areas and the magnets in these areas serve 69.6% of all magnet school students in the state. The Waterbury area also has substantial participation in four interdistrict magnet schools. This Report Section II of this report examines the extent to which interdistrict magnet schools have provided students in the cities of Hartford, New Haven, and Waterbury access to less isolated environments. In particular, we compared the racial/ethnic and socioeconomic composition of the typical interdistrict magnet schools attended by central city minority students to the composition of non-magnet schools in those students’ home districts. 1 Connecticut State Department of Education. (2006). Public School Choice in Connecticut: A Guide for Students and Their Families. Hartford, CT: Connecticut State Department of Education. 2 Ibid. 3 One interdistrict magnet has since lost its magnet status and eight additional interdistrict magnets were open during the 2008-09 school year. 1-6 Table 1.1 - Number of Interdistrict Magnet Schools by Location and Grade Level, 2007-08 Elementary Middle High School Total (Grades PK-5) (Grades 6-8) (Grades 9-12) Location Schoolsa Studentsc Schoolsb Studentsc Schoolsb Studentsc Schoolsb Studentsc Hartford Aread 21 7,388 8 2,468 7 2,028 11 2,892 New Haven Area 17 5,795 8 2,317 5 1,091 8 2,387 Waterbury Area 4 2,005 2 1,207 1 313 1 485 Bridgeport/Stamford Area 5 1,371 1 337 1 96 4 938 Other Areas 7 2,369 3 1,323 2 783 2 263 Total 54 18,928 22 7,652 16 4,311 26 6,965 a. Total number of schools does not equal sum of number at each grade level because several magnets serve students in more than one category. b. Includes all schools serving any students in the grade range. c. Student counts for October 2007. d. Students residing in Hartford are eligible to attend 20 of the schools in the Hartford area. The implicit theory underlying the goal of providing access to less racially, ethnically and economically isolated schools is that such schools can provide greater opportunities to learn. Thus, Section II also examines the extent to which interdistrict magnet schools have provided central city minority students access to environments more conducive to educational achievement. These comparisons show that interdistrict magnet schools provide environments for poor and minority students from the central cities that are considerably less isolated and arguably more conducive to academic achievement than these students would typically find in their home districts. However, we also find that the proportion of students from the central cities who attend diverse interdistrict magnet schools is low, which indicates that racial, ethnic, and economic isolation remains high for a large proportion of central city students. In Section III, we describe a survey of 9th and 12th grade students that we conducted and present analyses that compare the perceptions, attitudes, and self-reported behaviors of magnet school and non-magnet school students. These surveys provide information on students’ perceptions of academic climate, student-teacher relationships, peer norms, the quality of intergroup contact and relations, as well as students’ individual attitudes towards academic achievement and people from groups other than their own. Results from the analysis of this survey indicate that magnet schools provide an academic climate similar to that found in a wealthy, suburban non-magnet high school, and that the peer support for academic achievement is stronger in magnets than in non-magnet city schools. Also, students in magnet schools perceive more positive intergroup and racial relations than in either the city or suburban non-magnet schools in our sample. Differences between magnet schools and non-magnet city schools are found whether we focus on the perceptions of all students in magnet schools or on the perceptions of students who reside in one of Connecticut’s urban areas. These findings suggest that magnet schools are indeed succeeding in their efforts to create unique and positive learning environments for their students. 1-7 One area of concern raised by our survey is that teacher-student relationships and students’ sense of safety and belonging might be weaker in magnet schools than in some other schools Among 9th graders, students in both city and suburban non-magnets report stronger student teacher relations than do students in the magnet schools. However, within the magnet schools, 12th grade students report stronger relationships with teachers than do 9th grade students, and there are no statistically significant differences in reported teacher student relationships across schools for the 12th graders. Students in the magnet schools feel a stronger sense of safety and belonging than do students in the city non-magnets and a weaker sense of safety and belonging than students in the suburban non-magnet. The survey analysis also shows that, compared with city non-magnet students, magnet students have similar academic aspirations, higher expectations for post-secondary attainments, are less likely to be absent and are significantly less likely to skip classes. These differences between magnet and non-magnet city school students show up for both 9th and 12th graders, but are more marked among 12th graders. The results also indicate intergroup attitudes are more positive among magnet school students than non-magnet school students along several dimensions. In Section IV, we use longitudinal data on individual student test performance and information from admissions lotteries to estimate the achievement impacts of interdistrict magnet middle and high schools in the Hartford, New Haven and Waterbury areas. Results indicate that, on average, interdistrict magnet high schools have positive effects on both math and reading achievement, and interdistrict magnet middle schools have positive effects on reading achievement. Extensions of our analysis indicate that interdistrict magnet high schools have positive effects particularly on the achievement of central city students, and do so regardless of how much attending an interdistrict magnet high school reduces racial isolation. The positive effects of magnet middle schools appear to be limited to suburban students, except in those schools that are able to achieve substantial reductions in racial isolation for their central city students. 1-8 SECTION II: THE LEARNING ENVIRONMENTS IN INTERDISTRICT MAGNET SCHOOLS A primary purpose of interdistrict magnet schools is to provide students in Connecticut’s large, central cities access to less racially, ethnically, and economically isolated schools. In this section we examine the extent to which interdistrict magnet schools in the Hartford, New Haven, and Waterbury areas have achieved this goal. More specifically, we address two questions: • Do interdistrict magnet schools provide minority students in Connecticut’s central cities access to less racially and ethnically isolated schools? • Do interdistrict magnet schools decrease the extent to which minority students from Connecticut’s central cities attend schools of concentrated poverty? The implicit theory underlying the goal of reducing racial, ethnic, and economic isolation is that schools with less isolated environments can provide poor and minority students richer learning opportunities.4 Thus, in addition to examining whether interdistrict magnet schools provide access to less isolated environments, we also look at other indicators of the extent to which interdistrict magnet schools provide their students more positive learning environments. Specifically, we address a third question: • Do key elements of the environment in the interdistrict magnet schools attended by central city minority students differ substantially from those aspects of the environment found in non-magnet city schools? The analysis that follows indicates that interdistrict magnet schools clearly provide students of color from Connecticut’s most racially and economically isolated central cities the opportunity to join less isolated learning environments. The interdistrict magnet schools where central city minority students tend to enroll have higher percentages of White students and lower percentages of low-income students than the non-magnet schools located in the central city. In addition, in comparison with central city schools, magnet schools have fewer low achieving students, higher rates of attendance, fewer teacher absences, lower rates of in-grade retentions and dropping out, and higher shares of students taking advanced classes. We also find, however, that the proportion of students from the central cities who attend diverse interdistrict magnet schools is low, which indicates that racial, ethnic, and economic isolation remains high for a large proportion of central city students. Data and Measures The data used for this section were drawn from the Strategic School Profiles compiled by the Connecticut State Department of Education (CSDE). These data provide information on the racial and ethnic composition and the share of students eligible for free or reduced price lunch in 4 More diverse environments also provide benefits for higher income and White students. Given the focus of the original Sheff decision on the isolation of low income and minority students from the central city, the analyses here concentrate primarily on the schooling environments for those students. 2-9 individual schools, as well as indicators of varying aspects of school environment including average student attendance, average teacher attendance, the percent of students taking advanced classes, retention rates, and dropout rates. CSDE was able to provide information on racial composition and free-lunch eligibility for the 2007-08 school year, which are used in the analyses presented below. At the time that this report was prepared, the most recent year for which measures of other elements of school environment were available is 2006-07, and in some cases, 2005-06. In addition to data from the Strategic School Profiles, we use information from the Magnet School Racial Survey by Town of Residence reports also provided by CSDE. These data provide counts of students enrolled in each magnet school by town and by racial and ethnic group within each town.5 Perhaps the simplest approach to determining whether interdistrict magnet schools provide less isolated schooling options for central city students of color is to compare the average racial composition, share of free-lunch eligible students, or other elements of school environment for interdistrict magnet schools to the average of those same measures for non-magnet schools in the central cities.6 If, however, minority students from a given central city are more heavily represented in some interdistrict magnet schools than others, a simple average for interdistrict magnet schools might not represent the typical experience for central city students of color. Likewise, if minority students are not evenly distributed across non-magnet schools, the average characteristics of non-magnet schools might not represent the typical experience of minority students.7 Thus rather than comparing simple averages across interdistrict magnet and non-magnet schools, we calculate a series of what are known as exposure indices. These measures indicate the exposure that the typical central city minority student has to members of another group or to some specified school characteristic. Given counts of minority students residing in the central city (M), of students in some specified group (N), and of all students (T) in each school, the exposure index (E) for a set of i schools can be computed as: (1) This measure is a weighted average of the percent of students from Group N for a given set of schools, where schools with more minority students from the central city are weighted more 5 Both the Strategic School Profile and Magnet School Racial Survey by Town of Residence, and thus this report, uses state-designated racial/ethnic classifications (American Indian, Asian, Black, White and Hispanic). We acknowledge the limitations these descriptors have with respect to portraying a student’s ethnic heritage or racial identity. 6 Throughout we use the term non-magnet school to refer to a public school that is not an interdistrict magnet. Some of these schools we are calling non-magnet schools are intradistrict magnet schools. 7 A second issue also limits the conclusions that one can draw from comparisons of simple averages. Because the students enrolled in interdistrict magnet schools might otherwise have enrolled in non-magnet schools in the central city, the presence of magnet schools influences the student composition of non-magnet schools. Thus, the average conditions in the central city district’s non-magnet schools do not necessarily represent the conditions interdistrict magnet school students would encounter in the absence of magnet schools. The data available for this analysis do not allow us to address this issue. 2-10 heavily than schools with fewer minority students from the central city. This exposure index can be interpreted as the percentage of Group N students in the typical central city minority student’s school.8 Do Interdistrict Magnet Schools Provide Minority Students in Connecticut’s Central Cities Access to Less Racially and Ethnically Isolated Schools? Table 2.1 compares the exposure of Black students to White students in central city school districts to the same measure of exposure for interdistrict magnet schools that serve Hartford, New Haven, and Waterbury students. For each of the three areas examined in Table 2.1 and at each grade level, exposure of centralcity Black students to White students is many times greater in the area’s interdistrict magnet schools than in the central city’s non-magnet schools. These results indicate that interdistrict magnet schools do indeed provide substantially less racially isolated environments for Black students from Connecticut’s central cities than is available in their city’s non-magnet schools. Table 2.1 - Exposure of Black Students to White Students, 2007-08 Non-Magnet Schoolsa Interdistrict Magnetsb Elementary Schools Hartford 2.7 14.6 New Haven 7.6 18.1 Waterbury 21.5 53.8 Middle Schools Hartford 0.9 33.3 New Haven 5.9 26.0 Waterbury 22.4 49.8 High Schools Hartford 1.7 22.7 New Haven 5.2 20.6 Waterbury 24.3 53.6 a. Weighted average of percent White in non-magnet schools. Weights based on the number of Black students in the school. b. Weighted average of percent White in interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of Black students from that city in the magnet school. In order to fully assess the extent to which interdistrict magnet schools provide access to less racially isolated environments, we also need to know how many central city minority students are gaining access to less isolated schools. Table 2.2 shows the percentage of Black students from Hartford, New Haven, and Waterbury who are attending interdistrict magnet schools generally, and the percentage attending interdistrict magnets that have more than 10% White 8 To calculate exposure to some specified school characteristic, the term Ni/Ti (i.e., the share of group N students in school i) is replaced by a measure of the specific school characteristic such as average student attendance or percent of students taking advanced courses. The resulting measures are interpreted as average attendance or percent taking advanced courses in the typical central city minority student’s school. 2-11 students, more than 20% White students, and more than 30% White students. More than 20% of Black students in Hartford and New Haven are attending interdistrict magnet schools, and most of these students are attending interdistrict magnet schools that have more than 10% White students. However, only 12.2% of Black students in Hartford and 8.5% of Black students in New Haven are attending interdistrict magnet schools that are more than 20% White. Even more striking, only 3.5% of Black students in Hartford and only 1.5% in New Haven are attending interdistrict magnet schools that are more than 30% White. Thus, although interdistrict schools are less racially isolated than the district schools in Hartford and New Haven, the limited number of seats available in diverse interdistrict magnet schools has limited the number of students who gain access to these less racially isolated environments. In Waterbury, 8.2% of Black students are attending magnet schools, and all of these students are in magnet schools that are more than 30% White. Table 2.2 - Percent of Black Students from Urban Districts in Interdistrict Magnet Schools, 2007-08 Hartford New Haven Waterbury % in Magnets 20.8% 21.8% 8.2% % in Magnets >10% Whitea 17.5% 19.7% 8.2% % in Magnets >20% Whitea 12.2% 8.5% 8.2% % in Magnets >30% Whitea 3.5% 1.5% 8.2% Table 2.3 compares the exposure of Hispanic students to White students in interdistrict magnet schools and in non-magnet schools in the central city. As in the case of Black students, interdistrict magnet schools provide Hispanic students significantly greater exposure to White students in each city and at each grade level. Table 2.3 - Exposure of Hispanic Students to White Students, 2007-08 Non-Magnet Schools Interdistrict Magnetsa Elementary Schools Hartford 4.7 17.3 New Haven 6.4 18.0 Waterbury 21.0 53.7 Middle Schools Hartford 2.3 34.0 New Haven 5.9 26.3 Waterbury 23.2 49.8 High Schools Hartford 4.0 22.9 New Haven 8.5 21.2 Waterbury 25.2 53.6 a. Weighted average of percent White in non-magnet schools. Weights based on the number of Hispanic students in the school. b. Weighted average of percent White in interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of Hispanic students from that city in the magnet school. 2-12 Table 2.4 indicates the percent of Hispanic students in each city who attend interdistrict magnet schools and interdistrict magnet schools with more than 10% White, more than 20% White, and more than 30% White. The percentage of Hispanic students in these cities who attend interdistrict magnet schools is significantly lower than the percentage of Black students. As a result, an even lower percentage of Hispanic students in Connecticut’s largest and most ethnically isolated cities have gained exposure to non-minority students through interdistrict magnet schools. Only 10.5% of Hispanic students in Hartford, 12.5% of Hispanic students in New Haven, and 5.1% of Hispanic students in Waterbury attend interdistrict magnet schools that are more than 10% White. Only 5.7% of Hispanic students in Hartford and 6.0% of Hispanic students in New Haven attend an interdistrict magnet school that is more than 20% White, and less than 1% of Hispanics in these two cities attend interdistricts magnet schools that are more than 30% White. Table 2.4 - Percent of Hispanic Students from Urban Districts in Interdistrict Magnet Schools, 2007-08 Hartford New Haven Waterbury % in Magnets 10.9% 13.3% 5.1% % in Magnets >10% White 10.5% 12.5% 5.1% % in Magnets >20% Whitea 5.7% 6.0% 5.1% % in Magnets >30% Whitea 1.1% 0.5% 5.1% Do Interdistrict Magnet Schools Decrease the Extent to Which Minority Students from Connecticut’s Central Cities Attend Schools of Concentrated Poverty? Table 2.5 compares the level of poverty concentration in interdistrict magnets and in non-magnet schools. More specifically, for each city and grade level, Table 2.5 compares the average percent of students who are free-lunch eligible in the non-magnet city schools most heavily attended by minority students to the average percent of students who are free-lunch eligible in the interdistrict magnet schools most heavily attended by central city minority students.9 These comparisons indicate that in Hartford and Waterbury, across all grade levels, interdistrict magnet schools provide much less economically isolated environments than the non-magnet schools that central city minorities attend. The differences in the level of economic isolation between interdistrict magnets and non-magnets are smaller in New Haven. At the elementary and high school levels, the interdistrict magnet schools attended most heavily by minority students from New Haven have lower percentages of free-lunch eligible students than non-magnet schools in New Haven. However, interdistrict magnet middle schools in New Haven have somewhat higher percentages of low-income students than the typical non-magnet middle school in New Haven. Table 2.6 indicates the percent of minority students in each city who attend interdistrict magnet schools and interdistrict magnet schools with fewer than 50% free-lunch eligible students, fewer than 40% free-lunch eligible students, and fewer than 30% free-lunch eligible students. Only 7.4% of minority students in Hartford, 5.0% of minority students in New Haven, and 6.3% of 9 Specifically the index of exposure of minority students to free-lunch eligible students is computed as described above. This index can be interpreted as the percent free-lunch eligible in the school attended by the typical minority student. 2-13 minority students in Waterbury attend interdistrict magnet schools in which the share of freelunch eligible students is less than 50%. Even smaller percentages gain access to schools that are less than 40% and less than 30% free-lunch eligible. Table 2.5 - Poverty Concentration, 2007-08 Non-Magnet Schoolsa Interdistrict Magnetsb Elementary Schools Hartford 72.4% 45.5% New Haven 77.0% 67.8% Waterbury 82.7% 37.9% Middle Schools Hartford 84.6% 32.6% New Haven 67.6% 72.3% Waterbury 75.5% 39.9% High Schools Hartford 69.2% 43.2% New Haven 70.7% 50.5% Waterbury 66.1% 37.3% a. Weighted average of percent free-lunch eligible in district schools. Weights based on the number of minority students in the school. b. Weighted average of percent free-lunch eligible in interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of minority students from that city in the magnet school. Hartford New Haven Waterbury Table 2.6 - Percent of Minority Students from Urban Districts in Interdistrict Magnet Schools, 2007-08 % in Magnets % in Magnets % in Magnets <50% Free-Lunch <40% Free-Lunch <30% Free-Lunch % in Magnets Elig. Elig. Elig. 1.9% 15.2% 7.4% 5.7% 2.1% 18.2% 5.0% 2.5% 0.0% 6.3% 6.3% 6.3% Do the Learning Environments in Interdistrict Magnet Schools Differ Substantially from What Central City Minority Students Would Otherwise Encounter? The implicit theory underlying the goal of reducing racial, ethnic, and economic isolation is that schools with less isolated environments can provide poor and minority students stronger opportunities to learn. We do not try to comprehensively assess the learning environments in interdistrict magnet schools. Rather, we focus on a small number of indicators that are readily accessible from administrative data maintained by CSDE, that are plausibly related to the amount of racial and ethnic isolation in a school, and that are likely to influence learning environments. Schools that have relatively few low-performing students might allow teachers in the school to spend less time on basic and remedial instruction and more time on advanced topics or 2-14 enrichment activities. Also, schools with higher levels of achievement might foster higher expectations for students among teachers and stronger norms favoring academic effort among students. For these reasons, the average level of achievement in a school might be an indicator of the learning environment. Other indicators that are likely to influence the learning environment include average attendance of students, average attendance of teachers, percent of students taking advanced classes, and at the high school level, retention rates and dropout rates. As shown in Tables 2.7-2.9, across all grade levels and subjects, minority students from the central cities who attend interdistrict magnet schools are exposed to higher achieving students, on average, than their counterparts enrolled in non-magnet schools. In many instances, the differences in academic peer environment between interdistrict magnets and non-magnets are large. Table 2.7 - Grade 3 CMT Scores, 2007-08 Non-Magnet Schoolsa Average scale % at goal % proficient score Interdistrict Magnetsb % at goal % proficient Average scale score Mathematics Hartford 22.4 48.6 206.2 30.6 62.4 220.7 New Haven 32.7 60.4 219.8 45.9 74.1 238.0 Waterbury 39.4 67.4 229.3 55.1 85.1 251.5 Reading Hartford 15.3 31.6 199.4 25.4 45.9 215.3 New Haven 17.3 32.7 200.8 29.1 48.1 216.7 Waterbury 25.0 46.2 211.5 47.9 66.7 233.0 Writing Hartford 31.6 59.4 221.7 48.7 75.9 236.8 New Haven 32.5 61.5 222.2 51.4 77.2 239.6 Waterbury 42.3 68.6 230.3 69.6 88.3 254.2 a. Weighted average of percent scoring proficient in non-magnet schools. Weights based on the number of minority students enrolled in the school. b. Weighted average of percent scoring proficient in interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of minority students from that city in the magnet school. Microsociety Magnet Elementary is not included because 2007-08 test score results were not available. Among 3rd graders (Table 2.7), the share of students scoring at or above proficiency in the interdistrict magnet schools that students from Hartford and New Haven attend are, on average, about 15 percentage points higher than in the non-magnet schools in those cities, and in Waterbury the difference between interdistrict magnets and non-magnets is closer to 20 percentage points. Among 6th graders (Table 2.8), the share of students at or above goal is nearly twice as high in the interdistrict magnet schools that students from Hartford and Waterbury attend than in the non-magnet schools in those cities. The high proportion of students achieving at goal indicates that a high proportion of students are in supportive academic environments in interdistrict magnet schools. 2-15 Table 2.8 - Grade 6 CMT Scores, 2007-08 Non-Magnet Schoolsa Interdistrict Magnetsb Average % scale % Average % at goal proficient score % at goal proficient scale score Mathematics Hartford 26.5 54.7 219.3 54.8 81.4 251.1 New Haven 39.2 66.2 231.1 48.7 75.2 241.8 Waterbury 32.4 59.2 224.7 62.7 82.4 253.2 Reading Hartford 24.8 40.1 212.0 59.3 72.8 248.1 New Haven 34.0 47.9 221.3 48.3 63.8 235.5 Waterbury 32.5 48.8 220.1 73.5 89.2 257.6 Writing Hartford 32.5 64.4 223.5 64.5 87.4 255.3 New Haven 29.4 60.7 220.9 38.0 72.9 228.1 Waterbury 27.5 55.9 216.9 61.8 86.3 248.5 a. Weighted average of percent scoring proficient in non-magnet schools. Weights based on the number of minority students enrolled in the school. b. Weighted average of percent scoring proficient in interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of minority students from that city in the magnet school. Table 2.9 - Grade 10 CAPT Scores, 2007-08 Non-Magnet Schoolsa Interdistrict Magnetsb Average % scale % Average % at goal proficient score % at goal proficient scale score Mathematics Hartford 4.5 34.8 197.3 36.3 73.6 242.8 New Haven 12.9 39.0 206.4 16.5 54.2 221.9 Waterbury 12.6 46.1 211.6 30.8 71.8 239.8 Reading Hartford 5.0 40.3 192.1 26.1 80.4 230.1 New Haven 14.4 52.2 206.7 18.8 70.6 221.8 Waterbury 13.8 55.2 207.6 28.8 85.6 235.7 Writing Hartford 12.0 56.6 212.9 52.4 90.0 252.8 New Haven 19.6 64.7 222.1 36.9 86.8 243.3 Waterbury 27.8 76.7 230.6 44.9 89.8 251.5 Science Hartford 12.0 56.6 212.9 52.4 90.0 252.8 New Haven 19.6 64.7 222.1 36.9 86.8 243.3 Waterbury 27.8 76.7 230.6 44.9 89.8 251.5 a. Weighted average of percent scoring proficient in non-magnet schools. Weights based on the number of minority students enrolled in the school. b. Weighted average of percent scoring proficient in interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of minority students from that city in the magnet school. Half-day interdistrict magnet schools not included. 2-16 Differences between interdistrict magnet schools and non-magnet schools are perhaps most marked among 10th graders in Hartford (Table 2.9). The shares of students achieving proficiency in math and in reading in the interdistrict magnet high schools that students from Hartford attend are more than twice that in Hartford’s non-magnet high schools. And the share at or above goal in these subjects is more than five times as high in interdistrict magnet schools as in the non-magnets. Part of the higher levels of achievement in interdistrict magnet schools might reflect superior teaching and learning. Very likely, however, part of the differences between magnet and nonmagnet schools is due to magnet schools attracting students who are better prepared and more motivated to start. The analysis in Section IV is designed to isolate how much of the difference in achievement between magnet and non-magnet schools can be attributed to superior teaching and learning. Tables 2.10 - 2.12 present other indicators of academic climate and opportunities to learn in interdistrict magnet schools. Table 2.10 compares student and teacher attendance at the interdistrict magnet schools that serve central city students to attendance at non-magnet city schools. High rates of both student and teacher absences can disrupt instruction and reduce time spent on learning activities. In addition, attendance rates might be influenced by the quality of the school environment more generally. The results in Table 2.10 indicate that the interdistrict magnet schools attended by central city students have higher rates of student attendance and fewer teacher absences than non-magnets from the same city. Table 2.10 - Attendance in Schools Serving Central City Minority Students, 2006-07 Percent of students in attendance, October 1 Classroom teacher average days absent Non-Magnet Interdistrict Non-Magnet Interdistrict Schoolsa Magnetsb Schoolsa Magnetsb Elementary Schools Hartford New Haven Waterbury Middle Schools Hartford New Haven Waterbury 93.7 94.1 95.7 95.1* 96.5** 96.6 9.6 10.0 11.8 8.5 8.4** 10.1* 91.7 93.5 92.1 96.2** 94.3 95.9** 9.5 10.1 11.9 5.9** 10.8 9.2 High Schools Hartford 82.0 95.7** 10.0 7.3** New Haven 87.0 91.1* 9.9 8.1** Waterbury 90.1 95.6* 10.7 8.6* a. Weighted average for non-magnet schools in the central city, weights based on the number of minority students enrolled in the school. b. Weighted average for interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of minority students from that city in the magnet school. Half-day interdistrict magnet schools not included. * indicates difference between interdistrict magnets and non-magnets is statistically significant at 0.10 level. ** indicates difference between interdistrict magnets and non-magnets is statistically significant at 0.05 level. 2-17 Table 2.11 compares the percentage of 8th grade students taking advanced classes in the interdistrict magnet schools serving central city minority students and in their counterpart nonmagnet schools. Having a high percentage of students taking advanced classes may make it is easier for administrators to justify devoting resources to those classes, and also may reflect an environment that encourages students to pursue more challenging coursework. In both Hartford and New Haven, far higher percentages of the 8th grade students in interdistrict magnet schools take advanced classes than in non-magnet city schools. The percentage taking high school level math is also higher in the Waterbury interdistrict magnet middle school than in the nearby city schools, but the difference in Waterbury is not as marked as in Hartford and New Haven. Table 2.11 - Access to Advanced Classes in Middle Schools Serving Central City Minority Students, 2006-07 Percent of 8th graders taking high school Percent of 8th graders taking a world level math language class Non-Magnet Interdistrict Non-Magnet Interdistrict Schoolsa Magnetsb Schoolsa Magnetsb Hartford 7.8 34.6** 4.1 50.7* New Haven 14.1 44.3** 12.4 59.6** Waterbury 12.7 26.7 NA 26.7 a. Weighted average for non-magnet schools in the central city, weights based on the number of minority students enrolled in the school. Includes only non-high schools serving8th graders. b. Weighted average for interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of minority students from that city in the magnet school. Half-day interdistrict magnet schools not included. Includes only non-high schools serving 8th graders. * indicates difference between interdistrict magnets and non-magnets is statistically significant at 0.10 level. ** indicates difference between interdistrict magnets and non-magnets is statistically significant at 0.05 level. Table 2.12 examines retention and dropout rates. The interdistrict magnet high schools that serve central city students have far lower rates of retention and dropping out than the nearby nonmagnet city schools. Table 2.12 - Retention and Dropouts in High Schools Serving Central City Minority Students, 2005-06 Percent of students retained in grade Dropout rates for grades 9-12 Non-Magnet Interdistrict Non-Magnet Interdistrict Schoolsa Magnetsb Schoolsa Magnetsb Hartford 22.7 4.4** 7.5 0.7** New Haven 11.8 8.9 5.6 2.1** Waterbury 1.6 0.0** 3.0 0.0** a. Weighted average for non-magnet schools in the central city, weights based on the number of minority students enrolled in the school. Includes only non-high schools serving 8th graders. b. Weighted average for interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of minority students from that city in the magnet school. Half-day interdistrict magnet schools not included. Includes only non-high school serving 8th graders. * indicates difference between interdistrict magnets and non-magnets is statistically significant at 0.10 level. ** indicates difference between interdistrict magnets and non-magnets is statistically significant at 0.05 level. Finally Table 2.13 examines the percentages of students taking advanced classes in interdistrict magnet high schools and in non-magnet high schools. Nearly 40% of juniors and seniors in Hartford’s non-magnet high schools take courses for college credit. This relatively high percentage suggests that providing access to college courses in high school is a need adequately 2-18 met by non-magnet schools in Hartford, and thus, not a high priority for interdistrict magnet schools serving Hartford students. The percent of students who take courses for college credit is indeed lower in the interdistrict magnet schools where minority students from Hartford are concentrated than in the Hartford non-magnet high schools. In contrast, the interdistrict magnet high schools serving New Haven students have higher percentages of students taking advanced courses than the nearby non-magnet, city high schools. Table 2.13 - Access to Advanced Courses in High Schools Serving Central City Minority Students, 2005-06 Percent of juniors & seniors enrolled in a Percent of seniors who took at least one course for college credit AP test Non-Magnet Interdistrict Non-Magnet Schoolsa Interdistrict Magnetsb Schoolsa Magnetsb Hartford 38.9 11.5** 13.1 19.0 New Haven 8.5 16.5 22.7 31.5* Waterbury 17.1 NA 15.3 NA a. Weighted average for non-magnet schools in the central city, weights based on the number of minority students enrolled in the school. Includes only non-high schools serving 8th graders. b. Weighted average for interdistrict magnet schools that students from the city indicated are eligible to attend. Weights based on the number of minority students from that city in the magnet school. Half-day interdistrict magnet schools not included. Includes only non-high school serving 8th graders. * indicates difference between interdistrict magnets and non-magnets is statistically significant at 0.10 level. ** indicates difference between interdistrict magnets and non-magnets is statistically significant at 0.05 level. Conclusions Clearly interdistrict magnet schools provide less isolated learning environments for the minority students from Connecticut’s most isolated central cities than do the non-magnet schools located in those cities. Interdistrict magnet schools have higher percentages of White students and lower percentages of low-income students than the nearby city schools. Also, compared to nonmagnet, city schools, the interdistrict magnet schools where minority students from the central city are concentrated tend to have: • lower percentages of students scoring below proficiency, higher percentages of students scoring at or above goal, and higher average scales scores on CMT and CAPT exams; • higher rates of student attendance and fewer teacher absences; • higher percentages of 8th graders taking high school math and/or a world language; and • at the high school level, lower rates of retentions in grade and lower dropout rates. However, interdistrict magnet schools provide access to less isolated learning environments for only a small percentage of minority students in the state’s central cities, limiting the overall effect of the program on racial, ethnic, and economic isolation. 2-19 SECTION III: THE PERCEPTIONS, ATTITUDES AND BEHAVIORS OF INTERDISTRICT MAGNET SCHOOL STUDENTS Magnet school environments are intended to serve students in two primary ways. First, they are meant to improve academic achievement. This is done by bringing together students and teachers with similar academic or vocational interests to experience specialized, thematic curricula. This thematic approach is expected to improve student learning through increased engagement with school. A community with a common purpose is expected to instill higher academic value orientations and elevate academic aspirations—both preconditions of student learning. Second, by providing diverse learning environments, magnet schools are meant to enhance multicultural understandings and broaden worldviews in ways that better prepare students to access opportunities and play constructive roles in our increasingly diverse society. A student questionnaire was designed to examine these fundamental assumptions and better understand the experiences of students in magnet high schools. The results from the survey complement the more traditional analysis of achievement presented in Section IV of this report. Schools are frequently assessed based on the performance of their students on standardized tests. However schools affect students in other ways that may influence performance on standardized assessments, but which also may have direct influences on future academic and vocational pursuits. The questionnaire reported on here provides alternative indicators of student outcomes and attainment, and attempts to find linkages between these indicators and school environments. The main goals of the analyses presented in this section are: • To compare school climates in Connecticut’s magnet high schools with non-magnet high schools with respect to academic orientation, teacher-student relationships, intergroup relations, comfort, and safety. • To compare attitudes of magnet high school students with non-magnet high school students with respect to academic aspirations, effort in school, attitudes towards members of other groups, and multicultural interests. Given the focus of interdistrict magnet schools on improving opportunities and educational outcomes for low income, minority students in the state’s central cities, we are particularly concerned with examining the experiences and attitudes of students who reside in the central city. The results suggest that Connecticut magnet high schools provide academic environments that are particularly conducive to learning. Magnet schools, on average, appear to provide an academic climate similar to that found in a wealthy, suburban high school, and peer support for academic achievement is stronger in magnet than in non-magnet city schools. Magnet students have similar academic aspirations, higher college expectations, and are less likely to be absent or skip classes than non-magnet city students. Magnet students demonstrate slightly lower college expectations, but report slightly higher academic aspirations and are less likely to skip class than 3-20 non-magnet suburban students. Moreover, magnet school students report more positive intergroup relations than in either the city or suburban non-magnet schools that we surveyed. White magnet students feel significantly more close to minority students and report higher percentages of multiple minority friends than do White students from the non-magnet suburban school. Magnet school students expressed stronger future multicultural interests and are significantly more likely than students in the suburban non-magnet school to report that their school experience helped them understand people from other groups. Alongside these successes is some indication that teacher-student relationships are relatively weaker among 9th grade magnet students. Finally, students in the magnet schools feel a stronger sense of safety and belonging than do students in the city non-magnets and a weaker sense of safety and belonging than students in the suburban non-magnet. Data Collection In this section, we describe the origins and development of the survey, our sampling strategy, the schools and students who participated in the survey, and survey administration procedures. Questionnaire Development During the winter and spring of 2007, our research team designed a prototype for the high school student survey. Previous discussions with the CSDE Bureau of Educational Equity guided our overall charge. That charge was to capture non-traditional indicators of student outcomes and school performance. We considered the theories of action underlying magnet school programming and began the iterative process of identifying measures that could begin to test those assumptions. Our understanding of Connecticut’s interdistrict magnet school program, our own experiences and expertise, and an expansive body of academic literature informed the survey’s design. We were interested in developing items that measured student engagement, academic norms, racial attitudes, sense of belonging, classroom environment, safety, teacher-student relationships, and career expectations, among other indicators. To do so, we consulted the research literature, existing survey instruments and measurement scales, as well as a number of experts with measurement and content area knowledge. Survey items were borrowed, adapted, or developed for each area of interest, and continually reviewed and refined. With each item, or collection of items, we sought clarity, validity, and authenticity. We were mindful of the age level and cultural diversity among our target populations. In some cases, entire batteries of items were eliminated in the interest of economy or in being sensitive to student perspectives. Throughout the piloting and final administrations, we operated with approval from the University of Connecticut Institutional Review Board, which reviews all research involving human subjects. In the summer and fall of 2007, a 130-item survey was piloted with nearly 200 students. The pilot led to refinements and preliminary validation evidence for the instrument. Factor analysis of item responses revealed over a dozen coherent measurement constructs. Further revisions resulted in a final 100-item survey that was available in both print and online form, and in both English and Spanish. A copy of the final survey instrument is included in this report as Appendix A (English version). 3-21 Survey Sample and Administration Procedures We targeted magnet high schools and comparison high schools located primarily in the Hartford and New Haven regions. This sampling strategy was chosen because the majority of the magnet schools are located in these areas. These schools are the major focus of the Sheff settlement agreement. Our sampling strategy excluded participation from five magnet high schools located in Norwalk, Waterbury, Bridgeport and Stamford. We worked with CSDE personnel, regional service center administrators, district administrators, and principals to arrange administration of the surveys at participating schools. Of the 21 Connecticut magnet high schools in the Hartford and New Haven region, 13 completed the survey. Of the eight that did not participate, three declined participation or did not respond to multiple messages, three agreed to participate but did not set a date for the survey, and two asked to administer the survey the following year. Three of the magnet schools offer half-day programs and one additional magnet school is relatively new. Five of the magnets are administered by a Regional Educational Service Center (RESC) and the remaining eight are operated by a school district (three by New Haven, two by Hartford, one by Bloomfield, and one by East Hartford). See Appendix B for a complete listing of the population of high school magnets and their status with the survey. In addition, 11 non-magnet high schools were identified for comparison purposes. These schools were selected on the basis of their geographic location, urbanicity and racial composition. Four schools agreed to administer the survey; however, only three completed the task in spring 2008. Several others declined or did not return messages. Of the three participating schools, two were predominantly minority urban high schools and the third a predominantly non-minority suburban high school. We were interested in surveying schools located in districts adjacent to central cities (sometimes referred to as inner ring districts in the Sheff region), but we were not successful in recruiting any of these schools. By the end of the 2008 academic year we had administered the survey to 16 schools (13 magnet and three non-magnet). For each school, we targeted 9th and 12th graders for participation in the survey. In one school that did not serve 12th graders we targeted 11th graders instead and in one school that did not serve 9th graders we targeted 10th graders. In the magnet schools, which are considerably smaller than nearby non-magnet high schools, we attempted to survey all students in the targeted grades. The size of the non-magnet schools that we surveyed made targeting all students in a grade infeasible. In these cases, we worked with principals to obtain a reasonably representative sample of classrooms. For instance, we asked them to identify a set of English classrooms that would best capture the variability of students at their school. If 20% of their students were in the high academic track, 50% in the college prep track, and 30% in a basic curriculum, we sought those proportions in the school sample. At five of the schools in the sample, questionnaires were completed by students online in school computer labs during designated periods of the day. In eight schools, surveys were completed using paper and pencil. Data entry and analysis was conducted over the summer and early fall of 2008. 3-22 It is difficult to determine if the school samples are representative of their respective student populations. Students who took the survey may be different from those who declined participation (or did not have the opportunity to participate) in non-random ways. Furthermore, the non-participation of several magnet high schools limits the degree to which one can claim the sample represents magnet high schools in the state. These caveats notwithstanding, we believe we have obtained information from a relevant set of magnet schools and a small, but informative set of non-magnet schools, and that the set of students surveyed in each school provide a reasonably representative picture of the experience and perceptions of students in these schools. Table 3.1 contains the number of students sampled and overall response rates for each school. For the magnet schools, response rates were computed by dividing the number of completed surveys we obtained by the number of students officially enrolled in the grades surveyed in the school as of October 1, 2007. Students who did not respond were either absent on the day the test was administered, were no longer enrolled in the school at the time of administration, or declined to participate in the survey. Response rates were not calculated for the non-magnet schools due to the nature of our sampling strategy; that is, it was difficult to accurately estimate student populations for each classroom surveyed. Table 3.1 - Sample of Schools Participating in the Survey, Spring 2008 No. of Students completing survey Magnet Schools a Response Rate Grades 9 & 12 Magnet LEA % Minority in School Big Picture High School Collaborative Alternative Magnet CT International Baccalaureate Academy EASTCONN'S ACT Arts Magnet High School b Great Path Academy at MCC - Manchester Greater Hartford Academy of the Arts b Gtr Hartford Academy of Math and Science b High School in Community Hill Regional Career High School Metropolitan Business High School Metropolitan Learning Center - Bloomfield Sport and Medical Sciences Academy University High School for Science & Eng. c 21 27 79 48 36 165 67 115 296 63 142 165 58 60% 93% 88% * 59% 67% 64% 62% 72% 68% 73% 82% 46% Bloomfield N. Branford E. Hartford EASTCONN CREC CREC CREC New Haven New Haven New Haven CREC Hartford Hartford 88% 37% 57% 30% 55% 25% 45% 73% 73% 92% 76% 76% 80% Non-Magnet Comparison Schools Suburban Public High School Urban Public High School Urban Public High School c 292 118 94 - Suburb Central City Central City 11% 98% 90% a - New school. b Half-day magnet programs offered at these schools. GHAMS: 7th-10th graders who started in 08-09 are enrolled in full-day program. These will be the first full-day cohorts when they reach 9th grade and progress through. Currently 9th-12th students are half-day. GHAA: 9th graders are full time and will remain so in 10th-12th grades. Currently 10th-12th students are half-day. ACT: Currently 9th and 10th graders are full-time, 11th and 12th graders are half-time. c At this school primarily 9th graders completed the survey; response rates are based on 9th grade enrollments only. *This calculation was not available at the time of printing this report. 3-23 Survey Measures The questionnaire we developed, The High School Student Survey (HSSS), measures a variety of school climate, student attitude, and student background variables. Some of the questionnaire items measure concepts or what is referred to in research as constructs. Constructs are measured by multiple survey items of similar focus. When certain psychometric conditions are met, scores from these items can be averaged to form an overall “scale score” for the construct. For instance, taken together, HSSS items 2, 3, 4, 6, 8, 9, 10, 11, and 13 provide a measure for the construct called “Academic Aspirations.” Given that these items are scored on a common 1-5 Likert scale, they can be averaged to create an Academic Aspirations scale score. The use of multiple items for a single construct makes for a more reliable and valid scale measure. Statistical tests can confirm the internal consistency of a collection of items and support their use as a single scale. Appendix C provides details on the psychometric properties of the constructs described in this section. In the next section we describe each of the school climate and student attitude constructs assessed by the questionnaire. Details on item responses used to construct the scales described here are provided in Appendix D. School Climate Measures The school climate measures capture student perceptions of their school in terms of academics, comfort and safety, and the nature and quality of intergroup relations. Together these variables describe the conditions students experience in their schools. Academic Press. Academic press represents the degree to which students experience “a normative emphasis on academic excellence”.10 In their review of the research, Lee and colleagues report a connection between academic press, student effort and increased achievement.11 The HSSS measured academic press using seven items, each of which presented a statement about their teachers’ propensity to instill academic press; for instance, “My teachers want me to understand my work, not just memorize it” and “My teachers set high academic standards for me.” Respondents were asked to indicate their level of agreement with each statement using a 5-point Likert scale ranging from strongly disagree to strongly agree. Together these seven items constitute the Academic Press scale. School Influence on College Aspirations. Teachers’ expectations of their students are highly influential to the post-secondary aspirations of students.12 Three survey items explored the degree to which teachers and counselors provided encouragement or important guidance to 10 McDill, E.J., Natriello, G., & Pallas, A.M. (1986). A population at risk: Potential consequences of tougher school standards for school dropouts. American Journal of Education, 94, 135-81. Also see Murphy, J., Weil, M., Hallinger, P., & Mitman, A. (1982). Academic press: Translating high expectations into school policies and classroom practices. Educational Leadership, 40, 22-26. 11 Lee, V. E., Bryk, A. S., & Smith, J. B. (1993). The organization of effective secondary schools. In L. DarlingHammond (Ed.), Review of Research in Education, 19, 171-268. Washington, DC: American Educational Research Association. 12 Metheny, J., McWhirter, E. H., & O'Neil, M. E. (2008). Development measuring perceived teacher support and its influence on adolescent career. Journal of Career Assessment, 16, 218. 3-24 students regarding college attendance. Together these three items constitute the School Influence on College Aspirations scale. Peer Academic Norms. Student academic expectations can also be influenced by peer academic orientations. Peer groups that value (or devalue) academics can have positive (or negative) effects on an individual student’s performance in school.13 Two survey items asked students their level of agreement with respect to their peers’ academic orientations (e.g., “Most of my friends care about doing well in school.”). A third question asked how many of their close friends “tried hard to do well” in school. Together these three items represent the Peer Academic Norm scale. Social Sanctions for Achievement. Fordham and Ogbu suggested that in response to systematic discrimination, which constrains opportunities and reduces returns to education, some minority students may develop negative orientations toward academics and, as a result, students from such groups who attempt to do well in school face social sanctions from peers.14 This phenomenon has been referred to as the “acting White hypothesis.” Four survey items were developed to detect the presence of social sanctions for achievement among all students. For instance, respondents were asked how much they agreed with the following statement: “I usually avoid answering questions in class because I don’t want other students to think I am trying too hard.” Together the four items constitute the Social Sanctions for Achievement scale. Teacher-Student Relationship. Successful school reform models have demonstrated the importance of students having positive relationships with adults in school.15 Caring teachers make differences in students’ lives and their connection to school.16 The HSSS asked students their level of agreement with three items related to their relationships with teachers (e.g., “My teachers respect me.”). Together these three items comprise the Teacher-Student Relationship scale. Classroom Disruptions. Classrooms that exhibit disruptive behaviors can impede learning and student motivation to learn. Two survey items asked students about the general climate in their classrooms. Students were asked how often their classes “were interrupted by the misbehavior of other students” and how many of their classes had “significant student behavior problems.” Together these two items represent the Classroom Disruptions scale. Safety and Belonging. Students should feel connected to their schools and feel safe in their surroundings. Three questions were asked of students along these criteria. For instance, 13 Kao, G. (2001). Racial and ethnic differences in peer influences on educational achievement. In Massey & Anderson, (Eds.) The problem of the century: Racial stratification in the U.S. at the millennium. New York: Russell Sage Foundation. 14 Fordham, S., & Ogbu, J. U. (1986). Black students' school success: Coping with the "burden of ‘acting White’." The Urban Review, 18(3), 176-206. 15 Anson, A. R., Cook, T. D., Habib, F., Grady, M. K., Haynes, N., & Comer, J. P. (1991). The Comer school development program: A theoretical analysis. Urban Education, 26, 56-82; Chicago Annenberg Challenge (1995). Request for proposals: An invitation to reinvent public schools for the benefit of Chicago's children. Chicago: Author. 16 Noddings, N. (1988). An ethic of caring and its implications for instructional arrangements. American Journal of Education, 96(2), 215-230. 3-25 respondents indicated the degree to which they felt “safe and secure” in their school. Together these three items constitute the Safety and Belonging scale. Intergroup Relations: Equal Status, Interaction, Interdependence, Supportive Norms. The next four constructs derive from intergroup contact theory. Contact theory suggests that fourenvironmental conditions need to be met in order for meaningful intergroup relationships to occur. Such meaningful contact leads to enhanced cross-racial understanding, reductions of group stereotypes, and authentic intergroup friendships. Contact theory requires that all peer groups have equal status, interactions must occur between the groups in a cooperative and not competitive manner, and the sanctioning authority must provide opportunities and support for these interactions. Twelve items were adapted from an instrument used by Gaertner and associates.17 Gaertner and colleagues’ instrument measured four factors consistent with the conditions of contact laid out by Green and colleagues.18 For example, to measure Equal Status, students were asked their level of agreement that “Teachers at this school are fair to all groups of students.” Factor and reliability analyses of our survey data indicated that these same four factors (equal status, interaction, interdependence, supportive norms) are being measured by the HSSS. In all, four scales were created: the Equal Status scale (3 items), Interaction scale (3 items), Interdependence scale (3 items), and Supportive Norms scale (2 items). One of the original items was dropped from the Supportive Norms scale, as it did not correlate sufficiently well with the other two items. Racial Tension. HSSS also attempted to gauge race relations in schools using two items. Students were asked how much “tension” existed in their school between students of different racial or ethnic groups and how much they thought their school experiences have impacted their ability to understand members of other races and ethnic groups. Measures of Student Attitudes and Behaviors Student attitude and outcome variables include student self-reports of their own attitudes and behaviors, which might be influenced by their particular school experience. These variables represent students’ individual orientations toward academics, race, and culture. Many factors undoubtedly influence the student attitudes and outcomes measured here. To some degree, however, the school environment plays a role in shaping these attitudes and outcomes. Academic Aspirations. Students who demonstrate high academic aspirations and academic engagement typically do better in school.19 They are more engaged and see the 17 P. 235 in Gaertner, S. L., Rust, M. C., Dovidio, J. F., Bachman, B. A., & Anastasio, P. A. (1994). The contact hypothesis: The role of a common ingroup identity on reducing intergroup bias. Small Group Research, 25(2), 224249. 18 Ibid; Green, C. W., Adams, A. M., & Turner, C. W. (1988). Development and validation of the school interracial climate scale. American Journal of Community Psychology, 16, 241–259. 19 Newmann, F. M. (1992). Student engagement and achievement in American secondary schools. New York: Teachers College Press. 3-26 relevance of the work they do in the classroom.20 Schools can play a major role in fostering these aspirations. The HSSS asked respondents to indicate their level of agreement with a series of statements such as, “getting good grades (As and Bs) is important to me,” “I care about the quality of my class work,” and “I work very hard on my school work.” Together, a total of nine questions constitute the Academic Aspirations scale. College Expectations. Post-secondary attainment is a key determinant of one’s vocational pursuits and overall life outcomes. The survey asks students three separate questions related to their college aspirations and preparations. Have they taken or been planning to take the SAT or ACT? What is the educational degree they expect to get? Do they plan to continue their education right after high school? The survey also asked students to write in the job or occupation they expect to have at age 30. In all, three items contribute to the College Expectations scale. Attendance. Students who are frequently absent in school are obviously less likely to learn and more likely to dropout altogether. Attendance can be an indicator of students’ engagement in school, their sense of connection to the school, and values orientation toward school. The HSSS asked students the frequencies with which they were late and missed class over a two-week time period. Social Closeness. Two sets of items were posed to students with respect to how close and how comfortable they felt toward members of certain racial or ethnic groups (i.e., Black, Latino/a, White, Asian, and Multi-racial categories). This item is adapted from what are referred to as “feeling thermometers.” Feeling thermometers are often used in political science research to gauge feelings and attitudes respondents report having toward others.21 With our items, responses fell along a 7-point Likert scale ranging from “not at all close (or comfortable)” to “extremely close (or comfortable).” Factor and reliability analyses of these items suggested they should be separated into two distinct scales: one that measures social distance towards students of color (in this case, Black, Latino/a, and Multi-race categories) and another scale that measures social distance towards White and Asian students. In the results sections that follow, we present findings on the 6-item Social Distance toward Minorities scale and the 4-item Social Distance toward Whites/Asians scale. Close Friends by Race/Ethnicity. Magnets are intended to bring together students from diverse backgrounds. Integration alone, however, does not necessarily lead to increased cultural competency or closer intergroup friendships. The survey asked students the following question to gather information on their closest friends by race and ethnicity: “Think of your 10 closest friends. Indicate how many of those 10 closest friends are from each of the following racial or ethnic groups by checking either: None, One, or More than One for each.” This item does not lend itself to a singular measurement scale. Each close friend by race or ethnic group is reported on separately. 20 Newmann, F. M. & Associates (1996). Authentic achievement: Restructuring schools for intellectual quality. San Francisco: Jossey-Bass. 21 Alwin, D. F. (1997). Feeling thermometers versus 7-point scales. Sociological Methods and Research, 25(3), 318340; Miller, W. E. (1982). American National Election Study, 1980. Ann Arbor, MI: Inter-University Consortium for Political and Social Research. 3-27 Future Multicultural Interest. Since magnet schools focus on providing an entryway to an ethnically and racially dynamic world, we measured students’ inclination toward having a significant continued presence of multiculturalism in their lives through adulthood. Students were asked to indicate their level of interest with respect to six future multicultural pursuits (e.g., “speaking a foreign language,” “living in a racially/ethnically diverse neighborhood”). Together these six items constitute the Future Multicultural Interest scale. Analytic Strategy Figure 1 summarizes the variables and constructs measured by the HSSS. It groups the measures assessed by the HSSS into three main categories: background variables, school climate variables, and student attitudes and outcomes. The framework posits that school organization together with the set of students who enroll in a school interact to create a school climate, which can be assessed along the dimensions defined above. Together with family influences, this school climate shapes the set of experiences a student has and the set of attitudes that student develops. The theory behind magnet schools is that a thematic focus which brings together students with shared interests interacts with a racially, ethnically and economically diverse student body to create a positive school climate. That climate then helps to foster positive attitudes toward academic achievement and people from different backgrounds. Figure 3.1. Analytic model: Variables measured by the High School Student Survey 3-28 In order to examine the key assumptions in this theory of action, we use the results of the survey to conduct a number of analyses. The first set of analyses focus on school climate and compares average measures of school climate in magnet schools to the school climates in the city and suburban non-magnet schools in our sample. These comparisons tell us whether or not there are significant differences between magnet schools and the non-magnet public schools their students might otherwise attend. Given the relationship between interdistrict magnet schools and the Sheff decision, the effects of interdistrict magnet schools on students who reside in central cities are of particular interest. Thus, we also present analyses that focus on students who reside in a central city. These analyses compare students’ perceptions of school climate between city students in magnet schools and city students in non-magnets. These comparisons indicate whether magnet schools are providing city students access to substantially different school environments than they might otherwise encounter. After examining these indicators of school climate, we turn to student attitudes and behaviors. We compare averages of our measures of student attitudes and behaviors across magnet school students, students in city, non-magnet schools, and students in suburban, non-magnet schools. We present a second set of comparisons that limit the sample to students who reside in a city and compare the attitudes and behaviors of magnet school city students to non-magnet city students. Interpreting any differences in attitudes and behaviors between magnet and non-magnet school students is difficult. Students and their parents self-select into either magnet or non-magnet schools, which suggests there might have been important attitudinal differences between the two groups of students before any exposure to the magnet school treatment. Therefore, any observed differences between magnet and non-magnet school students cannot necessarily be attributed to the magnet school experience. Because of the nature of the data available for these analyses, we are not able to address concerns around self-selection bias as we have in our analysis of magnet school effects on achievement. We can, however, take some steps to explore the possibility that any observed differences between magnet and non-magnet school students are due to differences in school experiences. One step we take is to compare the differences between the attitudes of magnet school students from Hartford and New Haven and non-magnet school students from those cities controlling for observed family background characteristics. These controls for family background are achieved through multiple regression analysis, where our measures of individual students’ attitudes and behavior are regressed on an indicator of whether or not a student attends a magnet school, an interaction of that variable on an indicator of what grade the student is in and an extensive set of student background characteristics that we collected as part of the survey. The results of these regressions provide estimates of the differences in attitudes and behaviors between magnet and non-magnet students based on comparisons of similar students. Although suggestive, this type of analysis cannot rule out the possibility that any differences in attitudes and behaviors between magnet and non-magnet students are due to unobserved differences in family background between the two sets of students. 3-29 Results We present the analyses of the survey in two parts. The first part examines differences in perceptions of school climate between magnet and non-magnet school students, and the second part examines differences in attitudes and behaviors between magnet and non-magnet school students. In all cases, we make separate comparisons for 9th graders and for 12th graders.22 School Climate Table 3.2 compares perceptions on nine separate dimensions of school climate across three groups of students—magnet school students, students in city non-magnet schools, and students in the suburban non-magnet schools in our sample. An important caveat to keep in mind when examining this table is that the figures reported for magnet schools are average responses of students across a large number of magnet schools. Thus, these figures hide considerable variation across individual magnet schools, and statements made here about differences between magnet schools generally and particular types of non-magnet schools might not apply to specific magnet schools. It is also important to note that these data are student self-report data. We did not collect additional data that would allow us to triangulate students’ perceptions. Nonetheless, student perceptions are critical and Table 3.2 reveals potentially important differences between the experiences of the typical magnet school student and those of nonmagnet school students. Comparisons across the different types of schools indicate that magnet schools provide an academic climate similar to that found in a wealthy, suburban non-magnet high school, and that the peer support for academic achievement is stronger in magnets than in non-magnet city schools. Also, students in magnet schools perceive more positive intergroup and racial relations than in either the city or suburban non-magnet schools in our sample. These findings suggest that magnet schools are indeed succeeding in their efforts to create unique and positive learning environments for their students. Comparisons across the types of schools in Table 3.2 raise some potential concerns for magnet school administrators as well. Most notably, there is some indication that teacher-student relationships and students’ sense of safety and belonging might be weaker in magnet schools than in some other schools. Several considerations mitigate this concern. First, although perceptions of student teacher relations are less favorable in magnet schools than the non-magnet schools, they are still quite strong, with the majority of students agreeing with the statements that my teachers respect me, my teachers understand me, and my teachers really listen to what I have to say. Second, among magnet school students, perceptions of teacher student relations are stronger for 12th graders than 9th graders. Finally, the figures in Table 3.2 are averages, and thus, student-teacher relations in some magnet schools might compare more favorably with nonmagnets. Nonetheless, the results in Table 3.2 suggest that some magnet schools might need to devote more attention to ensuring that all of their students feel safe, respected and understood. Below we discuss the results in Table 3.2 in greater detail. 22 In the analysis here, 10th graders are grouped together with 9th graders and 11th graders with 12th graders. The number of 10th and 11th graders surveyed is small relative to the number of 9th and 12th graders, and this grouping does not significantly influence the results presented here. 3-30 Academic Expectations. The first two aspects of school climate in Table 3.2 reflect the academic expectations that staff in the school set for students. Academic press represents the degree to which students perceive an emphasis on academic excellence, and student perceptions of academic press are quite similar across all three types of schools. Ninth grade students in the suburban non-magnet school tend to perceive slightly more academic press than 9th grade students in either the magnet or the city non-magnet schools, but this difference is substantively small and there are no significant differences in perceived academic press across 12th graders in the three types of schools. Table 3.2 - Students' Perception of School Climate Student Level Means and (Standard Deviations) Magnet Schools City Non-Magnets Suburban Non-Magnet 9th Graders 12th Graders 9th Graders 12th Graders 9th Graders 12th Graders Academic Press 3.839 (0.704) 3.872 (0.730) 3.804 (0.732) 3.813 (0.604) 3.965** (0.604) 3.793 (0.506) School Influence on College 3.466 (0.730) 3.918 (0.658) 3.678** (0.727) 3.651** (0.735) 3.296** (0.634) 4.043 (0.578) Peer Academic Norms 3.253 (0.659) 3.387 (0.685) 3.188 (0.639) 2.901** (0.684) 3.417** (0.689) 3.427 (0.583) Social Sanctions for Achievement 2.019 (0.700) 1.949 (0.717) 2.430** (0.964) 2.381* (0.958) 2.072 (0.676) 2.000 (0.658) Classroom Disruptions 2.782 (1.120) 2.273 (1.054) 2.681 (1.045) 2.881** (1.120) 2.548 (0.942) 2.171 (0.775) Teacher-Student Relationship 3.357 (0.946) 3.599 (0.897) 3.648** (0.860) 3.770 (0.876) 3.512** (0.833) 3.559 (0.654) Safety and Belonging 3.766 (0.867) 3.788 (0.903) 3.406** (0.978) 3.573 (0.811) 4.161** (0.747) 4.456** (0.636) Intergroup Relations 14.648 (2.576) 14.800 (2.447) 13.942 (2.327) 13.718** (2.203) 14.571 (2.226) 14.050* (2.057) Racial Tension 2.328 (1.172) 2.166 (1.162) 2.780** (0.998) 3.214** (0.976) 2.642** (1.348) 2.584** (1.079) 744 479 168 42 165 127 N * Significantly different than value of same grade among magnet school students at 0.10 level ** Significantly different than value of same grade among magnet school students at 0.05 level There are, however, significant differences across the three types of schools in the degree to which teachers and counselors provided encouragement and guidance to students regarding 3-31 college attendance. Ninth graders in the city non-magnets perceive more emphasis from teachers and counselors on college than do 9th graders in magnet schools, and 9th graders in magnets perceive more emphasis on college than 9th graders in the suburban non-magnet. This pattern, however, is reversed for 12th graders. Among 12th graders, students in city non-magnet schools perceive significantly less encouragement and guidance about college than students in the magnet schools and in the suburban non-magnet. Generally, the encouragement and guidance about college in the magnet schools appears more similar to that in the suburban non-magnet school than that in the city non-magnets. In particular, the magnet schools are like the suburban non-magnet in placing less emphasis on college for 9th graders, and providing greater encouragement and guidance about college for 12th graders, than the city non-magnet schools. Peer Academic Climate. The next three aspects of school environment in Table 3.2 reflect the academic norms and climate fostered by peer attitudes and behaviors. On average, magnet schools appear to provide a peer environment more conducive to achievement than the city non-magnets, and similar to the peer environment in the suburban non-magnet school, especially among 12th graders. Twelfth grade magnet school students report that their peers place a higher value on academic achievement, that they face less social penalty for achievement, and that they experience fewer classroom disruptions than students in the city non-magnets. There are no significant differences in perceptions of peer academic environment between 12th graders in magnet schools and 12th graders in the suburban non-magnet school. Among 9th graders, the differences in peer academic environment between magnet schools and the non-magnet city schools are less marked. The perceptions of peer academic norms and classroom disruptions among 9th grade magnet school students are not significantly different than those of the 9th grade students in the city non-magnets. Ninth grade magnet school students do, however, report less social penalty for achievement and effort than do 9th graders in the nonmagnet city schools. Also, perceived academic norms are slightly higher among 9th graders in the suburban non-magnet school than among 9th graders in the magnets. The fact that differences between magnet and non-magnet city schools in peer academic environment are greater among 12th graders than among 9th graders has two potential explanations. One potential explanation is that 12th graders have had a longer time than the 9th graders to interact with each other and with the school staff and curriculum, and as a result 12th graders have more fully developed and internalized the norms and climate that typify the school. This explanation suggests that the magnet schools themselves have a strong, positive effect on academic norms and behaviors among students. A second explanation is that the students who have entered magnet high schools more recently, the 9th graders, are different in important ways than those who entered several years ago, the 12th graders. As a result, the younger cohort of students in magnets have created and face a climate different than the 12th graders - a climate that is more like that in the city non-magnets schools. This second explanation suggests that magnet schools might face greater challenges in trying to create strong academic environments moving forward. Our data do not allow us to determine which explanation is most accurate.23 23 Two additional explanations are possible. First, the 12th grade sample only includes students from one of two city non-magnets for which we have 9th grade responses. However, the non-magnet city school for which we do not have 12th grade responses had lower peer academic norms, stronger social sanctions for achievement and more classroom disruptions among 9th graders than the non-magnet city school for which we do have 12th grade 3-32 Comfort and Community.The next two aspects of school climate in Table 3.2 reflect students’ level of comfort with and sense of community in their school. On these dimensions magnets schools, on average, do not compare favorably, especially to the suburban non-magnet. Among 9th graders, students in all non-magnets report stronger student teacher relations than do students in the magnet schools. However, within the magnet schools, 12th grade students report stronger relationships with teachers than do 9th grade students, and there are no statistically significant differences in reported teacher student relationships across schools for the 12th graders. For both the 9th and 12th grade level, magnet school students’ sense of safety and belonging falls between that reported by city non-magnet school students and that reported by students in the suburban non-magnet. Students in the magnet schools feel a stronger sense of safety and belonging than do students in the city non-magnets and a weaker sense of safety and belonging than students in the suburban non-magnet. Among 12th graders, magnet school students’ perceptions of safety and belonging, on average, are more similar to that of students in the city non-magnet schools than students in the suburban non-magnet. Intergroup and Race Relations. The final two aspects of school climate in Table 3.2 reflect the quality of intergroup and racial relations within the school. Among 9th graders, there are no significant differences across the three types of schools in the quality of intergroup relations. Among 12th graders, however, magnet school students report significantly more positive intergroup relations than students in both types of non-magnet schools. Magnet school students, both 9th and 12th graders, also report significantly less racial tension in their school than their counterparts in both the city and suburban non-magnet schools. These important results suggest that magnet schools, in general, are meeting one of their primary goals by providing students access to environments where students from different backgrounds can interact in a supportive, positive fashion. Focus on City Students. The effects of interdistrict magnet schools on students who reside in central cities are of particular interest. The comparisons in Table 3.2 include all magnet school students including those who reside in suburban districts. The perception of magnet school climates might be different for students who reside in the city for two reasons. First, city students are more heavily represented in some magnet schools than others. If the magnet schools that city students are more likely to be enrolled in differ from magnet schools they are less likely to attend, then the school climate experienced by the typical magnet school student from the city will differ from that reported in Table 3.2. Second, because of differences in perspectives that they bring from their homes and communities, city students and suburban students might perceive the climates in magnet schools differently. responses. Thus, it is unlikely that the differences between magnets and city non-magnets that emerge in 12th grade are due to the change in sample. Second, attrition rates might differ across the magnet schools and the non-magnet school. If magnet schools are more likely to retain students with a positive perception of the academic environment than the non-magnet school, that could lead to differences that do not exist among 9th graders to emerge in comparisons of 12th graders. 3-33 Table 3.3 presents results from analyses that limit the sample to students who reside in a central city, and compares the perceptions of school climate among city students in magnet schools to perceptions of school climate among city students in non-magnets. These comparisons indicate whether magnet schools are providing city students access to substantially different school environments than they might otherwise encounter. The patterns of differences between magnet schools and non-magnet schools in Table 3.3 are very similar to those in Table 3.2. Ninth grade city students in magnets perceive less emphasis on college from teachers and counselors than 9th grade city students in non-magnets, but among 12th graders, city students in magnets report more encouragement and guidance about college than city students in non-magnets. City students’ perceptions indicate that peer environments are more conducive to academic achievement in magnet schools than in the non-magnet schools. Teacher-student relations, as perceived by city students, are weaker in magnet schools than nonmagnet schools, but city students in magnet schools feel a stronger sense of safety and belonging than do students in the city non-magnet schools. Finally, city students who attend magnet schools perceive more positive intergroup relations and less racial tension among their classmates than do students in the city non-magnet schools. 3-34 Table 3.3 - Perception of School Climate Among Students Who Reside in Central City Student Level Means and (Standard Deviations) Magnet Students Non-Magnet Students Effect Size 9th Graders 12th Graders 9th Graders 12th Graders 9th Graders 12th Graders Academic Press 3.844 (0.649) 3.798 (0.648) 3.803 (0.734) 3.804 (0.617) School Influence on College 3.456 (0.739) 3.947 (0.587) 3.674** (0.728) 3.633** (0.734) Peer Academic Norms 3.259 (0.663) 3.388 (0.672) 3.186 (0.639) 2.888** (0.697) Social Sanctions for Achievement 2.028 (0.793) 2.019 (0.772) 2.434** (0.966) 2.325 (0.947) Classroom Disruptions 2.838 (1.109) 2.362 (0.948) 2.685 (1.047) 2.886** (1.077) Teacher-Student Relationship 3.397 (0.884) 3.549 (0.824) 3.644** (0.861) 3.733** (0.881) -0.280 Safety and Belonging 3.685 (0.872) 3.632 (0.856) 3.405** (0.981) 3.628 (0.784) 0.305 Intergroup Relations 14.436 (2.544) 14.394 (2.215) 13.936 (2.333) 13.717* (2.230) Racial Tension 2.442 (1.262) 2.194 (1.122) 2.772** (0.996) 3.200** (0.992) -0.294 0.625 0.713 -0.490 -0.529 -0.220 0.304 -0.278 -0.864 N 324 188 167 40 * Significantly different than value of same grade among magnet school students at 0.10 level ** Significantly different than value of same grade among magnet school students at 0.05 level The last two columns of Table 3.3 report estimated effect sizes for the differences in city students’ perceptions of their magnet and their non-magnet schools. The effect size is the difference between the average value of the indicator for magnet school students and the average value for non-magnet school students, divided by the standard deviation of the indicator.24 Effect sizes give a sense of the magnitude of the differences between magnet and non-magnet schools. Standard practice in the social sciences is to characterize effect sizes around 0.20 as small, 0.50 as moderate, and 0.80 as large. The effect sizes in Table 4.3 indicate that differences in peer academic environment are moderate to large among 12th graders and small to moderate for 9th graders; differences in teacher-student relations and safety and belonging are small, particularly among 12th graders; differences in the quality of intergroup relations are small to 24 Standard deviations are calculated for the pooled sample of city students including both those in magnets and those in non-magnets. 3-35 moderate; and differences in perceived racial tension are small for 9th graders but large for 12th graders. Student Attitudes and Behaviors Table 3.4 compares the attitudes and behaviors of magnet students and non-magnet students. The attitudes and behaviors examined can be divided into two sets. The first set are academic attitudes and behaviors including academic aspirations, college expectations, average number of absences and average number of skipped classes. The second set is intergroup attitudes. Compared with city non-magnet students, magnet students have similar academic aspirations, higher expectations for post-secondary attainment, are less likely to be absent, and are significantly less likely to skip classes. These differences between magnet and non-magnet city school students show up for both 9th and 12th graders, but a more marked among 12th graders. These results are generally consistent with what we might expect given the differences in academic climate between magnet and city non-magnets documented above. The results in Table 3.4 also indicate that magnet school students have slightly higher academic aspirations, slightly lower expectations for post-secondary attainment, and are less likely to skip class than suburban non-magnet students. Comparisons also indicate that intergroup attitudes are more positive among magnet school students than non-magnet school students along several dimensions. Specific results in Table 3.4 are discussed in more detail below. Academic Attitudes and Behaviors. The levels of academic aspirations among magnet school and city non-magnet school students are very similar, and both are slightly higher than among suburban non-magnet school students. The differences between magnet school students and suburban, non-magnet students expressed as effect sizes are 0.15 for 9th graders and 0.31 for 12th graders, which are considered small. Thus, the most marked finding in the first row of Table 3.4 is that, as in the case of Academic Press (see Table 3.2), there is little difference across schools in individual student academic aspirations. Differences in expectations concerning college completion across students in the three types of schools, however, are more marked. Students in magnet schools tend to have higher expectations for post-secondary attainment than students in the city non-magnet schools, but lower expectations than students in suburban, non-magnet schools. These differences are more marked and have higher levels of statistical significance at the 12th grade level. Students in magnet schools skip class less frequently than both city and suburban non-magnet school students and the differences between magnet and non-magnets are particular marked among 12th graders. Among 9th graders, magnet school students have fewer absences than city, non-magnet students, but more absences than suburban, non-magnets. Among 12th graders, however, magnet school students have significantly fewer absences than both city and suburban non-magnet students 3-36 Table 3.4 - Student Attitudes and Behaviors Student Level Means and (Standard Deviations) Magnet Schools City Non-Magnets Suburban Non-Magnet 9th Graders 12th Graders Academic Aspirations 4.116 (0.532) 4.166 (0.599) 4.114 (0.693) 4.104 (0.525) 4.032** (0.613) 3.990** (0.496) College Expectations -0.048 (0.775) 0.158 (0.628) -0.461* (0.958) -0.470** (1.002) 0.021 (0.564) 0.326** (0.378) 1.321 1.459 1.764** 2.951** 0.564** 2.169** (2.404) (2.222) (2.570) (3.402) (1.290) (2.405) 0.636 (1.757) 0.576 (1.411) 1.158** (2.109) 2.244** (2.844) 0.921** (2.417) 1.055** (1.862) 5.470 (1.316) 5.518 (1.234) 4.823** (1.368) 4.345** (1.151) 4.830 (1.631) 5.063 (1.501) 5.735** (1.337) 5.606** (1.016) % of White students with multiple minority friends 93.1 88.9 54.5** 43.4** % of minority students with multiple White friends 71.7 72.8 46.8** 47.5** 97.7** 96.4** 3.468 (0.929) 3.624 (0.904) 3.390 (0.895) 3.413** (1.047) 2.912** (0.992) 3.231** (0.939) % reporting that school experience has helped understanding of other groups 59.3 67.8 64.3 66.7 38.8** 35.4** N 744 479 168 42 165 127 Average # of Absences Average # of Skipped Classes Social Closeness to Minorities (among White students) Social Closeness to Whites (among minority students) Future Multicultural Interests 9th Graders 12th Graders 9th Graders 12th Graders 4.457** (1.681) 4.549** (1.475) * Significantly different than value of same grade among magnet school students at 0.10 level ** Significantly different than value of same grade among magnet school students at 0.05 level Intergroup Attitudes. Minority students in magnet schools feel significantly closer to Whites and are more likely to have multiple White friends than minorities in the city, nonmagnet schools. These results are not surprising given the very small number of White students in the city, non-magnet schools, and the generally positive intergroup relationships in magnet schools that was documented above. Twelfth graders in magnet schools also express significantly stronger interest in pursuing future activities in multicultural settings than 12th graders in the city non-magnet school. Magnet schools students, however, are not more likely 3-37 than non-magnet, city school students to report that their school experiences have helped them to develop an understanding of other groups. Compared to White suburban, non-magnet students, White magnet school students feel significantly closer to minority students and are much more likely to report having multiple minority group friends than are non-magnet, suburban school students. Minority students in the suburban non-magnets, however, tend to feel closer to Whites and are more likely to have multiple White friends than minority students in magnet schools, a result which is not surprising given that the student population in this suburban, non-magnet high school is predominantly White. Both among 9th graders and 12th graders, magnet school students expressed stronger future multicultural interests and are significantly more likely to report that their school experience has helped them understand people from other groups than students in the suburban, non-magnet school. Focus on City Students. The impacts of magnet schools on city students are of particular interest given the mandate of the Sheff decision to improve educational conditions for central city minority students. The analysis presented in Table 3.5 limits the sample of students to those who reside in the city and compares the attitudes and behaviors of magnet and non-magnet school students. The most marked finding in Table 3.5 is that there is very little difference in attitudes between 9th grade magnet students and 9th grade students in non-magnet schools, but marked differences among 12th graders. There are no significant differences in either academic or intergroup attitudes between 9th grade magnet school students who live in a central city and 9th grade city students attending non-magnets. Ninth grade city students in magnets skip class significantly less often those in non-magnets, but even this difference in behavior is small (effect size=-0.20). Among 12th graders, in contrast, students from the central cities that attend a magnet school feel significantly closer to White students, are more likely to have multiple White friends, have higher expectations for post-secondary attainment, have fewer absences, and skip class much less frequently than do other students who live in the central city. There are two ways to interpret these findings. One interpretation is that there are few differences between city students who attend magnet high schools and those who attend nonmagnet high schools when the two groups begin high school, but significant differences between the two groups emerge by the end of their high school career. This interpretation suggests that magnet schools have significant positive effects on students’ attitudes towards academic achievement and towards people from backgrounds different from their own. A second interpretation is that the sample of 9th grade magnet students who participated in this survey is significantly different than the sample of 12th grade magnet students who participated. This interpretation suggests that magnet school administrators and teachers may face stiffer challenges in the near future than they have in the recent past in trying to foster positive academic and intergroup attitudes. 3-38 Table 3.5 - Attitudes and Behaviors Among Students Who Reside in Central City Student Level Means and (Standard Deviations) Magnet Students Non-Magnet Students Effect Size 9th Graders 12th Graders 9th Graders 12th Graders 9th Graders 12th Graders Social Closeness to Minorities 5.860 (1.182) 5.842 (1.108) 5.672 (1.166) 5.526* (1.518) 0.265 Social Closeness to Whites 4.698 (1.672) 5.075 (1.581) 4.66 (1.710) 4.572** (1.452) 0.321 62.7 68.9 46.5 44.7** 0.452 3.517 (0.886) 3.701 (0.799) 3.386 (0.896) 3.430 (1.064) 56.5 70.2 64.1 65.0 Academic Aspirations 4.121 (0.502) 4.187 (0.540) 4.116 (0.694) 4.076 (0.521) College Expectations -0.057 (0.748) 0.135 (0.614) -0.459 (0.961) -0.512** (1.008) 0.876 Average # of Absences 1.590 (2.562) 1.824 (2.466) 1.766 3.064** -0.460 (2.578) (3.447) 0.786 (1.996) 0.739 (1.461) 1.156** (2.116) 2.179** (2.790) 324 188 167 40 % of minority students with multiple White friends Future Multicultural Interests % reporting that school experience has helped understanding of other groups Average # of Skipped Classes N -0.202 -0.784 * Significantly different than value of same grade among magnet school students at 0.10 level ** Significantly different than value of same grade among magnet school students at 0.05 level To investigate which of these two interpretations is more plausible, we compared the sample of 9th grade magnet school students with the sample of 12th grade magnet students on the family background variables that we collected in our survey. The results indicate that the two groups have a similar racial composition, similar percentage of parents with college degrees, and similar access to computers and space to do school work in the home. The two groups also report spending similar amounts of time talking with parents about their courses, school activities, and grades. The 12th grade students are, however, significantly less likely to be Hispanic and less likely to come from a home where the primary language spoken is Spanish. And of course, we cannot rule out the possibility of unobserved differences in family background between 9th and 12th grade magnet school students. Thus, there may be important differences between the 3-39 students who have entered magnet schools more recently and those who entered as part of earlier cohorts.25 To examine whether the difference in attitudes between magnet school and non-magnet school students from the city might be due to differences in their high school experiences, we estimated regression models that controlled for observed differences in family background characteristics between magnet school students and non-magnet school students. Specifically, we regressed the measures of individual student attitudes and behavior on an indicator of whether or not a student attends a magnet school, an interaction of that variable and an indicator of what grade the student is in and an extensive set of student background characteristics that we collected as part of the survey. These background characteristics include the student’s race, primary home language, access to resources in the home including a space to do school work, a regular newspaper subscription, and a computer, and frequency with which the student speaks to his or her parents about school activities and about college. The results of these regressions provide estimates of the differences in attitudes and behaviors between magnet and non-magnet students based on comparisons of students similar along these dimension of home life. The results of these regressions estimated using students who reside in a central city are presented in Table 3.6. Regarding academic attitudes, 12th grade magnet students have significantly higher expectations for college, fewer absences and fewer skipped classes than 12th grade non-magnet students with similar family background characteristics. The estimated effect sizes are moderate to large. Ninth grade magnet school students also show higher college expectations and lower propensity to skip class than 9th grade non-magnet students, but the estimated differences between 9th graders are smaller than those among 12th graders. With regard to racial and intergroup attitudes, 12th grade magnet school students report greater closeness to minorities and to Whites and are more likely to report having multiple White friends than non-magnet school students with similar family background characteristics. The estimated effect sizes here are small to moderate. No such differences are found across 9th graders. Ninth grade magnet school students are, however, less likely than non-magnet school students to report that their school experience has helped develop their understanding of other groups, a difference that is only marginally statistically significant. Generally the results of these regressions are consistent with the hypotheses that magnet schools have positive effects on academic attitudes and on intergroup attitudes that are larger for students who have been in the magnet school longer. These results, however, are only suggestive. Because our controls for family and other student background characteristics are incomplete, the analyses here cannot rule out the possibility that any differences in attitudes and behaviors between magnet and non-magnet students are due to something other than differences in school experiences. 25 Other significant differences between the 9th and 12th grade students were observed, however, these other differences are likely attributed to the age difference between the two groups of students. For instance, compared to the 9th graders, the 12th graders talked more frequently with their parents about college, and their own parents were slightly less like to have a high school diploma and no college and slightly more likely to have some college. 3-40 Table 3.6 - Differences in Attitudes and Behaviors Between Magnet and Non-Magnet Students Who Reside in the Central City, Controlling for Family Background Characteristics 9th Graders Estimated Difference Estimated Effect Size 12th Graders Estimated Difference Estimated Effect Size R2 N Social Closeness to Minorities 0.039 (p=0.788) 0.313* (p=0.084) 0.260 0.112 616 Social Closeness to Whites 0.027 (p=0.892) 0.329* (p=0.066) 0.199 0.135 610 15.5 (p=0.141) 21.3** (p=0.030) 0.434 NA 585 0.126 (p=0.482) 0.232 (p=0.200) 0.129 640 Minority students with multiple White friends Future Multicultural Interests Reports that school experience has helped understanding of other groups -0.115* (p=0.068) -0.237 0.021 (p=0.715) NA 650 Academic Aspirations -0.108** (p=0.019) -0.191 -0.032 (p=0.768) 0.221 642 College Expectations 0.279* (p=0.086) 0.341 0.433** (p=0.000) 0.529 0.251 647 # of Absences in last two weeks -0.114 (p=0.599) -1.228** (p=0.002) -0.470 NA 631 # of Skipped Classes in last two weeks -0.264* (p=0.099) -1.568** (p=0.001) 0.600 NA 632 0.101 Each row presents estimates derived from a separate student level linear regression model that in addition to a magnet school indicator and interaction between the magnet school indicator and grade includes controls for the student's grade, race, home language, parents level of education, educational resources in the home, and indicators of how often the student speaks with parents about school, grades, and college. P-value for estimated differences in parentheses. R-squared and sample size for each regression reported in the last two columns. For categorical dependent variable, R-squared is not an appropriate measure of a models goodness of fit and so is not reported. * Significantly different than zero at 0.10 level. ** Significantly different than zero at 0.05 level. All significance tests are based on Huber-White standard errors robust to clustering by school. Conclusions Magnet schools bring together students from diverse backgrounds to engage in a common curricular focus. Magnet environments are intended to advance multicultural understandings, broaden worldviews, and enhance conditions for learning and career attainment. A comprehensive student survey was developed to assess the degree to which Connecticut magnet high schools realize these goals. 3-41 Our analytic framework posits that school organization and the students who enroll in a school interact to create a school climate. School climate, along with home influences, shape the set of experiences a student has and the set of attitudes that student develops. We tested the theory of action underlying magnet schools: a thematic focus that brings together students and teachers with shared interests interacts with a racially, ethnically and economically diverse student body to create a positive school climate, and one which helps to foster positive attitudes towards academic achievement and toward people of different backgrounds. The survey measured school climate, student attitudes, and student behaviors among 9th and 12th grade students in 13 magnet high schools and three non-magnet high schools. In all, 1,791 students participated in either a paper or online administration of the survey. Given the emphasis of interdistrict magnet schools on improving the educational opportunities for economically and racially isolated students, our analyses paid particular attention to examining the experiences and perceptions of students who resided in the central city. Initial analyses involved comparisons of school climate among magnet high schools, city nonmagnet high schools, and suburban non-magnet high schools in our sample. Magnet schools appear to provide an academic climate similar to that found in a wealthy, suburban high school. Further, peer support for academic achievement is stronger in magnet than in non-magnet city schools. Magnet school students report more positive intergroup relations than in either the city or suburban non-magnet sample. Perceptions of teacher-student relationships and sense of safety and belonging were slightly lower among magnet students compared to suburban non-magnet schools, however. Overall, these results suggest that magnet schools are meeting their goal of creating more positive learning environments for their students. Comparisons between central city students attending either a magnet or non-magnet reveal that magnet schools, overall, provide city students access to more positive environments than they might otherwise encounter. Ninth grade city students in magnets report less emphasis on college from school staff compared to 9th grade city students in non-magnets. However, among 12th graders, city students in magnets perceive more encouragement and support about college than city students in non-magnets. Peer environments appear more conducive to academic achievement among city students in magnets compared to their counterparts in non-magnets. City students’ perceptions of teacher-student relationships are weaker in magnet schools but city students in magnets report higher sense of safety and belonging than do students in non-magnet city schools. Lastly, city students in magnets report more positive intergroup relations and less racial tension than do students in the non-magnet city schools. Our second set of analyses examined student attitudes and behaviors across our comparison groups. Results indicate that magnet students have similar academic aspirations, stronger college expectations, and are less likely to be absent or skip classes than city non-magnet students. Compared with suburban non-magnet students, magnet students also demonstrate slightly higher academic aspirations, slightly lower college expectations, and are less likely to skip class. In addition, intergroup attitudes are more positive among magnet school students than either city or suburban non-magnet students along several dimensions. For instance, minority students in magnet schools report feeling significantly more close to Whites and more likely to have 3-42 multiple White friends than minorities in the non-magnet city schools. White magnet students feel significantly closer to minority students and report higher percentages of multiple minority friends than do White students from suburban non-magnet schools. At both lower and upper grade levels, magnet school students expressed stronger future multicultural interests and are significantly more likely to report that their school experience helped them understand people from other groups than students in the suburban non-magnet school. Focusing on the city students, there were very few differences in attitudes between 9th grade magnet students and 9th grade students in non-magnet schools. However, in the 12th grade there were significant differences in favor of magnet school students. For example, 12th graders in magnet schools report feeling significantly closer to White students, more likely to have multiple White friends, have higher expectations for college, have fewer absences and skip classes much less frequently than do students attending non-magnet city schools. Of course, these comparisons are susceptible to the influence of self-selection bias discussed above. Regression analyses that attempted to control for several home and family background characteristics among the city students in our sample reveal similar patterns. While we cannot be certain that magnets are causing these presumed effects, the differences are occurring across a number of dimensions and are of large enough magnitudes to suggest that magnet schools are having positive influences on academic and racial attitudes. 3-43 SECTION IV: THE EFFECTS OF INTERDISTRICT MAGNET SCHOOLS ON STUDENT ACHIEVEMENT In addition to the goal of decreasing the racial isolation experienced by many of Connecticut’s students, another goal of interdistrict magnet schools is the improvement of academic achievement. In this analysis we use standardized test scores to determine the degree to which magnets are achieving this goal. We ask one overarching research question • Do interdistrict magnet schools improve student achievement? In answering this question we focus on the average effect of interdistrict magnet schools. The Sheff settlement was, however, aimed at improving the education of central city students in particular. Thus, in order to investigate the impact of interdistrict magnet schools on those students, we ask two more questions: • Do interdistrict magnet schools improve student achievement for central city students? • Do the effects of magnet schools on the academic achievement of central city depend on the extent to which they provide access to less racially isolated learning environments? Results indicate that, on average, interdistrict magnet high schools have positive effects on both math and reading achievement, and interdistrict magnet middle schools have positive effects on reading achievement. Extensions of our analysis indicate that interdistrict magnet high schools have positive effects particularly on the achievement of central city students, and do so regardless of how much attending an interdistrict magnet high school reduces racial isolation. The positive effects of magnet middle schools appear to be limited to suburban students, except in those schools that are able to achieve substantial reductions in racial isolation for their central city students. Data and Measures In order to determine the effects of a school on its students, researchers have to compare the students to another group of students that are similar. While this sounds simple, it can be very difficult. For example, in the case of interdistrict magnet schools, we know that students and parents have made special efforts to seek out alternatives to their geographically assigned school. Thus, we suspect that magnet students might differ from other students with similar ethnic and socioeconomic backgrounds in terms of “unobservables” such as motivation and parental support. This means that unobserved differences between interdistrict magnet school students and otherwise similar students might be causing any difference between the achievement of magnet and non-magnet school students that we observe. To address this issue, we used three different methods to estimate the effects of magnet schools: comparisons of magnet school lottery participants, propensity score matching, and regression 4-44 analyses. Each of these methods has strengths and limitations. Comparisons of lottery participants exploits the random selection of magnet school students from among those who apply to ensure that treatment group students (those enrolled in magnet schools) are similar on average to comparison group students (those who applied, but were denied admission to magnet schools) in both observed and unobserved ways. Random assignment is considered the gold standard for analysis of this kind. The limitation of lottery based analyses is that they can only be applied to a subgroup of magnet schools and magnet school students. Propensity score matching and regression analysis use statistical procedures to ensure that magnet school students are compared to similar non-magnet school students. These methods can provide defensible estimates of magnet school effects, particularly when pre-treatment measures of achievement are available, and also have the advantage of being applicable for a broader set of magnet schools. These methods, however, do not provide controls for potential unobserved differences between magnet school and non-magnet schools in the same way as lottery based comparisons. Full descriptions of three sets of analyses we conducted and the results are provided in Appendix E. Comparisons across the various estimates presented in Appendix E indicate that despite their different advantages and disadvantages, the three methods provide very similar estimates of magnet school effects. Thus, the analyses in Appendix E suggest that regression methods, which control for students’ prior achievement and demographic characteristics, provide sound estimates of the average effect of magnet schools. We report the results of those regressions in this section. We report estimates of average achievement effects for a set of 12 interdistrict magnet high schools and another set of six interdistrict middle schools.26 These schools include all of the fullday interdistrict magnet high schools and all but two of the interdistrict magnet middle schools that serve students from Hartford, New Haven, or Waterbury.27 We focus on estimating the effects of the interdistrict magnet high schools on 10th grade reading and math Connecticut Academic Performance Test (CAPT) exams, and the effects of the interdistrict magnet middle schools on 8th grade math and reading CMTs. The CAPT is the high school statewide testing program administered by the state. To construct our student sample for the analysis of the 12 interdistrict magnet high schools, we asked officials at the State Department of Education to extract the 2005-06 and 2006-07 10th grade CAPT records for all of the students attending either one of those interdistrict magnets or a high school in a district that sends students to one of those interdistrict magnets. We then asked the state officials to match those student records to records from earlier 8th grade and 6th grade test score files. Our sample for the middle school analysis was constructed in an analogous manner. A sample of 1,730 magnet high school students and 13,507 students from feeder districts were extracted from the 10th grade test score files. State officials were able to match 71.5% of these students to 8th grade test score records and 58.7% to 6th grade test score records. Longitudinal matching rates were higher for 8th graders. Of the sample of 1,248 magnet school students and 18,563 students from feeder districts extracted from the 8th grade test score files, 26 High schools here are schools that serve grades 9-12, and middle schools are schools that begin in grade 6 or 7. Four of the six “middle schools” end in grade 8, but two serve high school grades as well. 27 Two interdistrict magnet schools that serve students from New Haven start in Grade 5 and are not included in this analysis. 4-45 state officials were able to match 80.4% to 6th grade test score records and 65.9% to 4th grade test score records. Table E.1 in Appendix E presents summary statistics on the sample of 10th grade students who we were able to match to an 8th grade test score record and the sample of 8th grade students who we were able to match to a 6th grade test score. Do Interdistrict Magnet Schools Improve Student Achievement? Table 4.1 summarizes the results of the high school and middle school analyses. The left side of the table shows the estimated effect of magnet high schools on 10th grade math and reading scores, after controlling for 8th grade scores or 8th and 6th grade scores. The right side of the tables shows the estimates of middle school magnet effects on 8th grade math and reading scores. These estimates suggest that, on average, high school magnet schools have a positive effect on reading and math scores and middle school magnet schools have a positive effect on reading scores. Table 4.1 - Estimates of the Average Effect of Interdistrict Magnet Schools on Student Achievement, High Schools and Middle School High School Magnets Middle School Magnets Grade 10 Math Using Only Grade 8 Test Scores 4.601* (2.105) Using Grade 6 & 8 Test Scores 3.908* (2.018) Grade 8 Math Using Only Grade 6 Test Scores 2.902 (2.827) Using Grade 4 & 6 Test Scores 3.063 (2.848) Grade 10 Reading 4.705* 4.547* Grade 8 Reading 8.328* 8.045* (2.053) (2.066) (2.476) (2.519) OLS regressions include age, gender, ethnicity, free-lunch eligibility, special education status, and year as well as pre-treatment test scores and magnet enrollment indicator. The figures in parentheses are standard errors. Standard errors are adjusted for clustering at the school level. * Indicates statistically significant at 0.05 level. The high school results indicate that, on average, interdistrict magnet schools have had positive and statistically significant effects on both 10th grade math and 10th grade reading achievement. The estimated effects of attending a magnet school are not only statistically significant, but also significant from a policy perspective. The estimated effects on reading are 0.10 standard deviations, and on math are between 0.09 and 0.11 standard deviations. These represent the effects of two years of exposure. If we assume similar effects over the second half of these students’ high school careers, these estimates imply effect sizes of 0.20 for reading and between 0.18 and 0.22 for math. The results for middle schools show a mixed pattern of effects across reading and math. On average, middle school magnets have small, statistically insignificant effects on 8th grade math scores. The estimated effects on 8th grade reading, in contrast, are positive, statistically significant, and large (given the context of these scores). For reading, we see an effect of 0.180.19 standard deviations. To give the reader a sense of how large these effect sizes are it is helpful to consider the achievement gap between White and Asian students and Black and Latino students. Table 4.2 4-46 describes average test score performance in math and reading for city, suburban, White, and minority students. Minority-White test score gaps in standard deviation units are 0.91 in reading in grades 8 and 10, 0.95 in grade 8 math, and 1.04 in grade 10 math. Thus, the estimates we see for middle school suggest the magnet school effect is roughly 19.8% of the overall minorityWhite achievement gap in math. In high school, the magnet school effect is roughly 11.0% (reading) and 8.7% (math) of the overall minority-White achievement gap for two years and 22.0% (reading) and 17.4% (math) of the overall achievement gap if we presume the same gains in the second two years of high school. Table 4.2 - Average Test Scores, By Student's Residence and Racial Background Grade 8 Math Grade 8 Reading Grade 10 Math Grade 10 Reading Standard Deviation 46.7 44.7 49.8 48.0 Suburban Students 246.9 242.5 246.6 241.3 City Students 221.0 221.2 205.8 241.3 White 259.8 258.5 258.8 251.2 Minority 215.5 217.6 207.0 207.3 Calculated using 2006-07 CMT and CAPT data for all students described in Appendix E, Table E.1 Do Interdistrict Magnet Schools Improve Student Achievement for Central City Students? The preceding analysis considers the average effect of magnet schools. It is also important to know if these effects vary across students and schools. Such variations are particularly important because Connecticut’s interdistrict magnet program is partially motivated by the Sheff settlement, which sets out to decrease racial isolation and increase the achievement of central city minority students. Thus, we also estimated the effects of magnet schools by students’ residence and by the racial composition of their magnet school. In Table 4.3, we split both our high school and middle school samples into students who reside in Hartford, New Haven or Waterbury (city students) and students who reside in other districts (suburban students). We then calculated regression estimates of magnet school effects separately using these two subsamples. The left side of Table 4.3 presents the estimated effects of attending a magnet high school on 10th grade test scores. These estimates indicate that interdistrict magnet high schools have had significant, positive effects on both the math and reading achievement of their city students. The coefficient estimates imply that attending an interdistrict magnet high school increases the 10th grade math achievement of central city students by 0.12 standard deviations and the 10th grade reading achievement of city students by 0.15 standard deviations. Comparing these effect sizes to the magnitude of the minority-White achievement gap, these effects would close the gap in reading by 16.5% and in math by 11.5%. The estimated effects of interdistrict magnet high schools on their suburban students are smaller than the estimated effects on city students and are not statistically different than zero. The story is different for middle school magnets. The estimates on the right side of Table 4.3 indicate that interdistrict magnet middle schools have small, statistically insignificant effects on 4-47 city students and larger, statistically significant impacts on suburban students. What’s more, the estimated effects on 8th grade reading for suburban students are statistically different than the estimated effects on 8th grade reading for city students. These results suggest that the positive, average effects of interdistrict magnet middle schools are driven primarily by positive effects on suburban students. They also suggest that interdistrict magnet middle schools do not have as large of an effect as interdistrict magnet high schools on the achievement of students from Connecticut’s cities. Table 4.3 - Estimated Magnet School Effects on Student Achievement, By Student's Residence High School Magnets Middle School Magnets Grade 10 Math City Students 5.339* (2.222) Suburban Students 3.208 (2.470) Grade 8 Math City Students 1.523 (2.986) Suburban Students 5.846* (2.544) Grade 10 Reading 6.629* 3.643 Grade 8 Reading 3.077 12.000* (2.058) (2.819) (2.580) (2.291) OLS regressions include age, gender, ethnicity, free-lunch eligibility, special education status, year and pretreatment math and reading test scores from two previous periods. The figures in parentheses are standard errors. They are adjusted for clustering at the school level. * Indicates statistically significant at 0.05 level. Does a Magnet School’s Racial Composition Influence its Effect on the Academic Achievement of Central City Students? Supporters of interdistrict magnet schools suggest that these schools can help central city minority students improve their academic achievement by providing access to less racially isolated learning environments. The first section of this report documented that Connecticut’s magnet schools do, on average, provide less racially isolated environments. However, there is substantial variation in the level of integration these schools achieve. This section investigates the degree to which magnet school achievement effects vary with the racial composition of the magnet school. To explore this question, for each student living in a central city we identified the differences between the percent of White students in their magnet school and the percent of White students in their home district - which we label Δ%White (change in percent White). This difference measures the extent to which a magnet school, on average, reduces racial isolation for the students in their school.28 We then used regression to determine whether the effect of attending a magnet school varied with this measure. We began with the same regressions presented earlier, however, we added an interaction term. This interaction term (Magnet*Δ%White) captures the extent to which the effect of attending a magnet school varies with changes in the extent to which the magnet school relieves racial isolation. By examining this term we are able to see if 28 A better measure would be the difference in the racial composition of the magnet school and the school the student would otherwise attend, but we don’t have that information, and so the difference between the magnet school and the home district is our best approximation. 4-48 the effect of a magnet school depends on how much that school decreases racial isolation for its students. The results of these regressions are presented in Table 4.4. Table 4.4: Variation in Estimated Magnet School Effect on City Students, By Difference in Percent White Between Magnet and Home District High Schools Grade 10 Math Grade 10 Reading Magnet 4.907* 5.315* (2.099) (2.168) Magnet*Δ%White Magnet Magnet*Δ%White 0.049 (0.102) Grade 8 Math -7.948 (4.951) 0.396* (0.147) 0.103 (0.123) Middle Schools Grade 8 Reading -2.381 (4.998) 0.230 (0.153) Δ %White is the difference in percent White students between the magnet school attended by the student and the student's home district. Each column presents results from separate OLS regressions. All regressions are estimated using a sample of students who reside in Hartford, New Haven or Waterbury, and include controls for age, gender, ethnicity, free-lunch eligibility, special education status, year and pretreatment math and reading test scores from two previous periods. The figures in parentheses are standard errors adjusted for clustering at the school level. * Indicates statistically significant at 0.05level. For high school magnets, the coefficients on the magnet*Δ%White interaction term are positive, but are not statistically distinguishable from zero. This indicates that the effect of attending a magnet high school does not depend on the extent to which attending a magnet reduces racial isolation.29 Among middle schools, the relationship between the magnet school effect and the extent to which attending a magnet reduces racial isolation is stronger. The coefficients on the magnet*Δ%White interaction are positive but are larger for middle school magnets than for high school magnets. The interaction term is statistically significant in the Grade 8 math regression but not in the reading regression.30 Because these estimates are based on a small set of schools 29 In a second set of regressions we categorized magnet high schools into three groups based on how much they reduced racial isolation and estimated separate effects for each three groups of schools. The results did not show significant differences in effects between the three groups, confirming the result in Table 4.4. 30 In another set of regressions that categorized magnet middle schools based on how much they decrease racial isolation we found supporting results. In those regressions, the effect of attending a magnet middle school on central city schools is positive only when racial isolation is reduced by more than 40 points. Only two magnet middle school reduce racial isolation of its city students by more than 40 percentage points. This supports the results in Table 4.4, suggesting that the more magnet middle schools are able to reduce isolation, the greater effect they have on city students, but also shows that this result is based on a small number of schools. 4-49 we cannot draw strong conclusions from these results. It appears, however, that the effect of magnet middle schools on city students might depend on their ability to decrease racial isolation. Conclusions On average, interdistrict magnet schools have a statistically significant positive effect on the reading and math achievement of high school students, and on the reading achievement of middle school students. Interdistrict magnet high schools are particularly effective for city students regardless of the degree to which those schools decrease racial isolation. Conversely, interdistrict magnet middle schools show larger effects on suburban students. The average effect of middle schools may be influenced by the degree to which the school is able to decrease racial isolation. Though reasonable people disagree on the size an achievement effect must be before calling it large, these findings are both positive and substantively important for policy makers, practitioners, and students. The Sheff settlement was designed to provide city students increased access to quality schools. It is noteworthy then, that on average, magnet schools are effective for city students. This suggests that on the academic front, the Sheff goal is being met for the city students attending interdistrict magnet schools. 4-50 APPENDIX A: HIGH SCHOOL STUDENT SURVEY Project Title: Understanding Students’ School Experiences Director: Dr. Casey Cobb Casey Cobb is a professor from UConn working on a project for the State of Connecticut to study Connecticut schools. Dr. Cobb and his fellow researchers want to learn more about what students think about their schools. They have created this questionnaire to ask students how they feel about school, their classes, and their classmates. Some of the questions ask students about their perceptions of students of different races and ethnicities, their career and academic expectations, and their interest level in classes. Information gathered from the questionnaire will ultimately help them better understand how schools influence children. In order to do that, they would like you to answer the questions as honestly as you can. The whole questionnaire should take approximately 20 minutes. Your responses on the survey are completely confidential. Only Dr. Cobb and his researchers will see your questionnaires. No one at your school will know how you answered the questions. You can ask as many questions as you like about the study and either Dr. Cobb or his assistant will explain it to you. You may call Dr. Cobb, or ask your parent/guardian to call for you, at any time, if you have more questions about the study (Dr. Cobb phone: 860-486-0253). You do not have to be in this study if you don’t want to. If your parent/guardian turned in a form indicating they do not want you taking the survey, or you choose not to participate on your own, please sit quietly. Please check this box to indicate you have read, understand, and agree to participate in the study. A-1 High School Student Survey Section 1: Thoughts on School Directions: Indicate your level of agreement with each of the following statements by checking either: Strongly Disagree, Disagree, Neutral, Agree or Strongly Agree. Please mark your response clearly and check only one response for each question. 1. I feel like I belong at this school. 2. Getting good grades (As and Bs) is important to me. 3. In most of my classes I want to understand the subject. 4. It is important to me to graduate from high school. 5. It’s important to me that I don’t look smarter than others in class. 6. It’s important to me to do well in school. 7. Most of my friends care about doing well in school. 8. I regularly participate in classroom activities. 9. I usually pay attention in my classes. 10. I work very hard on my school work. 11. In my classes, I want to learn more than what is required. 12. Students at my school value academics. 13. I care about the quality of my classwork. A-2 Strongly Disagree Disagree Neutral Agree Strongly Agree                                                                  Section 2: Teachers Directions: Indicate your level of agreement with each of the following statements by checking either: Strongly Disagree, Disagree, Neutral, Agree or Strongly Agree. Please mark your response clearly and check only one response for each question. 14. My teachers set high academic standards for me. 15. My teachers understand me. 16. My teachers make sure that the work I do really makes me think. 17. My teachers expect me to do my best work. 18. My teachers respect me. 19. My teachers encourage me to try out new ideas (think independently) on academic tasks. 20. My teachers want me to understand my work, not just memorize it. 21. My teachers really listen to what I have to say. 22. My teachers press me to do thoughtful work. 23. My teachers ask me to explain how I get my answers. Strongly Disagree Disagree Neutral Agree Strongly Agree                                                   Directions: For the next few questions, please check the response that most closely reflects your opinion. Please mark your response clearly and check only one response for each question. 24. To what extent have your teachers and counselors encouraged you to attend college?  Strongly encouraged  Somewhat encouraged  Neither encouraged nor discouraged  Somewhat discouraged  Strongly discouraged A-3 25. How much information about college admissions have your teachers and counselors given you (such as SAT, ACT, financial aid, college fairs, college applications)?  A lot  Some  A little  None 26. To what extent have your teachers and counselors encouraged you to take Honors, AP, and/or college prep classes?  Strongly encouraged  Somewhat encouraged  Neither encouraged nor discouraged  Somewhat discouraged  Strongly discouraged 27. What are you getting for grades this year?  As  As/Bs  Bs  Bs/Cs  Cs  Cs/Ds  Ds/Fs A-4 Section 3: More Thoughts on School Directions: Indicate your level of agreement with each of the following statements by checking either: Strongly Disagree, Disagree, Neutral, Agree or Strongly Agree. Please mark your response clearly and check only one response for each question. 28. I feel safe and secure in this school. 29. I feel safe on my way to and from school. 30. Getting to and from school makes it difficult for me to participate in after- school activities (sports, clubs). 31. In this school getting good grades makes you less popular. 32. I usually avoid answering questions in class because I don’t want other students to think I am trying too hard. 33. If I did well on a school assignment, I wouldn’t want other students to see my grade. Strongly Disagree Disagree Neutral Agree Strongly Agree                               Directions: For the next few questions, please check the response that most closely reflects your opinion. Please mark your response clearly and check only one response for each question. 34. How often are your classes interrupted by the misbehavior of other students?  Rarely  About once a class  A few times a class  Several times a class  Most all of class A-5 35. How many of your classes have significant student behavior problems?  None  Less than half  Half  More than half  All of them 36. How often were you late for school over the past two weeks?  0 days  1-2 days  3-5 days  6-8 days  9 or more days 37. How often did you cut or skip a class over the past two weeks?  0 times  1-2 times  3-5 times  6-8 times  9 or more times 38. How many of your close friends try hard to do well in school?  None  Some  Most  All A-6 Section 4: School Climate Directions: Indicate your level of agreement with each of the following statements by checking either: Strongly Disagree, Disagree, Neutral, Agree or Strongly Agree. Please mark your response clearly and check only one response for each question. Strongly Strongly Disagree Disagree Neutral Agree Agree 39. Students of different races in this school need each other.      40. Students of different races have important things to offer each other.      41. After students of different races get to know each other, they find they have a lot in common. 42. Teachers encourage students to make friends with students of different races.      43. In this school everybody is encouraged to be friends.           44. Teachers do not encourage students to make friends with students of different groups. 45. I talk to students of different races only when I have to.                49. All students in this school are treated equally.                50. Some students at this school get more opportunities to do things because of their race.      46. My friends would think badly of me if I ate lunch with students of a different race. 47. Students of different races don’t have much to do with each other at this school. 48. Teachers at this school are fair to all groups of students. A-7 Section 5: Getting to School Directions: For the next few questions, please check the response that most closely reflects your opinion. Please mark your response clearly and check only one response for each question. 51. How do you typically get to school each day?  Bus  Walk  Someone drives me  I drive myself  Other 52. How long does it take you to get to school (in minutes)? ____________ minutes 53. How do you feel about your travel/ride to school?  I don’t mind it  It’s long, but worth the trip  It’s too long  Other (please specify): __________________________________________ A-8 Section 6: Interacting with Other Students Directions: Indicate how often you do each of the following by marking the appropriate answer for each as either: Never, Once a Month, Once a Week, Several Times a Week or Everyday. Please mark your response clearly and check only one response for each question. Once a Once a Several Times a Month Week Week Never Everyday 54. Work together in class with students from a different      race/ethnicity than your own. 55. Play games/sports/clubs with students from a different      race/ethnicity than your own. 56. Spend time socially with students from a different      race/ethnicity than your own. 57. Work together on class assignments outside of class with      students from a different race/ethnicity than your own. 58. Talk with students from a different race/ethnicity than      your own at the lunch table. A-9 Directions: Think of your 10 closest friends. Indicate how many of those 10 closest friends are from each of the following racial or ethnic groups by checking either: None, One or More than One for each. Please mark your response clearly and check only one response for each question. None One More than One 59. Black    60. Latino/a    61. White 62. Asian 63. Multi-racial          Directions: For the next few questions, please check the response that most closely reflects your opinion. Please mark your response clearly and check only one response for each question. 64. How much tension exists in your school between students of different racial or ethnic groups?  None  Very little  Some  A lot  Don’t know 65. How do you believe your school experiences have impacted your ability to understand members of other races and ethnic groups?  Helped a lot  Helped somewhat  Had no effect  Did not help  Hurt my ability A-10 The next set of questions asks how you feel toward members of various racial/ethnic groups. Please be as honest as you can. Your responses will remain confidential. Directions: Please indicate how close you feel to each group by circling a number on the 7 point scale where 1 means “not at all close” and 7 means “extremely close.” Please mark your response clearly and check only one response for each question. Not at All Extremely Close Close 66a. Black 1 2 3 4 5 6 7 66b. Latino/a 1 2 3 4 5 6 7 66c. White 1 2 3 4 5 6 7 66d. Asian 1 2 3 4 5 6 7 66e. Multi-racial 1 2 3 4 5 6 7 Directions: Please indicate how comfortable you feel toward each group by circling a number on the 7 point scale where 1 means “not at all comfortable” and 7 means “extremely comfortable.” Please mark your response clearly and check only one response for each question. Not at All Extremely Comfortable Comfortable 67a. Black 1 2 3 4 5 6 7 67b. Latino/a 1 2 3 4 5 6 7 67c. White 1 2 3 4 5 6 7 67d. Asian 1 2 3 4 5 6 7 67e. Multi-racial 1 2 3 4 5 6 7 Directions: For the next few questions, please check the response that most closely reflects your opinion. Please mark your response clearly and check only one response for each question. 68. What grades did you get in eighth grade?  As  As/Bs  Bs  Bs/Cs  Cs  Cs/Ds  Ds/Fs A-11 69. Which best describes your ENGLISH class: (If you have more than one English class, pick the one that is required by your school.)  Basic  College Preparatory  Honors, AP or International Baccalaureate  A Mix of Levels  Don’t know 70. How many students in your ENGLISH class are from racial or ethnic groups that are different from your own?  A few  Quite a few, but less than half  About half  Most Section 7: Your Future Directions: For the next few questions, please indicate your own level of interest in each by checking either: Very interested, Interested, Somewhat interested, Not interested or, if you have never thought about it, check “Hadn’t considered.” Please mark your response clearly and check only one response for each question. Very Somewhat Not Hadn’t Interested Interested Interested Interested Considered 71. Taking a foreign language class after high school.      72. Taking a course focusing on other cultures after high school. 73. Attending a racially/ethnically diverse college campus. 74. Speaking a foreign language. 75. Living in a racially/ethnically diverse neighborhood when you are an adult. 76. Working in a racially/ethnically diverse setting when you are an adult. A-12                          Directions: Indicate your level of agreement with each of the following statements by checking either: Strongly Disagree, Disagree, Neutral, Agree or Strongly Agree. Please mark your response clearly and check only one response for each question. Strongly Strongly Disagree Disagree Neutral Agree Agree 77. Your race      ...will prevent you from getting the kind of job you want to have. 78. Not knowing the right people      ...will prevent you from getting the kind of job you want to have. Directions: For the next few questions, please check the response that most closely reflects your thoughts about your future. Please mark your response clearly and check only one response for each question. 79. Have you taken or are you planning to take the SAT or ACT for college admission?  No  Maybe  Yes  Not sure what these are 80. As things stand now, what educational degree do you expect to get?  Less than high school graduation  High school graduation or GED  Complete a 2-year program at a community college or vocational school  Graduate from a 4-year college  Obtain a graduate degree (e.g., Master’s degree, Law or Medical degree, Ph.D.)  Don’t know A-13 81. Do you plan to continue your education right after high school or at some time in the future?  Yes, right after high school  Yes, after taking some time off from school  No, I don’t plan to continue my education after high school  I don’t know if I will continue my education after high school 82. Write in the name of the job or occupation that you expect to have at age 30.  ____________________________  I don’t expect to work when I’m 30  Don’t know Section 8: Final Questions About You Directions: For these final questions, please answer provide the information requested about yourself. Your responses and identity will be held completely confidential. 83. When were you born? Month Day 1 Year 9 84. Are you...  Male  Female 85. Are you Hispanic or Latino/a? (mark only one)  No, not Hispanic or Latino/a  Yes, Mexican, Mexican American, Chicano  Yes, Puerto Rican  Yes, Cuban  Yes, another Hispanic, Latino/a, or Spanish origin: __________________________ (please print) A-14 86. Please select one or more of the following choices to describe your race. Please be more specific whenever possible.  White (example, Polish):__________________  Black (example, African American, Caribbean):_________________  Asian: (example, Korean, Korean-American):__________________  Pacific Islander (example, Samoan):________________  American Indian or Alaska Native (example, Pequot):________________  Multiracial:_______________________  Other race:_______________________ 87. What is the main language that your family speaks at home?  English  Spanish  Chinese language (Mandarin or Cantonese)  Other Asian language  Other (please specify): ______________________ 88. What is your mother or female guardian’s highest level of education? (If you have both a mother and a female guardian, answer for the one with whom you are currently living.)  Less than high school graduation  High school graduation or GED  2-year degree from a community college or vocational school  4-year college degree (Bachelor’s)  Graduate degree (e.g., Masters, Law, Medicine, Ph.D.)  Don’t know 89. What town or city do you live in? ____________________________________ A-15 90. How many people in your home NEIGHBORHOOD are from racial and ethnic groups that are different from your own?  A few  Quite a few, but less than half  About half  Most 91. What grade are you in? (circle one) 9th 10th 11th 12th 92. In what grade did you start in this school? (circle one) 6th 7th 8th 9th 10th 11th 12th Directions: Please indicate whether you “have” or “do not have” each of the following items in your home by checking the appropriate response for each. Please mark your response clearly and check only one response for each question. Have Do not have 93. A specific place for study   94. A daily newspaper   95. A computer   Directions: Please indicate how often in the first half of this school that you have discussed each of the following with either or both of your parents or guardians by checking either: Never, Rarely, Sometimes or Often. Please mark your response clearly and check only one response for each question. Never Rarely Sometimes Often 96. Selecting courses or programs at school.     97. School activities or events of particular interest to you.     98. Your grades.     99. Going to college.     100. Help with your homework.     A-16 Please indicate the school in which you are taking this survey:  Academy of Information Technology - Stamford  Hill Regional Career High School  Academy of Performing Arts - Norwalk  Hillhouse High School  ACES Education Center for the Arts  Hyde Leadership High School  ACT Comprehensive Arts Magnet High School - Windham  Manchester High School  Big Picture High School  Metropolitan Business High School  Capital Preparatory Magnet School - Hartford  Metropolitan Learning Center  Center for Global Studies - Norwalk  New Haven Academy  Collaborative Alternative Magnet - North Branford  Pathways to Technology  Cooperative Arts and Humanities High School  Regional Center for the Arts - Bridgeport  Connecticut International Baccalaureate Academy  Science & Technology Magnet High School of Southeast CT - New London  East Hartford High School  Simsbury High School  East Haven High School  Sport and Medical Sciences Academy – Hartford  Great Path Academy at MCC  University High School for Science and Engineering  Greater Hartford Academy of the Arts  Waterbury Arts Magnet School  Greater Hartford Academy of Math and Science  Weaver High School  Hamden High School  West Haven High School  Hartford Public High School  Wilbur Cross High School  High School in Community  Other: _______________________________________  Hill Regional Career High School Did you attend a magnet or charter school any time during grades K-8?   No Yes, name of school:_____________________________ A-17 Thank you for participating in our survey. A-18 You may remove this page and take it with you. Dear Student, Thank you for taking our survey about your school experiences. We realize that asking about personal topics (such as your goals, race, school, and family background) can be uncomfortable and you may be wondering why we asked those questions. We will try to explain our thinking below. Connecticut is trying to make it possible for all students to go to a school that provides access to opportunities for future life outcomes. One way to do this is to allow students to attend different schools, such as magnet, charter, and technical schools. This survey is designed to help the state understand how its magnet school strategy is working. The questions we asked you will give the state information about students’ experiences in these schools. Magnet schools were created to make it possible for students of different backgrounds to go to school together. Through our survey we are trying to see if students in different kinds of schools have different thoughts on school, their peers, race, and their future. That sounds like an easy thing to do but it is very hard to ask about some of these issues without offending people. We decided to ask about your experiences because we feel the state should know how its magnet school policies are working, Please feel free to ask us any questions you might have about the items on the survey or how we will report the results. You can do that today by asking a UConn researcher in the room, or later by phone or by email. Our contact information is below. We are also willing to return to your school should you wish to have a discussion about this in more detail. Please contact your principal to make this request. Thank you again for your willingness to share your thoughts. Your responses will help shape future state policy on education. Best wishes, Casey Cobb casey.cobb@uconn.edu 860.486.0253 Bob Bifulco robert.bifulco@uconn.edu 860.570.9029 A-19 Courtney Bell courtney.bell@uconn.edu 609.243.6594 APPENDIX B: SURVEY SAMPLING FRAME Magnet Sampling Frame Academy of Information Technology – Stamford Survey Participation Status Special Type (Outside sample area) LEA Grades Stamford 9-12 Academy of Performing Arts - Norwalk ACES Education Center for the Arts - New Haven (Outside sample area) 1/2 Day CES 9-12 Declined 1/2 Day ACES 9-12 Big Picture High School Completed New Bloomfield 9-11 Capital Preparatory Magnet School Never returned call Hartford 6-12 Norwalk 9-12 Completed ACES 7-12 Not completed New Haven 9-12 Completed East Hartford 9-12 EASTCONN 9-12 CREC 10-12 CREC 9-12 Hartford 6-12 CREC 9-12 Phasein Center for Global Studies - Norwalk Collaborative Alternative Magnet - North Branford Cooperative Arts and Humanities HS - New Haven Ct International Baccalaureate Acad - E Hartford EASTCONN'S Arts Magnet High School Windham (Outside sample area) Great Path Academy at MCC - Manchester Greater Hartford Academy of the Arts – Hartford Completed Greater Hartford Classical Magnet Gtr Hartford Acad. of Math and Science – Hartford Declined High School in Community - New Haven Hill Regional Career High School - New Haven Completed New Haven 9-12 Completed New Haven 9-12 Hyde Leadership High School- New Haven Metropolitan Business High School - New Haven Fall 08 admin New Haven 9-12 Completed New Haven 9-12 Metropolitan Learning Center - Bloomfield Completed CREC 6-12 New Haven Academy - New Haven Completed New Haven 9-12 Completed Completed Completed B-1 1/2 day 1/2 Day 1/2 Day Pathways to Technology - Hartford Fall 08 admin Hartford 6-12 Regional Center for the Arts - Bridgeport Science & Technology Magnet HS of SE CT - New London Sport and Medical Sciences Academy – Hartford University High School for Science and Engineering (Outside sample area) 1/2 Day CES 9-12 Never returned call New New London 9-12 Completed Hartford 9-12 Completed Hartford 9-12 Waterbury Arts Magnet School Outside Waterbury 9-12 Central City HS Declined Central City 9-12 Central City HS Completed Central City 9-12 Central City HS Fall 08 admin Central City 9-12 Central City HS Never returned call 9-12 Outer Ring Urban HS Never returned call Outer Ring Urban HS Declined Outer Ring Urban HS Fall 08 admin Central City Outer Ring City Outer Ring City Outer Ring City Suburban HS Declined Suburb 9-12 Suburban HS Not completed Suburb 9-12 Suburban HS Completed Suburb 9-12 Non-Magnet Sampling Frame B-2 9-12 9-12 9-12 APPENDIX C: SCALE CONSTRUCTION AND INTERNAL CONSISTENCY Below, items are listed for each survey construct. Cronbach’s alpha (∝) is reported for each scale. Cronbach’s alpha is a measure of internal consistency, and indicates the degree of reliability for a scale. Alphas that are .65 and above are deemed sufficient for attitudinal instruments. Items named with an “R” indicate the item was reverse coded. Academic Aspirations (∝ = .867) Items 2, 3, 4, 6, 8, 9, 10, 11, 13 Academic Press (∝ = .878) Items 14, 16, 17, 19, 20, 22, 23 Social Closeness with Minorities (∝ = .868) Items 66a, 66b, 66e, 67a, 67b, 67e Social Closeness with Whites/Asians (∝ = .801) Items 66c, 66d, 67c, 67d Teacher-Student Relationship (∝ = .854) Items 15, 18, 21 Social Sanctions for Achievement (∝ = .648) Items 5, 31, 32, 33 Classroom Disruptions (∝ = .759) Items 34, 35 Peer Academic Norms (∝ = .634) Items 7, 12, 38 Safety and Belonging (∝ = .684) Items 1, 28, 29 (required standardization of scores due to asymmetric response categories) Equal Status: Contact Theory (∝ = .723) Items 48, 49, 50R Interaction: Contact Theory (∝ = .731) Items 45, 46, 47 Interdependence: Contact Theory (∝ = .755) Items 39, 40, 41 Supportive Norms: Contact Theory (∝ = .650) Items 42, 43 C-1 School Influence on College Aspirations (∝ = .630) Items 24, 25, 26 College Expectations (∝ = .628) Items 79, 80, 81 (required standardization of scores due to asymmetric response categories) Interracial Interaction (∝ = .858) Items 54, 55, 56, 57, 58 Future Multicultural Interest (∝ = .848) Items 71R, 72R, 73R, 74R, 75R, 76R C-2 APPENDIX D: DESCRIPTIVE SUMMARIES FOR MAGNET SCHOOLS AND STUDENTS The following results are for magnet schools only. Results are presented in the aggregate (i.e., lower and upper grades were combined). SCHOOL CLIMATE MEASURES Academic Press For the most part, magnet school students reported that teachers fostered high degrees of academic press. Seven out of 10 magnet students reported that teachers “set high academic standards” and “make sure the work [they] do really makes [them] think.” Roughly 8 out of 10 students indicated that their teachers expected them to do their best work and to explain how they get their answers. Table D.1 - Academic Press Item Scores My teachers want me to understand my work, not just memorize it. My teachers press me to do thoughtful work. My teachers encourage me to try out new ideas (think independently) on academic tasks. My teachers set high academic standards for me. My teachers make sure that the work I do really makes me think. My teachers expect me to do my best work. My teachers ask me to explain how I get my answers. SD D % N A SA Mean (SD) n 2.9 5.6 18.5 41.5 31.5 3.93 (.99) 1,274 2.6 6.5 29.9 43.0 18.0 3.67 (.93) 1,258 2.9 6.9 23.3 43.4 23.4 3.78 (.98) 1,259 3.1 5.2 21.8 44.9 25.1 3.84 (.96) 1,276 2.7 8.0 28.4 40.2 20.7 3.68 (.98) 1,268 1.7 3.3 12.9 42.3 39.8 4.15 (.89) 1,267 2.5 6.0 23.8 42.9 24.9 3.82 (.96) 1,273 D-1 School Influence on College Aspirations While well over half (56%) of magnet students reported that their teachers and counselors “strongly encouraged” them to attend college, a smaller percentage (29.6%) “strongly encouraged” them to take honors, AP, and/or college prep classes in pursuit of that goal. Moreover, 25.9% of the students reported that teachers and counselors gave them “a lot” of information with regard to college admissions. One-fifth (20.6%) indicated that magnet staff provided only “a little” college admissions information, and 13.9% reported “none.” Somewhat Discouraged Strongly Discouraged 1.2 1.0 1.63 (.84) 1,240 29.6 32.9 30.6 3.4 3.5 2.18 (1.01) 1,268 2.22 (.99) 1,272 None Neither Encouraged nor Discourag 14.0 A little Somewhat Encouraged How much information about college admissions have your teachers and counselors given you (such as SAT, ACT, financial aid, college fairs, college applications)? 27.7 Some To what extent have your teachers and counselors encouraged you to take Honors, AP, and/or college prep classes? 56.0 A lot To what extent have your teachers and counselors encouraged you to attend college? Strongly Encouraged Table D.2 - School Influence on College Aspirations Item Scores % 25.9 39.5 20.6 13.9 D-2 Mean (SD) n Peer Academic Norms Magnet school students appear to have high achievement motivation. The majority of magnet school students (62.8%) agreed or strongly agreed that “most of their friends care about doing well in school.” Similarly, the majority of students (64.1%) reported that most or all of their close friends “try hard to do well in school.” However, a smaller percentage (43.3%) believed students at their school “value academics.” One-third of the students indicated only some of those close friends tried hard to do well in school. There is a distinction between performing successfully in school and embracing academics. Table D.3 - Peer Academic Norms Item Scores % SD D N A SA Mean (SD) n Most of my friends care about doing well in school. 2.5 6.4 28.3 41.7 21.1 3.72 (.95) 1,271 Students at my school value academics. 3.3 11.7 41.7 30.6 12.7 3.38 (.96) 1,260 None Some Most All 2.80 (.77) 1,269 How many of your close friends try hard to do well in school? 2.8 33.1 45.6 18.5 Social Sanctions for Achievement Overall, magnet school students do not appear to place sanctions on peers who do well in school. The overwhelming majority disagreed that getting good grades made you less popular, that they avoided answering questions in class, and that they wouldn’t want other students to see high academic grades. There was one exception where a plurality of students did not indicate any disagreement with the statement: “It’s important to me that I don’t look smarter than others in class.” Table D.4 - Social Sanctions for Achievement Item Scores % SD D N It’s important to me that I don’t look smarter than others in class. In this school getting good grades makes you less popular. I usually avoid answering questions in class because I don’t want other students to think I am trying too hard. If I did well on a school assignment, I wouldn’t want other students to see my grade. A SA Mean (SD) n 26.7 30.7 29.6 9.0 4.0 2.33 (1.08) 1,257 44.5 33.3 16.5 3.3 2.5 1.86 (.97) 1,245 44.6 34.9 13.5 4.9 2.1 1.85 (.97) 1,259 45.9 27.4 17.3 6.1 3.3 1.94 (1.08) 1,263 D-3 Teacher-Student Relationship Nearly 60% (59.7%) of magnet students believed that their teachers respected them. However, one out of five magnet school students indicated that their teachers did not understand them. Table D.5 - Teacher-Student Relationship Item Scores My teachers respect me. My teachers really listen to what I have to say. My teachers understand me. SD 5.5 D 8.3 % N 26.6 A 37.0 SA 22.7 Mean (SD) 4.04 (1.09) n 1,266 6.8 12.0 33.6 33.2 14.5 4.49 (1.08) 1,270 6.1 14.1 33.0 35.6 11.2 4.38 (1.04) 1,273 Classroom Disruptions Magnet school students reported fairly significant frequencies of classroom disruptions. Nearly 4 out of 10 (38.5%) students indicated that at least half of their classes had “significant student behavior problems.” Three out of 10 (30.4%) students also reported that their classes are interrupted by the misbehavior of other students “several times a class” or “most all of class.” Table D.6 - Classroom Disruption Item Scores Mean (SD) % How often are your classes interrupted by the misbehavior of other students? How many of your classes have significant student behavior problems? Rarely About once a class A few times a class Several times a class Most all of Class 24.8 16.0 28.7 18.3 12.1 None Less than half Half More than half All of them 23.2 38.2 20.7 10.9 6.9 D-4 n 2.77 (1.33) 1,253 2.40 (1.16) 1,270 STUDENT ATTITUDE MEASURES Academic Aspirations Overall, magnet school students appear to have high academic aspirations. Over half the students (55.7%) strongly agreed and another 30.1% agreed that getting good grades was important to them. Over 60% (61.5%) strongly agreed that it was important to them to do well in school. About one-fifth of the sample (22.0%), however, were neutral on the statement “I usually pay attention in my classes,” and another 3.8% disagreed with it. Table D.7 - Academic Aspirations Item Scores Getting good grades (As and Bs) is important to me. In most of my classes I want to understand the subject. It is important to me to graduate from high school. It’s important to me to do well in school. I regularly participate in classroom activities. I usually pay attention in my classes. I work very hard on my school work. In my classes, I want to learn more than what is required. I care about the quality of my classwork. SD D % N 1.2 1.6 11.5 30.1 55.7 4.38 (.84) 1274 1.3 2.2 10.8 40.9 44.9 4.26 (.83) 1,272 1.0 1.0 2.6 7.1 88.4 4.81 (.61) 1,262 .9 1.5 1.3 1.2 2.8 1.5 5.1 2.5 5.0 10.7 6.6 22.7 22.0 28.4 36.0 29.5 46.7 52.0 41.8 32.7 61.5 24.0 22.1 23.6 17.9 4.49 (.77) 3.87 (.89) 3.91 (.81) 3.82 (.89) 1,265 1,269 1,270 1,271 3.52 (.99) 1,267 1.2 2.8 16.9 49.0 30.2 4.04 (.83) 1,270 D-5 A SA Mean (SD) n College Expectations Slightly over 80% of magnet students indicated they have taken or are planning to take a college admissions exam (i.e., SAT or ACT). Half (49.5%) of the students indicated they plan on obtaining a graduate degree. Thirty-one percent (30.8%) reported they expect to obtain a 4-year college diploma as their highest educational degree. Eighty percent (80.3%) plan on continuing on to post-secondary education immediately following high school. Table D.8 - College Expectations Item Scores % Have you taken or are you planning to take the SAT or ACT for college admission? No Maybe Yes Not Sure what these are n 4.4 12.0 80.6 3.0 1,245 5.8 30.8 Don’t know 5.2 Obtain a graduate degree .8 Complete a 2-year program at a comm..college or voc. school Graduate from a 4year college High school grad or GED As things stand now, what educational degree do you expect to get? Less than high school graduation % 49.5 8.0 n 1,232 Do you plan to continue your education right after high school or at some time in the future? 80.3 14.7 D-6 1.9 I don’t know if I will continue my education after high school No, I don’t plan to continue my education after high school Yes, after taking some time off from school Yes, right after high school % 3.2 n 1,227 Attendance Well over half the students (56.5%) in the sample reported not being late to school over a twoweek period before taking the survey. Over three-quarters (78.6%) indicated that they did not skip a class during that time period. Seventeen percent (17.4%) reported being late to school at least three to five times during the two weeks prior to the survey. Roughly 8% of the students reported skipping at least three to five classes during that same time period. Table D.9 - Attendance Item Scores % 0 1-2 3-5 6-8 9+ n How often were you late for school over the past two weeks?(# days) 56.5 26.1 10.9 2.3 4.2 1,268 How often did you cut or skip a class over the past two weeks? (# times) 78.6 13.8 5.0 0.7 1.9 1,271 Safety and Belonging The majority of magnet students felt safe and secure in their school (60.5%). Roughly two-thirds (68.6%) of the sample reported they felt safe on the way to and from school. Finally, 65.0% felt like they belonged at their school. In contrast, minimally one-fourth of the magnet students were neutral or negative in terms of both safety and sense of belonging. Table D.10 - Safety and Belonging Item Scores I feel safe and secure in this school. I feel safe on my way to and from school. I feel like I belong at this school. SD 5.5 3.2 5.1 D 6.8 5.0 6.3 D-7 % N 27.3 23.1 23.7 A 35.5 42.9 36.2 SA 25.0 25.7 28.8 Mean (SD) 3.68 (1.09) 3.83 (.98) 3.77 (1.09) n 1,272 1,262 1,272 Intergroup Relations: Equal Status, Interaction, Interdependence, Supportive Norms (Contact Theory) There appears to be general agreement that students of various backgrounds are treated equally, interact with one another, work together on common goals, and that such interactions are supported by authorities in this sample of magnet schools. However there are some aspects of intergroup relations that are less favorable. For instance, 30.0% of the students neither agreed nor disagreed with the statement “students of different races have important things to offer each other.” Another 9.5% of students disagreed or strongly disagreed with that statement. There was also less agreement in terms of the school providing supportive norms for students of diverse backgrounds to interact. Nearly 40% of students neither agreed nor disagreed with the statement “teachers encourage students to make friends with students of different races.” Another 11.2% disagreed or strongly disagreed with this statement. Table D.11 - Equal Status, Interaction, Interdependence, and Supportive Norms Item Scores Equal Status % SD D N A SA Mean (SD) n Teachers at this school are fair to all groups of students. All students in this school are treated equally. Some students at this school get more support 9.0 10.6 23.4 28.3 28.7 3.57 (1.25) 1,257 12.4 15.3 24.5 24.5 23.3 3.31 (1.32) 1,255 6.4 12.6 27.7 22.7 30.6 3.58 (1.22) 1,265 D % N A SA Mean (SD) n 54.4 27.7 11.0 4.2 2.6 1.73 (.99) 1,259 63.9 22.2 9.5 3.0 1.4 1.56 (.89) 1,263 43.9 27.5 19.6 5.9 3.2 1.97 (1.07) 1,261 Interaction SD I talk to students of different races only when I have to. My friends would think badly of me if I ate lunch with students of a different race. Students of different races don’t have much to do with each other at this school. Interdependence Students of different races in this school need each other. Students of different races have important things to offer each other. After students of different races get to know each other, they find they have a lot in common. SD D % N A SA Mean (SD) n 7.7 12.3 42.0 25.6 12.3 3.23 (1.06) 1,264 3.7 5.8 30.0 39.7 20.7 3.68 (.99) 1,259 2.1 3.9 25.1 45.9 23.0 3.84(.90) 1,266 SD D % N A SA Mean (SD) n 6.9 14.3 39.9 25.8 13.1 3.24 (1.07) 1,255 7.1 11.9 30.0 31.4 19.6 3.44 (1.14) 1,248 Supportive Norms Teachers encourage students to make friends with students of different races. In this school everybody is encouraged to be friends. D-8 Social Closeness On average, magnet students reported being relatively “close to” and “comfortable towards” members of all races and ethnicities. The midpoint of the 7-point scale on both closeness and comfortableness is four. Mean scores for all groups were over that midpoint. However there is some degree of variability around those means. Note that these scores have considerably more meaning when considering the background of the individual respondent and their school context. For example, inferences can be made relative to an individual’s level of self-reported closeness with ingroup and outgroup members. We make those more nuanced inferences in the body of this report (Section 4). Table D.12 - Social Closeness Measures 7-point scale How close do you feel to each group? Black Latino/a White Asian Multi-racial 1 Not at All Close 7 Extremely Close Mean 5.67 5.44 5.35 4.19 5.50 How comfortable do you feel Not at All Extremely toward each group? Comfortable Comfortable Black 6.04 Latino/a 5.96 White 5.84 Asian 5.30 Multi-racial 5.93 D-9 SD 1.60 1.70 1.75 2.14 1.66 n 1,205 1,198 1,196 1,174 1,195 1.50 1.51 1.60 2.01 1.53 1,202 1,199 1,192 1,181 1,195 Racial Tension Nearly 30% of the students reported there was no racial tension at their school. Thirty-six percent (36.0%) believed there was “very little” racial tension, and 21.1% reported there was “some” racial tension. Less than 6% of the sample indicated there was “a lot” of racial tension. Thirty percent (30.2%) of the students believed that their school experiences have “helped a lot” in terms of impacting their ability to understand members of other races and ethnicities. A third believed their school has “helped somewhat” in this vein. Another 28.7% thought it had “no effect” on their understanding of other racial and ethnic groups. Only 5% thought it “did not help” and less than 3% thought it “hurt” their ability to understand. Table D.13 - Racial Tension at the School % How much tension exists in your school between students of different racial or ethnic groups? How do you believe your school experiences have impacted your ability to understand members of other races and ethnic groups? None Very little Some A lot Don’t know n 29.2 36.0 21.1 5.6 8.1 1,248 Helped a lot Helped somewhat Had no effect Did not help Hurt my ability n 30.2 33.4 28.7 5.2 2.5 1,247 D-10 Close Friends by Race/Ethnicity The survey asked students the following question to gather information on their closest friends by race and ethnicity: Think of your 10 closest friends. Indicate how many of those 10 closest friends are from each of the following racial or ethnic groups by checking either: None, One, or More than One for each. At least three-quarters of respondents indicated they have more than one friend who is Black, Latino/a, Multi-racial, or White. The lone exception was with friends of Asian descent, likely due to the relatively small percentages of Asian-Americans in the magnet schools from our sample. Nearly 80% of magnet students reported having more than one close friend who is White. This is noteworthy given that 6 of 13 magnets in the sample enroll over 75% students of color. One caveat, however: as the survey question was worded, one cannot be certain that the friends identified by respondents are classmates at their magnet school. Table D.14 - Close Friends by Race/Ethnicity Item Scores % Number of Close More Friends than One None One Black 8.1 9.6 82.4 Latino/a 11.5 14.6 73.9 White 8.9 12.2 78.9 Asian 40.8 28.4 30.7 Multi-racial 10.8 14.2 75.0 D-11 n 1,224 1,209 1,212 1,174 1,203 Future Multicultural Interest Almost half the sample (47.9%) said they are interested or very interested in taking a foreign language class after high school. Over 60% (61.1%) of the students reported they were interested or very interested in attending a racially and ethnically diverse college campus. Almost twothirds of the sample (65.5%) is interested or very interested in speaking a foreign language. Over 38% reported being “very interested” in this pursuit. Over 56% are interested or very interested in living in a diverse neighborhood as an adult. A higher percentage (63%) indicated they are interested or very interested in working in a diverse setting. Somewhat Interested Interested Very Interested Speaking a foreign language. Living in a racially/ethnically diverse neighborhood when you are an adult Working in a racially/ethnically diverse setting when you are an adult. Not Interested Taking a foreign language class after high school. Taking a course focusing on other cultures after high school. Attending a racially/ethnically diverse college campus. Hadn't considered Table D.15 - Future Multicultural Interest Item Scores % Mean (SD) 7.9 18.2 26.0 24.2 23.7 3.38 (1.24) 1,239 9.1 22.1 30.6 22.7 15.6 3.14 (1.19) 1,234 9.3 9.8 19.8 31.7 29.4 3.62 (1.26) 1,230 4.6 11.2 18.8 27.2 38.3 3.83 (1.19) 1,236 10.9 8.9 24.0 29.4 26.8 3.52 (1.27) 1,235 10.0 7.2 19.9 32.4 30.6 3.66 (1.26) 1,234 D-12 n APPENDIX E: STUDENT ACHIEVEMENT ANALYSIS In this appendix we describe three methods we used for estimating the effects of magnet schools on the academic achievement of Grade 8 and 10 students. These methods include lotteries, traditional regression, and propensity score matching. While lottery methods are highly desirable for reasons described below, there are no lottery data available for some students and interdistrict magnet schools. This requires us to use alternative analysis techniques for those schools and students. By using all three methods and comparing results, we are able to describe the range, robustness, and size of interdistrict magnet schools effects. Three Estimation Methods Recent studies of school choice programs demonstrate how admission lotteries can be used to address unobserved variable bias resulting from self-selection. These studies measure treatment effects by comparing the average outcomes of lottery winners who enroll in a given school to the average outcomes of students who apply but are denied admission because they lost the lottery. Because lottery winners and losers are determined through a random process, we expect that if the lottery is large enough, the two groups will not differ significantly from each other on either observed or unobserved characteristics. Comparisons of average outcomes across the two groups will, then, be free of unobserved variable bias. Although an excellent strategy for addressing potential biases due to self-selection, lottery based analyses have important limitations. Perhaps most challenging is the fact that the lottery approach can only be applied in limited situations. In any given choice program, some schools either will not be oversubscribed or will not select students randomly, and those that are oversubscribed might not be representative of all schools. What’s more, admission lotteries are not typically held on a school wide basis. Rather admission lotteries are held for specific grades and often particular subgroups within grades. As a result, there are often too few winners and losers in particular lotteries to gain the benefits of randomization. Conclusions from lottery studies are often limited to subgroups of schools and types of students within schools. Alternative approaches to addressing bias due to self-selection use matching and/or statistical procedures to control for as many observable differences between treatment and comparison groups as possible. Regression analysis and propensity score matching are commonly used to achieve control for observable characteristics. The most convincing studies of this kind include pre-treatment test scores as control variables. Pre-treatment test scores can help control for many factors that influence student achievement and learning. When test scores are available from two or more pre-treatment periods, these methods can be used to determine if treatment group students make larger or smaller test score gains than students with similar pre-treatment levels of and rates of growth in achievement. Because they do not use random assignment, estimates from these studies remain subject to potential biases from any unobserved differences between treatment and comparison group members who have similar levels of pre-treatment performance. They often have the advantage, however, of being applicable to a wider range of schools and students than lottery based studies. E-1 Hoxby and Murarka suggest a way to combine the advantages of lottery-based analyses and comparison-with-controls analyses.31 Specifically, when lottery-based estimates of school effects are available for some students in some schools, these can be used to test the extent to which potential biases due to selection on unobservables are likely to influence estimates derived from propensity score or regression-based analyses. If the researcher is able to specify propensity score procedures or regression models that replicate the results of lottery-based procedures, then one can have more confidence that those same models and procedures applied to all students in the program provide defensible estimates. This strategy of leveraging the results from lottery analyses still requires untestable assumptions. One must assume that the factors that influence selection into magnet schools for which lottery results can be obtained are similar to the factors that influence selection into other magnet schools. In our case, we can obtain effects estimates using lottery results for two schools--one that serves grades six through eight and another that serves grades six through twelve. The broader set of school that we examine using only propensity score matching and regression analysis are also limited to those serving either middle or high school grades. In addition, all of the schools in our analysis, including the two schools for which we use lottery results and the broader set of schools that we examine, serve students primarily from Hartford, New Haven or Waterbury and their surrounding inner ring suburbs. Thus, the schools for whom we have lottery results and the broader set of schools we examine draw students from similar types of districts and the students in these schools have chosen from a similar set of options. There are, then, reasons to think that the factors the influence selection into the schools for which we have lottery results are similar to those that influence selection into the other schools we examine. Therefore, the estimates we derive from propensity score matching and/or regression analyses are strengthened if confirmed by lottery-based analyses. The analyses below provide that confirmation. Lottery Data and Methods As described in Section IV of this report, we worked together with the CSDE to match both magnet and non-magnet students with their prior tests scores. Table E.1 describes the overall sample of students we were able to obtain through those matching efforts. We then obtained the results of admission lotteries from six interdistrict magnet schools operated by the Capitol Region Education Council (CREC) – three elementary schools, two schools that begin in grade six, and a half-day high school. The analyses in this section focus on the two schools that begin in grade six. Our primary purpose is to determine if regression and propensity score methods that include pre-treatment measures of achievement can replicate estimates obtained from lottery-based analyses. Pre-treatment measures of achievement are not available for students in elementary school magnets, and thus we exclude the three elementary schools here. Also, we are unable to identify which students attend the half-day high school in the dataset we use for the regression and propensity score analyses, so that school is not included 31 Hoxby, C. M., & Murarka, S. (2008). Methods of assessing the achievement of students in charter schools In M. Berends, M.G. Springer, H.J. Walberg (Eds.), Charter School Outcomes (7-38). New York: Lawrence Earlbaum & Associates. E-2 either. Both of the remaining schools are located in a first ring suburb close to the city of Hartford and serve the city of Hartford and four suburban districts.32 Table E.1 - Treatment and Comparison Group Samples for Achievement Analysis 10th Grade Magnet Students 1083 15.4 (0.494) 10th Grade Students from Feeder Districts 8062 15.4 (0.508) 8th Grade Magnet Students 1060 13.4 (0.473) 8th Grade Students from Feeder Districts 14873 13.5 (0.554) Male 0.394 (0.489) 0.472 (0.499) 0.498 (0.500) 0.499 (0.500) Black 0.516 (0.500) 0.287 (0.452) 0.384 (0.487) 0.263 (0.440) Hispanic 0.267 (0.443) 0.246 (0.431) 0.256 (0.436) 0.274 (0.446) White 0.194 (0.396) 0.444 (0.497) 0.340 (0.474) 0.436 (0.496) Asian 0.020 (0.141) 0.021 (0.142) 0.019 (0.136) 0.026 (0.159) Free lunch eligible 0.591 (0.492) 0.466 (0.499) 0.408 (0.492) 0.445 (0.497) Special Education 0.065 (0.246) 0.115 (0.319) 0.058 (0.234) 0.110 (0.313) 8th Grade Math Score 234.0 (35.7) 232.1 (42.3) 8th Grade Reading Score 241.5 (41.8) 237.5 (47.7) 6th Grade Math Score 239.4 (37.1) 238.3 (43.5) 248.3 (41.4) 239.7 (48.0) 6th Grade Reading Score 235.8 (40.4) 234.5 (46.2) 245.1 (41.7) 233.1 (47.2) 4th Grade Math Score 246.1 (42.4) 237.6 (45.8) 4th Grade Reading Score 243.6 (40.6) 235.4 (43.9) N Age Means with standard deviations in parentheses. The admission policies for these two schools are identical. Each school allocates a predetermined number of seats for each of the districts it serves. Students apply in the spring of fifth grade for admission to 6th grade the following fall. When applications are received, 32 One of the schools serves four suburban districts west of the Connecticut River, and the other serves four districts east of the Connecticut River. Hartford is the only district served by both schools. E-3 siblings of students currently enrolled in the school are placed in the first seats allocated to their district. The remaining applicants are randomly assigned a number. Applicants from each district are then assigned to the remaining seats allocated to the district in order of the randomly assigned number. The students awarded seats through this process are contacted and offered admission, and the rest of the applicants from that district are placed on a waiting list in order of their randomly assigned number. When a student from a specific district turns down an admission offer, a seat in that district becomes available and is offered to the next applicant from that district on the waiting list. Applicants are only accepted for 6th grade. If students leave the school after the start of 6th grade those spots are filled with individuals from the original waiting list. For both schools, we have admissions data on applications submitted in 2003 and 2004. These data were matched to test score file records from 2001-02 through 2006-07 to provide measures of student achievement from the fall of 4th grade, the fall of 6th grade and the spring of 8th grade. These individual test score records were then matched over time. Thus, we have one post-treatment and two pre-treatment measures of student achievement.33 Information on the students’ age, gender, ethnicity, free-lunch status, and special education status is also available from the test score files. Since admissions lotteries are district and year specific, we have a total of 22 potential lotteries.34 In the analysis here, we drop applicants who did not participate in any of the lotteries because they had siblings enrolled in the school and students from eight potential lotteries which did not have any losers. All of the applicants in these lotteries were eventually offered a seat in the school, and thus, these lotteries do not contribute randomly assigned comparison group students. We also drop the remaining Hartford lotteries. All the applicants from Hartford to one of our schools were offered admission, so they did not participate in a true lottery. The two Hartford lotteries for the other school are also dropped for different reasons. First, attrition rates are high among participants in these lotteries—only 50% of these applicants have 8th grade tests scores, our outcome of interest. Second, unlike students from other districts, Hartford students have many different ways to opt out of the regular public schools—including other magnet schools, Open Choice, and charter schools. As a result, very few students who are lotteried out of the magnet school we are examining end up in Hartford public schools, which complicates interpretation of the magnet school effect we are trying to estimate. Random assignment helps to ensure that lottery winners are similar to lottery losers on both observed and unobserved characteristics. However, randomization alone does not guarantee that our treatment and comparison groups have no significant differences. First, a few of the lotteries in these schools are small. When lotteries are small, large differences between lottery winners and losers can emerge by chance. Second, we have substantial attrition from our samples. We 33 Prior to 2005-06, the Connecticut Mastery Tests (CMTs), which are part of Connecticut’s statewide testing program, were administered in the fall, early in the school year and only in grades 4, 6, and 8. So applicants in 2003 did not take statewide tests in seventh grade, and none of the applicants in our sample have fifth grade test scores. Beginning in 2005-06, tests were administered in the spring. All 8th grade test scores are from the spring of either 2005-06 or 2006-07. 34 Five district specific lotteries in both 2003 and 2004 for both schools implies 5x2x2=20 lotteries. However, for one of the districts served by one of these interdistrict magnets, seats are allocated by the middle school to which the student would be assigned, so there are two separate lotteries each year for that district. E-4 are missing records from the test score files for any student who participated in a magnet school lottery but whom we could not match to a test score record either because they attended a school outside the Hartford metropolitan region, enrolled in a private school, or otherwise could not be located in the test score file. We observe 8th grade test scores, our outcomes of interest, for 67.4% of the lottery participants in our sample. Attrition rates are similar for lottery participants offered admission and those not offered admission - we observe 8th grade test scores for 70.0% of those offered admission and 66.0% of those never offered admission. Nonetheless, if the lottery losers who attrit from our sample differ in systematic ways from the lottery winners who attrit, then there might be important differences between lottery winners and lottery losers. To test whether our lotteries produce balanced treatment and control groups, we examined differences between the non-attriting winners and losers of each lottery on a range of observable characteristics, including scores on math and reading tests administered at the beginning of the 6th and 4th grades. These tests revealed that in one of the smaller lotteries non-attriting winners had significantly lower test scores than non-attriting losers.35 This lottery was dropped from our final sample. The remaining lotteries produced groups of non-attriting winners and losers that are similar on observed characteristics. Similarity on observable characteristics does not guarantee that systematic differences in attrition did not result in unobserved differences between winners and losers in these lotteries, but we have no strong reason to suspect such differences. Our final sample includes applicants from 11 lotteries. For each lottery we can define those who are assigned low enough random numbers to be offered admission immediately, those who are not offered admission until other students have declined, and those who are never offered admission. We refer to these as on-time winners, delayed winners, and lotteried-out applicants, respectively. The 11 lotteries used in our analysis include a total of 866 applicants—229 on-time winners, 94 delayed winners, and 543 lotteried-out applicants. We can observe 8th grade test scores for 593 of these applicants, and among these, 173 are on-time winners, 53 are delayed winners, and 367 are lotteried-out. To demonstrate that lottery winners and losers are balanced on observable characteristics Table E.2 presents the results of a series of regressions. Each row in this table presents the regression of an observable characteristic on an indicator of whether or not the student won the lottery and a set of lottery dummy variables. In these regressions, on-time winners are counted as winners and delayed winners are excluded from the sample.36 In all of the regressions except the first one, the coefficients on the lottery winner indicator is not statistically distinguishable from zero, which indicates that except for age there are not statistically significant differences between lottery winners and lottery losers. Given that t-tests from 12 separate regressions are reported in Table E.2, it is not unreasonable to expect one significant result to emerge by chance. Most importantly, there are no significant differences between lottery winners and losers on pretreatment measures of achievement. 35 This lottery included 56 “winners” and only 11 students who were not offered admission. Of these 67 students, 25 are missing 6th grade test file records. For only five of these 25 attriters do we observe information from either the 4th or 8th grade test score files. Thus, we have no way to determine if this lottery produced an unbalanced set of winners or losers by chance, or whether differences between winners and losers result from differential attrition. 36 We also ran analogous regressions using all applicants that we observe in 8th grade and including on-time and delayed winners as lottery winners. The results of these regressions were quite similar to those reported in Table 1. E-5 Table E.2 - Comparison of Lotteried-in and Lotteried-Out Students On-Time Winner Constant Observations R-squared 0.080* (0.036) 13.837* (0.039) 540 0.031 Female 0.085 (0.047) 0.523* (0.051) 540 0.027 Black -0.015 (0.040) 0.417* (0.043) 540 0.218 Hispanic 0.020 (0.044) 0.359* (0.047) 540 0.153 White 0.021 (0.029) 0.176* (0.031) 540 0.052 Asian -0.026 (0.019) 0.046* (0.021) 540 0.013 Free lunch eligible -0.003 (0.038) 0.260* (0.041) 540 0.024 Special Education 0.027 (0.023) 0.031 (0.025) 540 0.041 6th Grade Math Score -3.142 (3.901) 244.866* (4.173) 496 0.086 6th Grade Reading Score -2.514 (4.016) 243.210* (4.299) 497 0.048 4th Grade Math Score -0.583 (4.260) 241.159* (4.621) 458 0.062 4th Grade Reading Score -0.611 (4.256) 241.030* (4.620) 458 0.068 Dependent Variable Age in Years Samples include students with either an 8th grade math or 8th grade reading test score. All regressions include a lottery fixed effect. Standard errors in parentheses. * Significant at 0.05 level. Estimates of the effects of these magnet schools on achievement can be derived from this sample of lottery participants using the following regression: (1) where is the 8th grade test score of student i who participates in lottery L; is an indicator of whether student i won an admission offer through the lottery, where this variable can be defined to include on-time winners only, or both on-time winners and delayed winners; E-6 represents lottery specific fixed effects; and is a random error term. can be estimated using a fixed effect or least squares dummy variable estimator. This coefficient is a weighted average of the difference in mean 8th grade test scores between the winners and losers of each lottery. If there are indeed no systematic differences between lottery winners and losers in each specific lottery, as random assignment helps to ensure, then the difference in mean 8th grade tests scores between the two groups is due solely to the lottery winners’ enrollment in the interdistrict magnets. However, not all lottery winners accept their invitation to enroll. The estimates of in equation (1) average together the effects of magnet schools on the achievement of those who choose to enroll and the presumably zero effect on those who do not enroll. Thus, as an estimate of the magnet effect, is biased toward zero. The estimates from this regression are sometimes referred to as the intention to treat effect.37 Hoxby and Murarkaargue that, unlike in the case of many medical treatments where subjects’ willingness and ability to comply with the treatment influences its efficacy, the intention to treat effect has little relevance for evaluating the effect of choice schools.38 Those who choose not to accept admission are not receiving the treatment in any meaningful sense. The standard solution to attenuation bias due to non-compliance uses the indicator of winning a lottery as an instrument for an indicator of magnet school enrollment in a two-stage least squares or instrumental variables procedure.39The first and second stage equations in such a procedure are: (2) where is an indicator that the student is enrolled in one of our two magnet schools during the 8th grade test administration, and is the predicted value of the magnet school indicator from the first stage equation. The estimate of from this procedure can be interpreted as the effect of the treatment on the treated, i.e., the effect of magnet schools on the students who attend them. There is some question about whether an indicator of on-time winning, with delayed winners excluded, or an indicator that includes on-time and delayed winners, is the most appropriate instrument for magnet school enrollment. Ballou argues that delayed winners who nonetheless accept an invitation to enroll may expect especially large gains from attending a magnet school. If so, an indicator that includes delayed winners might not be a valid instrument.40 We have used both definitions of lottery winners in the estimations presented here, and it turns out that our results are not sensitive to this issue. If lotteries are truly random, we would not expect any significant differences between lottery winners and losers, and the simple regressions above provide consistent estimates of the magnet 37 See Ballou, D. (2007). Magnet Schools and Peers: Effects on Student Achievement. Unpublished paper and Hoxby, C. M., & Rockoff, J. (2005). The Impact of Charter Schools on Student Achievement. Unpublished paper. 38 Hoxby & Murarka, Op. cit. 39 See Ballou, Op. cit. and Hoxby and Rockoff, Op. cit. 40 Ballou, Op. cit. E-7 school effect. Adding covariates to the above regressions is, nonetheless, desirable for two reasons. First, including covariates can significantly increase precision.41 Second, in any finite sample, we do not expect differences between randomly assigned treatment and control groups to equal zero. Adding covariates can help to control for differences between treatment and controls that arise by chance. Pre-treatment measures of achievement are especially useful for these purposes. Lottery Results Table E.3 presents estimated effects on 8th grade math and reading scores. The table includes: estimated effects of the intent to treat derived from regression (1); estimates of treatment on treated effects from (2) above; and estimates of the treatment on treated effects from versions of (2) that add individual level covariates including 6th grade math and reading test scores. The first three columns present results from regressions that define a lottery winner as an on-time winner. These regressions exclude delayed winners from the sample. The last three columns present the results from regressions that define lottery winners as including on-time and delayed winners. The results are very similar regardless of how we define lottery winners. As expected, the point estimates of the treatment on treated effect are larger than the estimates of the intent to treat effect and including covariates in the two-stage least squares estimates increases precision substantially. The estimated magnet school effects are larger when covariates are included, especially for math where estimates are more than twice as large and become statistically significant when covariates are added. The difference between the models with and without covariates might seem to suggest that randomization is not working. However, these results are not necessarily inconsistent with the results in Table E.2, which suggests that random assignment via the admission lotteries produced treatment and control groups with no significant differences. The estimated effects of the treatment on the treated obtained from the regressions without covariates are within the 95% confidence interval of the estimates obtained from the regressions that include covariates. Also, the results in Table E.2 suggest that although the differences between lottery winners and lottery losers in 6th grade test scores are not statistically significant, they are also not zero. The differences between the estimates from models with and without covariates are roughly equal to the differences in 6th grade test scores between lottery winners and lottery losers presented in Table E.2. Because the estimates that include covariates correct for these differences in pre-treatment test scores they are our preferred estimates. The results in Table E.3, then, indicate that these two interdistrict magnet schools have had positive effects on student achievement. The estimated effects on reading are larger than the estimated effects on math. To give an idea of the magnitude of these effects, the standard deviation in both 8th grade reading and math scores is approximately 45, and thus the estimates of the treatment on treated effect from the models that include covariates indicate that reading test scores of students in these magnet schools are between 0.26 and 0.29 standard deviations 41 See Ballou, op. cit. and also Betts, J., Rice, L., Zau, A., Tang, E. & Koedel, C. (2006). Does School Choice Work? Effects on Student Integration and Academic Achievement. Public Policy Institute of California. E-8 higher and their math scores are approximately 0.14 standard deviations higher than they would be if those students had attended other schools. Table E.3 - Lottery Based Estimates of the Effect of Interdistrict Magnet Schools on Achievement On-Time Lottery Winners On-Time+Delayed Lottery Winners Grade 8 Math N R-squared Intent to Treat 2.284 (3.445) 537 0.075 Treatment on Treated 2.735 (4.124) 537 0.076 Treatment-onTreated (w covariates) 6.242* (2.219) 492 0.770 Intent to Treat 1.837 (3.158) 590 0.068 Treatment on Treated 2.245 (3.859) 590 0.068 Treatment-onTreated (w covariates) 6.442* (2.049) 541 0.769 Grade 8 Reading 7.599* 9.075* 11.959* 7.722* 9.414* 12.961** (3.718) (4.444) (2.756) (3.430) (4.187) (2.563) N 538 538 493 591 591 542 R-squared 0.070 0.069 0.696 0.067 0.065 0.697 Each set of results are from separate regressions. Results in column labeled Intent to Treat are OLS regressions of test score on indicator of whether or not student won the admission lottery. Results in Treatment on Treated columns are IV estimates using indicator of students who won lottery as instrument for enrollment in interdistrict magnet school during 8th grade. Covariates included in models presented in columns 3 and 6 include student's age, gender, ethnicity, freelunch eligibility in grade 4, special education status in grade 4, grade 6 math and reading scores. In the first three columns only on-time lottery winners are counted as lottery winners and delayed winners are excluded from the sample. In the last three columns, delayed winners are included and counted as lottery winners. All regressions include lottery fixed effects. Standard errors in parentheses. * indicates statistically significant at 0.05 level. Regression Analysis In order to determine if comparison with controls can provide estimates similar to lottery methods, we assembled a dataset consisting of students who reside in the suburban districts served by the two interdistrict magnet schools used in the lottery based-analysis and who appear in the 2006 or 2007 8th grade test score files maintained by the state. Each of these student records were matched back to 6th and 4th grade test score records for the same student. Connecticut has only recently begun phasing in student identification numbers to facilitate the tracking of students across test score files from different years, so many of the longitudinal matches were made using name, date of birth, and other identifying information in the test score files. The state department of education was able to successfully match 85% of these student records to a 6th grade test score record and 75% to both a 6th grade and a 4th grade test score record. We begin by using these data to estimate the following regression model: (3) where is student i’s 8th grade test score; is an binary variable indicating whether or not the student was enrolled in one of the two magnet schools used in our lottery analysis at the time of the 8th grade test administration; is a vector of individual level covariates; is a year fixed E-9 effect; and is a random error term. Covariates include age, gender, ethnicity, special education status, free-lunch eligibility, and pre-treatment math and reading test scores. A model that includes 6th grade test scores controls for test score levels at the beginning of middle school, and a model that includes both 6th and 4th grade test scores controls for test score gains made between grade four and six as well as test score levels. Although it does not necessarily control for potential unobserved differences between treatment and comparison group students, regression analysis can be an effective method to control for observable characteristics. When pre-treatment measures of achievement are available, such controls for observables can provide plausible effect estimates. If the effects of observed covariates on the outcome of interest are not linear, however, a linear regression like that specified in (3) may not provide effective controls for differences between treatment and comparison group members. This problem can lead to large biases if treatment and comparison groups have very different covariate distributions.42 Also, if treatment effects vary across individuals, then regression returns a weighted average of the treatment effects on different individuals where the weights are designed to minimize variance in coefficient estimates. This weighted average need not correspond to the average effect of the treatment on the treated, which is usually the parameter of policy interest.43 Propensity Score Analysis Propensity score matching refers to a set of techniques for controlling for observed characteristics that avoids these potential problems with regression analyses. Propensity score methods begin by estimating a probability model to predict the likelihood that a student will select into a treatment. Here we estimate a logit model with enrollment in one of our two magnet schools in 8th grade as the dependent variable, and all the variables included as covariates in our regression analyses as independent variables. Estimates of this model can be used to compute a predicted likelihood of selecting one of our magnet schools, what is referred to as a propensity score. To ensure full conditioning on observable characteristics, specification of the probability model used to estimate the propensity score deserves careful attention. Dehejia and Wahba recommend an iterative procedure for determining a specification that balances the covariates across treatment and comparison group members.44 We use that procedure here. We estimated propensity scores using a model that, in addition to other covariates, includes 6th grade test scores only, and another that makes use of 4th grade test scores as well. The next step in propensity score matching is to limit the sample to the area of common support. In this case, we limit the set of potential comparison group students to those whose propensity scores are at least as high as the treatment group student with the lowest propensity score, and similarly limit our treatment group to students whose propensity score is at least as low as the 42 Stuart, E. (2007). Estimating Causal Effects Using School-Level Data Sets. Educational Researcher, 36, 187-198. Cobb-Clark, D. A., & Crossley, T. (2003). Econometrics for Evaluations: An Introduction to Recent Developments. The Economic Record, 79, 491-511. 44 Dehejia, R. H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and Statistics, 84(1), 151-161. 43 E-10 comparison group student with the highest propensity score. This step ensures that the treatment and comparison students used to estimate treatment effects have similar distributions on the covariates. Propensity scores can be used to compute estimates of treatment effects in a number of ways. Here we use two. The simplest approach is the nearest neighbor method, in which each student in our treatment group is matched to the one comparison group member with the closest propensity score, and the effect estimate is calculated as the difference in average 8th grade test scores between the treatment group and the matched comparisons.45 The second method uses a kernel density based estimator. In this procedure the 8th grade test score of each treatment group student is compared to a weighted average of all the comparison group members, where the weights are determined by a measure of the distance between the comparison group student’s propensity score and the treatment group member’s propensity score. The average of these differences across all treatment group members provides an estimate of the average effect of treatment on the treated. The nearest neighbor method ensures that treatment group members are compared only to the most similar comparisons available, and thereby minimize potential biases. In contrast the kernel density estimator makes maximal use of all the available information, and can substantially increase precision. Regression and Propensity Score Results Table E.4 presents the results of our comparison-with-controls estimates of the effect of these two interdistrict magnet schools. The first column presents results from OLS estimates of equation (3). The next two columns present estimates based on propensity score matching. The second column presents estimates based on nearest neighbor comparisons and the third column presents results from kernel density matching. Estimates in the upper panel are derived using only 6th grade test scores as pre-treatment achievement measures, and estimates in the lower panel are derived from models that include 4th grade test scores as well as 6th grade test scores. To facilitate comparison, we have also included lottery-based estimates of the average treatmenton-treated effect taken from the third and 6th columns of Table E.3. The results from the comparison-with-controls methods are similar whether test scores from two pre-treatment periods or only one pre-treatment period are used. Results are also similar across the three comparison-with-controls methods. In this case at least, regression based estimates do not appear to be subject to the potential problems advocates of propensity score matching emphasize. As expected, the kernel density matching provides much more precise estimates than nearest neighbor matching. As with the lottery estimates, the estimated effects of attending one of these magnet schools are positive and statistically significant for both math and reading, and the estimated effects on reading are approximately twice as large as the estimated effects on math. 45 Matching can be done with or without replacement. Matching without replacement maximizes the number of comparison group members selected, but can lead to large differences between particular treatment group members and their matched comparisons. Results of matching without replacement can also be sensitive to the order in which treatment units are matched. Here we use matching with replacement. E-11 Table E.4 - Regression, Propensity Score and Lottery Based-Estimates of the Average Effect of Treatment of Treated Using Only Grade 6 Test Score with Other Covariates Grade 8 Math Grade 8 Reading Grade 8 Math OLS 7.701* (3.553) Nearest Neighbor 10.074* (3.534) Kernel Density Matching 7.866* (1.597) Lottery Estimates 6.242* 6.442* (2.219) (2.049) 15.277* (0.921) 15.655* (3.166) 15.164* (1.458) 11.959* (2.756) 12.961** (2.563) Using Grade 4 and Grade 6 Test Scores with Other Covariates Kernel Nearest Density OLS Neighbor Matching Lottery Estimates 7.482* 6.959* 7.475* 6.242* 6.442* (3.440) (3.503) (1.587) (2.219) (2.049) Grade 8 Reading 15.090* 14.925* 14.991* 11.959* 12.961** (0.910) (3.413) (1.708) (2.756) (2.563) OLS regressions include age, gender, ethnicity, free-lunch eligibility, special education status, and year as well as pre-treatment test scores and magnet enrollment indicator. The same variables, plus some higher order terms and interactions are used to estimate propensity scores used in nearest neighbor and kernel density estimators. The figures in parentheses are standard errors. OLS standard errors are adjusted for clustering at the school level. Nearest neighbor and kernel density standard errors are bootstrapped. * indicates statistically significant at 0.05 level. For both math and reading, the point estimates from the comparison-with-controls methods are up to 25% larger than the estimates from the lottery-based analysis presented in the next to last column.46 The differences, however, are substantially small. For reading, the comparison-withcontrols methods imply an effect size of about 0.33 standard deviations compared to an effect size between 0.26 and 0.29 standard deviations implied by the lottery-based analysis. For math, the comparison-with-controls estimates in the lower panel of Table 4 imply an effect size of approximately 0.17 standard deviations compared with an effect size of about 0.14 standard deviations from the lottery analysis. Such small differences are unlikely to influence policy conclusions. Also, all of the comparison-with-controls estimates are comfortably within the 95% confidence interval for the corresponding lottery-based estimates, indicating that the differences between estimates from the comparison-in-control methods and the lottery-based analyses are not statistically significant. 46 Reading estimates from the comparison-with-controls methods are closer to 15 percent higher when compared with lottery-based estimates that include on-time and delayed winners, i.e. when compared to estimates in the last column. E-12