California Department of Education
Executive Office
SBE-003 (REV. 11/2017)
pptb-amard-july19item01

California State Board of Education
July 2019 Agenda
Item #01
Subject
Update on the Implementation of the Integrated Local, State, and Federal Accountability
and Continuous Improvement System: Recommended Action Regarding
Implementation of a Combined Four- and Five-Year Graduation Rate for the Graduation
Rate Indicator and the Application of the Three-by-Five Color Table to the
College/Career Indicator, and an Update on the Federal Waiver and Identification of
Schools Related to the English Learner Progress Indicator.

Type of Action
Action, Information

Summary of the Issue(s)
This item will present recommended changes to two state indicators—the Graduation
Rate Indicator and the College/Career Indicator (CCI)—beginning with the 2019
California School Dashboard (Dashboard) and provide an update on the federal waiver
and identification of schools related to the English Learner Progress Indicator (ELPI).
Staff will present an overview of the new Dashboard Mobile Application, new
College/Career Indicator and county-level reports, and discuss the additional website
language translations planned for the 2019–20 school year.

Recommendation
The California Department of Education (CDE) recommends that the State Board of
Education (SBE) approve: (1) a combined four- and five-year graduation rate for the
Graduation Rate Indicator; and (2) the application of the three-by-five color table to the
CCI.

 General Waiver
Page 2 of 3

Brief History of Key Issues
Combined Four- and Five-Year Graduation Rate
Currently, the Graduation Rate Indicator includes the four-year cohort graduation rate and the
one-year Dashboard Alternative Schools Status (DASS) graduation rate. It does not capture
the progress of students who take five years to graduate from high school. The SBE has
expressed an interest in using the five-year cohort graduation rate as part of the Dashboard,
as it could provide an opportunity for schools to demonstrate success with students who may
need additional time to earn a regular high school diploma (e.g., students with disabilities,
English learners [ELs], etc.).
The five-year graduation rate is intended to be a determination of the high school outcomes of
the preceding four-year cohort non-graduates in year five. As such, it compares the outcomes
of the same group of students one year later and offers schools and districts additional time
and incentive to work with and provide support to those students who did not graduate within
four years.
The Every Student Succeeds Act (ESSA) provides states the option to include a five-year
graduation rate in the accountability system. However, states are required to set a more
rigorous long-term goal for an extended-year adjusted cohort graduation rate (or five-year
rate), as compared to the long-term goal set for the four-year cohort graduation rate. Based
on a CDE review of the approved states plans for all 49 states and Puerto Rico, the U.S.
Department of Education (ED) approved plans that created a combined four- and five-year
graduation rate and used the same long-term goal. A summary of the review is provided in
Appendix A. Based on the ED’s prior approval of five-year graduation rates
(https://www2.ed.gov/admins/lead/account/stateplan17/statesubmission.html), using a
combined rate would allow California to take the five-year graduation rate into account without
having to create a separate long-term goal, add an additional indicator, and/or another fiveby-five colored table.
The CDE prepared several simulations, using graduation data from both the Class of 2018
(i.e., students graduating within four years) and the Class of 2017 (i.e., students graduating in
four years plus students who graduated in the five years). Distributions and analyses are
provided in Attachment 1.

 General Waiver
Page 3 of 3

Application of the Three-by-Five Color Table to the CCI
Application of the three-by-five color table results in a refiguring of the performance level
tables by removing two Change levels—Increased Significantly and Decreased Significantly—
and thus limiting extreme changes in small student populations. This methodology, which
limits large swings in the Change data that can be triggered by just a few students, has been
approved for three state indicators:


Graduation Rate Indicator: Applied if 149 or fewer students are in the graduating cohort



Suspension Rate Indicator: Applied if 149 or fewer students are cumulatively enrolled



Chronic Absenteeism Indicator: Applied if 149 or fewer students are enrolled

The CDE proposes that, beginning with the 2019 Dashboard, the three-by-five methodology
be applied to the CCI, which is based on the same cohort of students used in the calculations
for the Graduation Rate Indicator. The impact of this proposed change is shown in Attachment
2.

Every Student Succeeds Act Waiver and Identification of Schools Related
to the English Learner Progress Indicator
At the July SBE meeting, staff will update the SBE on two issues related the ELPI under the
ESSA.
Waiver
As described in more detail in an April 2019 information memorandum
(https://www.cde.ca.gov/be/pn/im/documents/memo-gad-lao-apr19item02.docx), the CDE and
the SBE received a final determination of disapproval letter from the ED related to California’s
request for a waiver, which had overwhelming support from California stakeholders, to allow
the inclusion of: (1) reclassified fluent English proficient (RFEP) students in the ELPI; and (2)
an additional weight in the ELPI calculation for long-term English Learners (LTEL) who
advanced at least one level on California’s English Proficiency assessment. This final
determination represents the final step in the waiver process, and there are not further
appeals available.
Identification of Schools
As described in more detail in a June 2019 information memorandum
(https://www.cde.ca.gov/be/pn/im/documents/memo-gacsb-gad-jun19item02.docx), the ED
sent a letter to the SBE and CDE with notification that they had designated California’s Title I,
Part A grant award as “high risk”. This is essentially a condition on the Title I, Part A grant for
the 2018-19 school year. It requires that California calculate the ELPI and use it for school
identification in 2019-20, which California had committed to doing before ED raised this issue.
The SBE and CDE sent a response letter dated June 13, 2019, arguing that this designation
is unnecessary as California provided the ED in December 2018 with a proposed Amendment

 General Waiver
Page 4 of 3
to the ESSA State Plan detailing our commitment to incorporate the ELPI into the
identification of CSI and ATSI schools for the 2019–20 school year, which is the specific
condition that ED noted as necessary to remove the high-risk determination on the Title I
grant.

Summary of Previous State Board of Education
Discussion and Action
Combined Four- and Five-Year Graduation Rate
At the March 2018 SBE meeting, the CDE provided an update on the development of the
work to incorporate the five-year graduation rate into the Dashboard
(https://www.cde.ca.gov/be/ag/ag/yr18/documents/mar18item01.docx).
In an August 2018 Information Memorandum, the CDE presented a set of recommended
business rules for calculating a five-year graduation rate.
(https://www.cde.ca.gov/be/pn/im/documents/memo-pptb-amard-aug18item02.docx).
At the March 2019 SBE meeting, the CDE outlined five options for incorporating the five-year
graduation rate into the 2019 Dashboard
(https://www.cde.ca.gov/be/ag/ag/yr19/documents/mar19item17.docx)

Application of the Three-by-Five Color Table to the CCI
In September 2017, the SBE approved an alternative methodology—known as the “Three-byFive” color grid—for assigning performance levels to local education agencies (LEAs) or
schools that serve small student populations
(https://www.cde.ca.gov/be/ag/ag/yr17/documents/sep17item02.doc).
In July 2018, the SBE approved that the methodology also be applied at the student group
level for the graduation rate and suspension rate indicators
(https://www.cde.ca.gov/be/ag/ag/yr18/documents/jul18item01.docx).
At the November 2018 meeting, the SBE approved applying this methodology to the Chronic
Absenteeism Indicator (reported for the first time in the 2018 Dashboard), when 149 or fewer
students are enrolled (https://www.cde.ca.gov/be/ag/ag/yr18/documents/nov18item04.docx).
At the March 2019 SBE meeting, the CDE proposed that, beginning with the 2019
Dashboard, the three-by-five methodology be applied to the CCI
(https://www.cde.ca.gov/be/ag/ag/yr19/documents/mar19item17.docx). This proposal was
made because the CCI is based on the same cohort of students used in the calculations for
the Graduation Rate Indicator, which already applies the three-by-five table to small student
populations. The proposal receives the support of the Technical Design Group.

 General Waiver
Page 5 of 3

Federal Waiver and Identification of Schools Related to the English
Learner Progress Indicator
Waiver
At the April 2018 SBE meeting, the SBE directed the CDE to request an ESSA waiver from
the ED regarding the inclusion of RFEPs and LTELs in the ELPI
(https://www.cde.ca.gov/be/ag/ag/yr18/documents/apr18item01.docx).
In an October 2018 Information Memorandum to the SBE, the CDE provided an update with a
detailed summary to the SBE on the ELPI waiver in the October 2018 Information
Memorandum (https://www.cde.ca.gov/be/pn/im/infomemooct2018.asp ). This memorandum
includes copies of the correspondence between the CDE and the ED through October 2018.
In an April 2019 Information Memorandum to the SBE, the CDE updated the SBE with the
ED’s final determination of disapproval letter
(https://www.cde.ca.gov/be/pn/im/documents/memo-gad-lao-apr19item02.docx). This
decision by the ED represents the final step in the waiver process under the ESSA.
Identification of Schools
A June 2019 Information Memorandum to the SBE provides information about the ED’s
designation of California’s Title I, Part A grant award as “high risk”.
(https://www.cde.ca.gov/be/pn/im/documents/memo-gacsb-gad-jun19item02.docx). The ED’s
rationale for this designation is that California did not include the Progress in the Achieving
English Language Proficiency Indicator (known as the ELPI) in its system of annual
meaningful differentiation pursuant to the ESSA. Included in the Memorandum are several
pieces of correspondence that contains both the history and substantial background on this
issue.

Fiscal Analysis (as appropriate)
The 2018–19 state budget funds the Proposition 98 Minimum Guarantee at $78.4 billion. This
reflects state funding of $54.9 billion and local funding of $23.5 billion, accounting for $11,631
in transitional kindergarten through grade twelve per-pupil funding. The budget package fully
funds the Local Control Funding Formula (LCFF) two years ahead of the estimated time
frame for implementation. Additionally, the 2018–19 state budget provided $50,000 to the San
Joaquin Office of Education to develop a Mobile Application for the Dashboard.

Attachment(s)
Attachment 1: Implementing a Combined Four-and Five-Year Graduation Rate for the
Graduation Rate Indicator (11 Pages)
Attachment 2: Applying the Three-by-Five Color Table to the College/Career Indicator
(4 Pages)

 General Waiver
Page 6 of 3
Attachment 3: Update on the Waiver and Identification of Schools Related to the English
Learner Progress Indicator (1 Page)
Attachment 4: California School Dashboard Educational Outreach Activities
(9 Pages)
Appendix A: Summary of States’ Graduation Rate Methods (3 Pages)
Appendix B Detailed Description for Three-by-Five Color Table for the College Career
Indicator—to Meet Section 508 of the Federal Rehabilitation Act of 1973 Requirements (1
Page)

 pptb-amard-july9item01
Attachment 1
Page 1 of 11

Attachment 1
Implementing a Combined Four- and Five-Year
Graduation Rate for the Graduation Rate
Indicator
Currently, the Graduation Rate Indicator includes the four-year cohort graduation rate
and the one-year Dashboard Alternative Schools Status (DASS) graduation rate. It does
not capture the progress of students who take five years to graduate from high school.
The State Board of Education (SBE) previously expressed an interest in using the fiveyear cohort graduation rate as part of the California School Dashboard (Dashboard), as
it could provide an opportunity for schools to demonstrate success with students who
may need additional time to earn a regular high school diploma (e.g., students with
disabilities [SWD], English learners, etc.). The addition of a five-year graduation rate
may also incentivize local educational agencies that currently do not offer students the
opportunity to remain at a comprehensive high school for an additional year to develop
programs that provide more time for students to earn a regular high school diploma
when necessary.
The Every Student Succeeds Act (ESSA) provides states the option to include a fiveyear graduation rate in the accountability system: however, states are required to set a
more rigorous long-term goal for an extended-year adjusted cohort graduation rate (or
five-year rate), as compared to the long-term goal set for the four-year cohort
graduation rate. Because no national reviews contained the detailed information needed
regarding the five-year graduation rate methodologies implemented by other states,
California Department of Education (CDE) staff reviewed the approved states plans for
all 49 states and Puerto Rico to obtain this information
(https://www2.ed.gov/admins/lead/account/stateplan17/statesubmission.html). See
Appendix A for a summary of the states’ graduation rate methods.
The review revealed that the U.S. Department of Education (ED) approved plans that
create a combined four- and five-year graduation rate (e.g., simple or weighted
average) and use the same long-term goal. Based on the ED’s prior approval of fiveyear graduation rates, using a combined rate would allow California to take the five-year
graduation rate into account without having to create a separate long-term goal, which
would require another set of Status cut scores along with a new five-by-five colored
table and the creation of an additional high school indicator.
Because schools with DASS have their Graduation Rate Indicator calculated using the
grade twelve one-year graduation rate, the five-year graduation rate will only be applied
to non-alternative schools.
The five-year graduation rate is intended to be a determination of the high school
outcomes of the preceding four-year cohort non-graduates in year five. As such, it

 pptb-amard-july9item01
Attachment 1
Page 2 of 11
compares the outcomes of the same group of students one year later and offers schools
and districts with additional time and incentive to work with and provide support to those
students who did not graduate within four years.
At the March 2018 SBE meeting, the CDE provided an update on the development of
the work to incorporate the five-year graduation rate into the Dashboard
(https://www.cde.ca.gov/be/ag/ag/yr18/documents/mar18item01.docx).
In an August 2018 Information Memorandum, the CDE presented a set of
recommended business rules for calculating a five-year graduation
rate(https://www.cde.ca.gov/be/pn/im/documents/memo-pptb-amardaug18item02.docx).
At the March 2019 SBE meeting, the CDE outlined the following five options for
incorporating the five-year graduation rate into the 2019 Dashboard
(https://www.cde.ca.gov/be/ag/ag/yr19/documents/mar19item17.docx):
1. Provide a performance level (color) for the four-year cohort graduation rate only
and continue to report the five-year graduation rate for informative data only.
2. Calculate a simple average for the four- and five-year cohort graduation rates. A
simple average provides the same weight to all four and five year graduates.
3. Calculate a weighted average for the four- and five-year cohort graduation rates.
This method provides more weight to students who graduate in four years (e.g.,
2/3) rather than five years (e.g., 1/3).
4. Report both the four- and five-year cohort graduation rates on the Dashboard
and assign the performance level (color) to the higher of the two rates. In order to
continue to use the most recent data available, two different cohorts would be
compared (the Class of 2019 four-year graduates and the Class of 2018 fouryear graduates plus five-year graduates).
5. Add the number of students who graduated in five years to the numerator and
denominator of the four-year cohort rate.
The CDE prepared several simulations, using graduation data from both the Class of
2018 (i.e., students graduating within four years) and the Class of 2017 (i.e., students
graduating in four years plus students who graduated in the five years). Option 1 was
ruled out because only reporting the data does not provide the same incentive for
schools to keep students for an additional year if needed to earn a standard diploma.
Options 2 and 3 (simple and weighted averages) were ruled out because the data
simulations revealed they produced no clear or consistent patterns of difference
between the four- and five-year rates for local educational agencies (LEAs), regardless
of whether the LEA had a low performance color (red or orange) or high performance
color (blue or green) for the 2018 four-year cohort rate: About 40 percent of LEAs had
negative differences in graduation rates, and about 60 percent had positive differences

 pptb-amard-july9item01
Attachment 1
Page 3 of 11
in graduation rates when the two rates were averaged. Moreover, both the simple and
weighted averages resulted in lower graduation rates for a significant number of
schools, simply due to the fact of combining two different groups of students from two
graduating classes. Option 4 was ruled out because assigning colors to the higher
graduation rate (four-year vs. five-year) results in some LEAs and schools with very low
four-year graduation rates to receive substantially higher performance levels based on
the five-year rates. For example, an LEA with a four-year graduation rate of 57.7
percent and a five-year graduation rate of 97.4 percent would move from a very low
status to a very high status.
At its August 2018 meeting, the Technical Design Group (TDG) recommended that the
five-year graduation rate be incorporated into the performance determination for the
Graduation Rate Indicator. One option of interest was to use the four-year cohort
graduation rate as the base, but provide additional credit for any five-year students from
the previous year’s cohort. An example of this option is provided below:
At Emerald High School, there are 100 students in the Class of 2018 (the four-year
graduation cohort). Of these 100 students, 95 graduated within four years (e.g., by
spring 2018). In addition, five students from the previous year’s four-year graduation
cohort (Class of 2017) graduated at the end of 2017–18 school year. These five
students are therefore counted in the five-year graduation rate.
To produce a combined graduation rate for the four- and five-year graduates for
2018, we first add the number of graduates in both cohorts: 95 + 5 = 100.
This number is the numerator of the combined rate.
The denominator is the sum of the 2018 four-year graduation cohort and the
additional five-year graduates from the previous year’s cohort: 100 + 5 = 105.
The combined rate is 95.2 percent (100/105).
At its February 2019 meeting, the TDG requested that the CDE conduct data analyses
to allow for comparisons between:


A combined graduation rate: Students who graduated from the Class of 2018 in
four years, plus students from the Class of 2017 who graduated in five years.



The four-year graduation rate (i.e., Graduates from the Class of 2018).

School-level distributions for the combined and four-year graduation rates are presented
in Table 1, below. The impact of the combined rate is more pronounced in the lower
percentiles for schools.

Table 1
School Level Analyses for Graduation Rates

 pptb-amard-july9item01
Attachment 1
Page 4 of 11
The Combined Graduation Rate vs. The 2017–18 Four-Year Cohort Rate
Percentil
e

Combined Rate

Four-year Rate

Difference Between the Two
Rates

5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95

64.6
76.3
82.1
85.1
87.4
89.0
90.2
91.4
92.6
93.4
94.0
94.6
95.3
95.9
96.5
97.0
97.6
98.1
99.0

61.9
74.3
81.5
84.8
87.1
88.8
90.1
91.3
92.6
93.4
94.0
94.6
95.2
95.9
96.4
97.0
97.5
98.1
99.0

2.7
2.0
0.6
0.3
0.3
0.1
0.1
0.0
0.1
0.1
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Note: School graduation rate distribution analysis includes only non-DASS (traditional
and charter) schools. There are a total of 1,418 schools with 30 or more students.

 pptb-amard-july9item01
Attachment 1
Page 5 of 11
Comparisons between charter and non-charter schools are presented in Table 2. In the lower percentiles, charter schools benefit
more from the combined rate than non-charter schools.
Table 2
Charter Schools vs. Non-Charter Schools and the Combined Graduation Rate vs. the 2017–18 Four-Year Cohort Rate
School Level Analyses
Percentile

Non-Charter
Combined
Rate

5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95

71.7
80.5
84.7
87.1
88.6
90.1
91.3
92.5
93.3
93.9
94.4
94.9
95.5
96.0
96.5
97.0
97.4
98.0
98.8

Non-Charter
Non-Charter
Four-year Rate Difference Between
the Two Rates
70.5
80.4
84.4
87.0
88.5
90.0
91.3
92.5
93.3
93.8
94.3
94.9
95.5
96.0
96.5
96.9
97.4
98.0
98.8

1.2
0.1
0.3
0.1
0.1
0.1
0.1
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Charter
Combined
Rate

Charter
Four-year Rate

47.2
61.3
68.6
75.8
79.2
82.9
85.0
87.3
89.3
90.1
91.2
92.6
93.7
95.0
96.2
97.3
98.1
98.7
100.0

42.6
58.7
66.2
74.1
78.3
82.5
84.0
86.9
89.0
89.8
91.0
92.6
93.6
94.9
96.2
97.3
98.1
98.7
100.0

Charter
Difference
Between the
Two Rates
4.6
2.6
2.4
1.7
0.9
0.4
1.0
0.4
0.3
0.2
0.2
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0

Note: The total number of schools with 30 or more students: charter schools = 312 and non-charter schools = 1,106.

 pptb-amard-july9item01
Attachment 1
Page 6 of 11
Comparisons between small and large schools are presented in Table 3. Small schools see larger increases in graduation rates
in the lower percentiles with the combined rate. This result is expected due to the impact that only a few students can have on
smaller n-sizes.
Table 3
School Comparison
Small vs. Large and the Combined Graduation Rate vs. the 2017–18 Four-Year Cohort Graduation Rate
Percentile
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95

Large
Combined
Rate
79.9
84.4
87.0
88.4
89.6
90.8
92.0
92.9
93.5
94.0
94.5
95.0
95.5
96.1
96.4
96.9
97.4
97.9
98.7

Large
Four-Year
Rate
79.3
84.0
86.8
88.2
89.5
90.7
91.9
92.8
93.4
94.0
94.5
95.0
95.5
96.0
96.4
96.9
97.4
97.9
98.6

Large
Difference Between
the Two Rates
0.6
0.4
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Small
Combined
Rate:
53.1
62.5
70.7
75.6
79.2
82.8
85.3
87.8
89.7
91.1
92.5
93.8
94.4
95.6
96.5
97.2
98.0
98.7
100.0

Small
Four-Year
Rate
47.4
61.0
68.4
73.7
78.0
82.5
84.7
87.5
89.5
90.9
92.5
93.7
94.4
95.5
96.5
97.2
98.0
98.7
100.0

Small
Difference
Between the Two Rates
5.7
1.5
2.3
1.9
1.2
0.3
0.6
0.3
0.1
0.2
0.0
0.1
0.1
0.1
0.0
0.0
0.0
0.0
0.0

Note for Table 3: Large schools are defined as having 150 or more students. The total number of large schools = 876. Small
schools are defined as having less than 150 students. The total number of small school = 542. The four-year graduation rate is
based on the 2017–18 graduation cohort.

 pptb-amard-july9item01
Attachment 1
Page 7 of 11
District-level distributions for the combined and four-year graduation rates are presented in Table 4. The impact of the combined
rate is more pronounced in the lower percentiles.
Table 4
District Level Graduation Rates Analyses: The Combined Graduation Rate vs. the 2017–18 Four-Year Cohort Rate
Percentil
e
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95

Combined Rate

Four-Year Rate

Difference Between the Two Rates

62.0
76.2
81.3
84.6
86.3
88.1
89.4
90.2
90.9
91.6
92.5
93.3
94.1
94.7
95.5
96.1
96.9
97.8
98.9

59.4
74.6
81.1
83.6
86.0
87.9
89.3
90.1
90.8
91.6
92.4
93.2
94.1
94.7
95.4
96.1
96.8
97.8
98.9

2.7
1.6
0.3
1.0
0.3
0.2
0.1
0.1
0.1
0.0
0.1
0.1
0.0
0.0
0.1
0.0
0.0
0.0
0.0

Note: LEA graduation rate distribution analysis includes districts and non-DASS charter schools (i.e., counties are excluded).
Comparisons between large and small districts are presented in Table 5. Small districts see larger increases in graduation rates
in the lower percentiles with the combined rate, due to the impact that only a few students can make on smaller n-sizes.
Table 5

 pptb-amard-july9item01
Attachment 1
Page 8 of 11
District Comparison: Large vs. Small and the Combined Graduation Rate vs. the 2017–18 Four-Year Cohort Rate
Percentile
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95

Large
Combined
Rate
78.3
83.8
85.9
87.2
88.6
89.6
90.4
90.9
91.4
91.8
92.4
93.2
94.0
94.4
95.1
95.6
96.1
96.9
97.6

Large
Four-Year
Rate
77.1
83.5
85.6
86.9
88.5
89.3
90.4
90.8
91.3
91.8
92.4
93.1
93.9
94.4
95.1
95.6
96.1
96.8
97.6

Large
Difference Between
the Two Rates
1.2
0.3
0.3
0.4
0.1
0.2
0.1
0.1
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Small
Combined
Rate
53.2
66.9
75.1
79.4
82.6
84.8
86.9
89.0
89.9
91.0
92.6
93.4
94.3
95.2
96.1
97.2
98.0
98.6
100.0

Small
Four-Year
Rate
51.0
64.0
73.7
78.7
81.9
83.7
86.6
88.8
89.8
90.8
92.4
93.4
94.3
95.1
96.1
97.2
98.0
98.6
100.0

Small
Difference Between
the Two Rates
2.2
2.9
1.4
0.6
0.6
1.0
0.4
0.2
0.1
0.2
0.1
0.0
0.0
0.1
0.0
0.1
0.0
0.0
0.0

Large LEAs/charters are defined as having 150 or more students. Large LEAs/charters = 358. Small LEAs/charters are defined
as having less than 150 students. Small LEAs/charters = 340. The four-year graduation rate is based on the 2017–18 graduation
cohort.
Finally, statewide and schoolwide comparisons by student group are presented in Tables 6 and 7, respectively.
Table 6
State Comparison: Student Groups
The Combined Graduation Rate vs. the 2017–18 Four-Year Cohort Rate

 Student Group

pptb-amard-july9item01
Attachment 1
Page 9 of 11
Difference Between the
Two Rates

2017–18 Graduation Rate
(Four-Year plus DASS Rate)

All Students

Combined Graduation Rate
(Four-Year plus DASS Rate
plus Five-Year)
83.7

83.5

0.2

African American

72.4

72.1

0.4

English Learner

71.7

70.9

0.8

Socioeconomically
Disadvantaged
Foster Youth

72.2

71.4

0.8

60.0

59.0

1.0

 pptb-amard-july9item01
Attachment 1
Page 10 of 11
Table 7
School Comparison: Student Groups
The Combined Graduation Rate vs. the 2017–18 Four-Year Cohort Graduation Rate
Percentile

All
Students
Combined
Rate

EL
Combined
Rate

SWD
Combined
Rate

5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95

64.6
76.3
82.1
85.1
87.4
89.0
90.2
91.4
92.6
93.4
94.0
94.6
95.3
95.9
96.5
97.0
97.6
98.1
99.0

55.8
62.3
68.8
71.1
73.2
75.4
77.5
79.2
81.0
82.5
84.0
85.3
87.0
88.5
89.8
90.9
92.6
93.8
95.8

56.1
60.5
64.6
67.0
69.5
71.2
72.7
74.5
76.5
78.2
79.7
81.3
83.0
84.8
86.7
88.2
89.5
91.4
94.6

All
Students
FourYear
Rate
61.9
74.3
81.5
84.8
87.1
88.8
90.1
91.3
92.6
93.4
94.0
94.6
95.2
95.9
96.4
97.0
97.5
98.1
99.0

El
FourYear
Rate
52.8
61.0
66.9
69.7
72.4
74.5
77.0
78.7
80.3
82.0
83.7
85.1
86.5
88.2
89.6
90.8
92.5
93.8
95.8

SWD
FourYear
Rate
53.1
59.5
63.0
66.7
68.4
70.0
72.2
73.8
75.7
77.4
79.4
81.1
82.6
84.4
86.4
88.0
89.2
91.4
94.5

All Students
Difference
Between the
Two Rates
2.7
2.0
0.6
0.3
0.3
0.1
0.1
0.0
0.1
0.1
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0

EL
Difference
Between
the Two
Rates
3.0
1.3
1.8
1.4
0.8
0.9
0.5
0.5
0.7
0.5
0.3
0.2
0.5
0.2
0.2
0.1
0.2
0.0
0.0

SWD
Difference
Between
the Two
Rates
3.0
0.9
1.5
0.3
1.1
1.2
0.5
0.6
0.8
0.8
0.3
0.2
0.4
0.5
0.2
0.2
0.3
0.0
0.1

 pptb-amard-july9item01
Attachment 1
Page 11 of 11
At the April 2019 meeting, the TDG examined the differences between the combined rate and
four-year graduation rates and recommended that the combined rate—which represents all
four and five year graduates—be used to calculate the Graduation Rate Indicator in the 2019
Dashboard. The TDG also recommended that the CDE provide a breakdown of the four- and
five-year graduation rates, as well as the Grade 12 DASS graduation rate, in the 2019
Dashboard Detailed Reports.
The CDE presented the combined rate to the California Practitioners Advisory Group (CPAG)
at its meeting on June 10, 2019. Most members agreed that the combined rate was the best
option for incorporating the five-year graduation rate into the Graduation Rate Indicator. In
addition, members requested that the four- and five-year graduation rates be reported
separately as well, for informational purposes.

 pptb-amard-july19item01
Attachment 2
Page 1 of 4

Attachment 2
Applying the Three-by-Five Color Table to the
College/Career Indicator
In September 2017, the State Board of Education (SBE) approved an alternative
methodology—known as the “three-by-five” color table—for assigning performance
levels to local education agencies (LEAs) or schools that serve small student
populations (https://www.cde.ca.gov/be/ag/ag/yr17/documents/sep17item02.doc).
Application of the three-by-five color table results in a refiguring of the performance level
tables by removing two Change levels—Increased Significantly and Decreased
Significantly—and thus limiting extreme changes in small student populations. This
methodology, which limits large swings in the Change data that can be triggered by just
a few students, was originally approved for two state indicators:


Graduation Rate Indicator: Applied if 149 or fewer students are in the graduating
cohort



Suspension Rate Indicator: Applied if 149 of fewer are cumulatively enrolled

In July 2018, the SBE approved that the methodology also be applied at the student
group level for these two indicators
(https://www.cde.ca.gov/be/ag/ag/yr18/documents/jul18item01.docx).
At its November 2018 meeting, the SBE approved applying this methodology to the
Chronic Absenteeism Indicator (reported for the first time in the 2018 Dashboard), when
149 or fewer students are enrolled
(https://www.cde.ca.gov/be/ag/ag/yr18/documents/nov18item04.docx).
At the March 2019 SBE meeting, the California Department of Education (CDE)
proposed that, beginning with the 2019 Dashboard, the three-by-five methodology be
applied to the College/Career Indicator (CCI)
(https://www.cde.ca.gov/be/ag/ag/yr19/documents/mar19item17.docx). This proposal
was made because the CCI is based on the same cohort of students used in the
calculations for the Graduation Rate Indicator, which already applies the three-by-five
table to small student populations. The proposal receives the support of the Technical
Design Group.
Figure 1 illustrates the three-by-five colored table for the CCI. As referenced above, the
two Change columns “Declined Significantly” and “Increased Significantly” are removed,
resulting in a table that has three Change columns (i.e., Declined, Maintained, and
Increased) and five Status rows (i.e., Very High, High, Medium, Low, and Very Low).
There are still five possible performance levels (colors)

 pptb-amard-july19item01
Attachment 2
Page 2 of 4
that LEAs, schools, and student groups can receive if their graduating cohort size is
between 30 to 149 students.
Figure 1: Three-by-Five Color Table for the CCI

The impact of applying the three-by-five color table at the LEA level is shown in Table 1.
Table 1
LEA Color Change Direction
Change Direction
Lower Performance
No Change
Higher Performance
Total

Number and Percent of LEAs
4 (1.0%)
406 (98.0%)
4 (1.0%)
414 (100%)

The impact is minimal. Only eight LEAs (2 percent) have a change in direction as a
result of applying the three-by-five color table to the CCI.

 pptb-amard-july19item01
Attachment 2
Page 3 of 4
Table 2 shows the impact of applying the three-by-five color table at the school level.
Table 2
School Color Change Direction
Change Direction
Lower Performance
No Change
Higher Performance
Total

Number and Percent of Schools
27 (1.5%)
1,722 (96.4%)
38 (2.1%)
1,787 (100%)

Only 65 schools (3.6 percent) have a change in direction as a result of applying the
three-by-five color table to the CCI.
The impact is greater at the student group level, as shown in Table 3.
Table 3
Student Group Change Direction
Change Direction
Lower Performance
No Change
Higher Performance
Total

Number and
Groups
172 (2.0%)
7,930 (93.9%)
345 (4.1%)
8,447 (100%)

Percent

of

Student

Six point one percent of student groups had a color change with the application of the
three-by-five table. Three hundred forty-five student groups received a higher
performance color: Red to Orange, Orange to Yellow, or Yellow to Green. One hundred
seventy-two student groups received a lower performance color: Yellow to Orange.

 pptb-amard-july19item01
Attachment 2
Page 4 of 4
Table 4 shows that applying the three-by-five color table has the most significant impact
for Hispanic, homeless, and socioeconomically disadvantaged students.
Table 4
Color Change by Student Group
Student Group
African American
American Indian
Asian
EL
Filipino
Foster
Hispanic
Homeless
Multiple Race
Pacific Islander
Socioeconomicall
y Disadvantaged
Students with
Disabilities
White

Lower
Performance
4 (1.1%)
0 (0.0%)
10 (2.2%)
2 (0.2%)
9 (4.5%)
0 (0.0%)
38 (2.2%)
7 (1.6%)
13 (8.0%)
0 (0.0%)

No Change
349 (91.8%)
7 (87.5%)
429 (92.3%)
883 (96.9%)
173 (86.1%)
76 (97.4%)
1,665 (94.7%)
367 (85.0%)
134 (82.2%)
10 (90.9%)

Higher
Performance
27 (7.1%)
1 (12.5%)
26 (5.6%)
26 (2.9%)
19 (9.5%)
2 (2.6%)
56 (3.2%)
58 (13.4%)
16 (9.8%)
1 (9.1%)

380
8
465
911
201
78
1,759
432
163
11

43 (2.2%)

1,891 (94.5%)

68 (3.4%)

2,002

4 (0.4%)

909 (98.0%)

15 (1.6%)

928

42 (3.8%)

1,037 (93.5%)

30 (2.7%)

1,109

The CDE shared this data at the CPAG meeting on June 10, 2019. Members agreed
that the three-by-five color table be applied to the CCI.

Total

 pptb-amard-july19item01
Attachment 3
Page 1 of 2

Attachment 3
Update on the Waiver and Idenfitication of
Schools Related to the English Learner Progress
Indicator
Waiver
On April 15, 2019, the California Department of Education (CDE) and the California
State Board of Education (SBE) received the final determination of disapproval letter
from the U.S. Department of Education (ED) on the inclusion Reclassified Fluent
English Proficient (RFEP) and Long Term English Learner (LTEL) students into the
English Learner Progress Indicator (ELPI)
(https://www.cde.ca.gov/be/pn/im/documents/memo-gad-lao-apr19item02.docx). This
decision by the ED represents the final step in the waiver process under the Every
Student Succeeds Acts (ESSA).
After convening with the ELPI Workgroup on May 20, 2019, members agreed that the
CDE had exhausted all available options in an attempt to include RFEP and LTEL
students into the final ELPI and should consider the matter settled.
One recommendation at the ELPI Workgroup meeting was to take into consideration the
performance of RFEP and LTEL students in the determination for the California
Distinguished Schools Program or other school award programs. The CDE will continue
to explore this as an option with the ELPI Workgroup moving forward. The CDE meets
with the ELPI Workgroup twice this fall to further discuss, among other items, this
option.
Identification of Schools
On June 3, 2019, the State Board of Education (SBE) and California Department of
Education (CDE), collectively California, received notification that the U.S. Department
of Education (ED) had designated California’s Title I, Part A grant award as “high risk”.
In response to ED’s letter, the SBE and CDE on June 13, 2019, stated that this
designation is unnecessary as California provided the ED in December 2018 with a
proposed Amendment to California’s ESSA State Plan detailing a commitment to
incorporate the ELPI into the identification of CSI and Additional Targeted Support and
Improvement (ATSI) schools for the 2019–20 school year, which is the specific condition
that ED noted as necessary to remove the high-risk determination on the Title I grant.
The CDE and SBE also highlighted California’s plan to incorporate ELPI in the annual
meaningful differentiation of schools as soon as the second year of data is available
which will be the 2019 Dashboard.

 pptb-amard-july19item01
Attachment 3
Page 2 of 2
Additionally, California has committed to, and proposed amendments to the State Plan
to reflect, a second consecutive year of CSI and ATSI identification for 2019–20 rather
than the every-three-year timeline as required by ESSA and reflected in California’s
currently approved State Plan. California proactively modified its timeline for identifying
schools for CSI and ATSI to include a consecutive identification in the 2019–20 school
year (based on the 2018 and 2019 Dashboard), before transitioning to the every-threeyear cycle, to ensure that the ELPI Status is included in the designation of schools at
the earliest practical time given California’s transition to the English Language
Proficiency Assessments for California.

 pptb-amard-july19item01
Attachment 4
Page 1 of 9

Attachment 4
California School Dashboard Educational Outreach Activities
Table 1. California Department of Education Policy Work Group Meetings
Date

April 10,
2019

Title

Growth Model Stakeholder
Session 2

Estimated
Number of
Attendees

18

Topics







April 11,
2019

Alternative Schools Task Force
Local Indicator Subcommittee

7


April 16,
2019

Technical Design Group (TDG)

7





Reviewed calendar and timeline
Lessons learned from last meeting
Gallery walk through interpretations and voting
Analyzed interpretations
Conducted rank-choice and rating scale voting
for interpretations
Discussed possible measures for
consideration in a positive transition local
indicator for the California School Dashboard
(Dashboard)
Inclusion of the California Alternate
Assessments in Distance from Standard
calculations for the Academic Indicator
English Learner Progress Indicator (ELPI)
Inclusion of the Five-Year Graduation Rate in
the Dashboard
Review data on the use of the District of
Residence Rule for the Indicator

 pptb-amard-july19item01
Attachment 4
Page 2 of 9
Date

Title

Estimated
Number of
Attendees

Topics


May 20, 2019

English Learner Progress Indicator
(ELPI) Workgroup


12



May 24, 2019

Alternative Schools Task Force
Local Indicator Subcommittee

12





May 29, 2019

Growth Model Stakeholder
Session 3

15





Every Student Succeeds Act (ESSA) Waiver
for the inclusion of reclassified fluent English
proficient and long-term English learners
The English Language Proficiency
Assessments for California (ELPAC)
summative assessment overview
ELPI Status in determining school and local
education agency (LEA) eligibility for
assistance
Potential modifications to the Academic
Indicator
Definitions of College/Career Indicator (CCI)
measures for collection in the 2018–19 and
2019–20 school years
Review of how selected other states present
growth model information
In-depth review of theories of action
Full group discussion of theories of action
Discussion of how to have conversations with
community and colleagues about the work of
the Growth Model Stakeholder Group

 pptb-amard-july19item01
Attachment 4
Page 3 of 9
Table 2. ln-person Meetings/Conferences

Date
March 22,
2019

CABE

March
28,
Date
2019

Council of Chief State School
Officers: Trends in State
Title
Accountability Systems—
Measuring and Using Chronic
Absenteeism Data

March 28,
29, 2019
February
2019
March 6, 2019

April 3, 2019

March 21, 2019
April 3, 2019

March 21, 2019
April 4, 2019
March 21, 2019
Date

April 8, 2019

April 9, 2019

Estimated Number of
Attendees

Title

70


Estimated
80
Number
of
Attendees

20 80

SystemCoordinators
of Support Network
Bilingual
Geographic
Leads
(BCN)
San Bernardino County
25
Superintendent of Schools
Counselor Conference:
Promoting
Equity
State Special
Schools:
9
California Presentation
School Dashboard
Dashboard
Overview and Deeper Dive into
the Chronic Absenteeism
Indicator
Spring Association
Foster YouthofServices
California
17
Coordinating
Statewide
50
School Administrator (ACSA)
Meeting
High School
Education Council
“Partnering with
K-12 on
Meeting
Data” for the California
California Association of
40
Community College
25
Bilingual Educators (CABE)
Association for Occupational
Education
(CCCAOE)
Linked Learning
Convention:
16
Making the AND Count in Estimated
Title
Number of
California’s
College and Career
Attendees
Indicator
Association of California
School Administrators
Legislative Action Day:
80
Accountability and System of
Support Update

Professional Development for
County Offices of Education
Eligible for Differentiated


















68

pptb-amard-july19item01
Overview of all EL data on the Dashboard
Attachment 4
Provided live demonstration of new featuresPage
of the
4 of 9
Dashboard, including the Spanish translation of
each page
Overview of how chronic
absenteeism is used in
Topics
California’s accountability system
Geographic Information System Mapping with the
 Update
on the
(ELPI)
state
indicators
(CCI
and Suspension): district- and
 Data update
on English
school-level
indicator
data learners (EL)
 Presented
features of the Dashboard,
Overview
of thenew
Dashboard
including
theIndicators
increased focus on equity and the
Review
of State
additional
reports
and resources available to
District of Residence Rule
schools
and
districts
on the California
Live Demonstration
Department of Education’s website.
Performance for California State Special Schools
 Provided an in-depth review of the Chronic
on State Indicators
Absenteeism Indicator, including how
Dashboard Resources
suspensions impact chronic absenteeism
Provided
updates
useofofthe
Foster
data on
 New Look
andon
Feel
2018Youth
Dashboard
the
Dashboard
and
DataQuest
 New resources for the Dashboard
Responded to technical questions and answers
 March 2019 State Board of Education revisions
The Dashboard and the CCI
under consideration for the 2019 Dashboard
 Leveraging existing EL data for school and
district planning purposes







Topics

Overview of current measures used in the CCI
New CCI measures planned for collection in
Topics
future years

Update on the work planned for the 2019
Dashboard and ongoing development of the
Dashboard
Overview of the system of support
Overview of the Dashboard
Review of Criteria Used to Identify LEAs for Support
under Local Control Funding Formula (LCFF)
County offices of education (COEs) and Dashboard

 Date

Title

Estimated
Number of
Attendees
14

May 15, 2019

Regional Assessment
Network (RAN)

May 15, 2019

California County
Superintendents Educational
Services Association
(CCSESA) CTE Meeting
CCSESA Curriculum and
Instructional Services
Accountability Sub-committee
Monterey County Office of
Education

40

State and Federal Programs
Directors Meeting

90

May 15, 2019

May 16, 2019

May 17, 2019

15

7

Topics
pptb-amard-july19item01
Application of the 3 X 5 Color Grid for the CCI
Attachment 4
Revisions to the 5 X 5 Reports
Page 5 of 9
Addition of a County List of Districts
Five-Year Graduation Rate
Inclusion of the California Alternate Assessment
(CAA) in the Academic Indicator
 Update on the Growth Model Work
 Definitions of career CCI measures to be collected
in CALPADS in the 2018-19 and 2019-20 school
years




















Date

Title

June 2, 2019

Bilingual Coordinators
Network
California Education Policy
Fellowship Program:
Using Data at the System
Level
California Practitioners
Advisory Group

June 7, 2019

June 10, 2019

Estimated
Number of
Attendees
90
20

15

Update on May 2019 State Board of Education
action related to accountability
Overview of 2019 Dashboard Workplan
Overview of the Dashboard
Deep Dive into State Indicators
Academic Indicator
Suspension Indicator
Chronic Absenteeism Indicator
DASS Graduation Rate
College/Career Indicator
English Learner Progress Indicator
Demographic Data for Student Groups
Update on the incorporation of the CAA in the
academic indicator
Demonstration of the new accountability reports
available
Topics








ELPI
EL Data on DataQuest
The policy framework for accountability
How local and state accountability improve
education
The limitations of the accountability system
Incorporating a Five-Year Graduation Rate into the
Graduation Rate Indicator

 pptb-amard-july19item01
Attachment 4
Page 6 of 9
Table 3. Webinars
Date

February 28,
2019

March 11,
2019
April 9, 2019

April 19, 2019

Title
California
Advisory Task
Force on
Alternative
Schools
Growth Model
Stakeholder
Webinar
CALPADS
Information
Meeting (CIM)

Every Student
Succeeds Act
Stakeholder
Meeting

Estimated
Number of
Attendees

Topics


14

8
940

120






Reviewed information provided at the February 4, 2019
stakeholder meeting for stakeholder that were unable to attend.




CDE Update
CALPADS Support Update





Federal Updates
Promoting Equitable Access to Teachers Program
The Local Control and Accountability Plan (LCAP) as the School
Plan
Wester Association of Schools and Colleges (WASC) and
schools eligible for Comprehensive Support and Improvement
Submission of the LCAP Federal Addendum
Five-Year Graduation Rate
Federal Updates: ELPI Waiver
Comprehensive Support and Improvement Funding Update
Submission of the LCAP Federal Addendum
Federal Budget and Appropriations
ESSA Report Cards
Per-Pupil Expenditure Reporting Expenditure Requirement






May 17, 2019

ESSA
Stakeholder
Meeting

100

Future Work on Students with Disabilities (SWDs): District of
Residence
Local Indicators: Next Steps
Incorporation of Modified Methods for CCI
Possible Modified Methods for Other State Indicators







 pptb-amard-july19item01
Appendix A
Page 1 of 3

Appendix A
Summary of States’ Graduation Rate Methods
Nineteen states use only the four-year graduation rate for accountability purposes.
Many of these states, like Massachusetts and Oregon, publish data for extended year
cohort graduation rates, broken down by student groups.
California
District of Columbia
Florida
Hawaii
Idaho
Kansas
Massachusetts
Minnesota
Mississippi
Missouri

Montana
North Carolina
Oregon
Puerto Rico
South Carolina
South Dakota
Tennessee
Texas
Virginia
Wyoming

Sixteen states use a Weighted Average of a four-year and five-year or additional
extended cohort rate for their Graduation Rate Indicator according to their Every
Student Succeeds Act (ESSA) State Plan.
Table 1: States’ Weighted Averages
State
Alabama
Alaska
Arizona
Arkansas
Delaware
Georgia
Illinois
Maine
Maryland
Michigan
Nevada
New Mexico
Ohio
Oklahoma
Pennsylvania
Utah*

Four-Year
80.0%
75.0%
45.0%
66.6%
Higher
66.6%
60.0%
n/a
66.6%
50.0%
66.6%
66.6%
66.6%
43.0%
n/a
90.0%

Five-Year
20.0%
25.0%
35.0%
33.3%
n/a
33.3%
30.0%
n/a
33.3%
30.0%
33.3%
22.2%
33.3%
36.0%
n/a
10.0%

Six- or Seven-Year
n/a
n/a
15%, 5%
n/a
n/a
n/a
10.0%
n/a
n/a
20.0%
n/a
11.1%
n/a
21.0%
n/a
n/a

* Utah does not include the five-year rate if it is not higher than the four-year rate.
-- Weightings not indicated.

 pptb-amard-july19item01
Appendix A
Page 2 of 3
Eight states use a Simple Average of their four-year, five-year and/or six-year cohort.
Table 2: States’ Simple Averages (Equal Weighting)
State

Four-Year
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes

Connecticut
Iowa
Kentucky
New Hampshire
New Jersey
New York
Rhode Island*
Vermont

Five-Year
n/a
Yes
Yes
Yes
Yes
Yes
Yes
n/a

Six-Year
Yes
n/a
n/a
n/a
n/a
Yes
Yes
Yes

* Rhode Island uses the four-year rate and a simple average of the four-year, five-year,
and six-year rate combined.
Three states use the highest graduation rate.
Table 3: States’ Highest Graduation Rates
State
Colorado

Description
Four-year is weighted by 1 percent and highest of the four, five, six, or
seven-year graduation rate is weighted by 99 percent

North Dakota

Highest of the four, five, or six-year graduation rate

Indiana

Sum of the four-year graduation rate and the percentage of students
from the previous cohort graduating exactly within five-years

Two states report graduation rates as two separate dashboard measures.
Table 4: States’ Separate Measures
State
Minnesota
West Virginia

Graduation Rate
Four-year
Four-year

Graduation Rate
Seven-year
Five-year

 pptb-amard-july19item01
Appendix A
Page 3 of 3
Three states use some other method of incorporating extended year cohorts.
Table 5: Additional Points Method
State

Description
Louisiana “Strength of diploma” index score, based on weighted
Louisiana
average point values across students within a cohort. (four, five and sixyear)
Four-year graduation rate plus adjustment for seven-year graduation
Nebraska
rate for students with disabilities
Four-year graduation rate plus potential bonus points based on sum of
Washington
four, five and six-year rates

 pptb-amard-july19item01
Appendix B
Page 1 of 1

Appendix B
Detailed Description for Three-by-Five Color
Table for the College/Career Indicator — to Meet
Section 508 of the Federal Rehabilitation Act of
1973 Requirements
A modified version of the College/Career Indicator five-by-five table. This modified
version is known as the three-by-five table and is used for small student populations
with an n-size of 149 or less to determine a performance color for the graduation rate,
suspension rate, and chronic absenteeism indicators only. In the three-by-five model,
the “Declined Significantly” and “Increase Significantly” columns for Change
Performance Levels are eliminated. To depict this modification, all the performance
color boxes under the both the “Declined Significantly” and “Increased Significantly”
columns found in a traditional five-by-five table are outlined with a red-dotted line
indicating these columns are not used to determine performance levels (color).
Removing the far right and far left columns can impact the performance color for a
school or student group if they have 149 or fewer students. For the Graduation Rate
Indicator, if a small population has a high Status Level and a Change Level of Declined
significantly, a performance color of Orange would be assigned based on the five-byfive table. However, because the Declined Significantly column is eliminated for small
populations, the performance level assigned based on the three-by-five table would be
Yellow.