March 2019 San Francisco?s Assisted Outpatient Treatment Program Three-Year Evaluation Report ha rder i 53523:?? Contents Executive Summary......................................................................................3 The AOT Team .................................................................................3 Calls to the AOT Care Team ...............................................................4 Snapshot of All AOT Participants .........................................................4 Progress toward Outcomes for all AOT Participants ................................4 Snapshot of AOT Participants with Court Order......................................5 Progress toward Outcomes for AOT Participants with Court Orders ..........6 Financial Analysis .............................................................................7 Introduction ................................................................................................8 The Assisted Outpatient Treatment Program (AOT) ................................8 Program Evaluation......................................................................... 10 Snapshot of AOT Participants ....................................................................... 12 Calls to the AOT Care Team ............................................................. 12 ................................................................................................... 13 Demographics of AOT Cases ............................................................. 13 Risk Factors ................................................................................... 15 Progress Toward Outcomes for All AOT Participants ......................................... 16 Psychiatric Emergency Services (PES)................................................ 16 Psychiatric Inpatient Hospitalization .................................................. 17 Summary of Outcome Attainment ..................................................... 19 Well-Being, Social Functioning, and Independent Living Skills................ 20 Engagement and Treatment Promotion .............................................. 20 Current Status and Disposition ......................................................... 21 Conservatorship ............................................................................ 22 Snapshot of Court Ordered AOT Participants................................................... 23 Demographics ................................................................................ 24 Risk Factors ................................................................................... 25 Progress Toward Outcomes for Court Ordered AOT Participants ......................... 27 Psychiatric Emergency Services, Inpatient Psychiatric Hospitalization, and Incarceration ................................................................................. 27 Treatment Plan Progress and Adherence ............................................ 29 Well-Being, Social Functioning, and Independent Living Skills................ 31 Financial Analysis ....................................................................................... 33 Overview and Method ...................................................................... 33 Finacial Implications........................................................................ 33 Benefits for Family Members ........................................................................ 36 Strengthening Partnerships.......................................................................... 37 Conclusion ................................................................................................ 38 Summary of Findings ...................................................................... 38 Lessons Learned ............................................................................. 38 Appendix A ............................................................................................... 40 San Francisco’s Assisted Outpatient Treatment Fact Sheet .................... 40 FAQs ............................................................................................ 42 2 Executive Summary On November 2, 2015, San Francisco Department of Public Health’s (SFDPH) Behavioral Health Services (BHS) Division launched the Assisted Outpatient Treatment (AOT) program (www.sfdph.org/aot). The program seeks to:    improve the quality of life of participants and support them on their path to recovery and wellness, prevent decompensation, and prevent cycling through acute services (e.g., psychiatric hospitalization) and incarceration. The program, authorized by San Francisco’s Board of Supervisors in 2014, is one of a handful of County-led programs in California to support the primary intent and purpose of the Assisted Outpatient Treatment (AOT) Demonstration Project Act of 2002 (Welfare and Institutions Code (WIC) §§5345-5349.5)—otherwise known as “Laura’s Law”—to (a) identify persons with serious mental illness who are not engaged in treatment, (b) assess if there is substantial risk for deterioration and/or involuntary detention (under WIC §5150) which could be mitigated by provision of appropriate services, and (c) petition the court to order participation in such services if the individual is not able to be successfully engaged by other means. Prior to implementation it was anticipated that less than 100 individuals would be referred to AOT each year, with approximately 30 individuals meeting eligibility criteria and being served by program staff. A recent statewide assessment for AOT programs by the Treatment Advocacy Center suggests that San Francisco has a higher penetration rate for clients served through AOT than approximately 60% of other counties with available program data1. The San Francisco AOT Model is utilized as an intervention and engagement tool designed to assist and support individuals with serious mental illness. San Francisco’s AOT program places an emphasis on promoting voluntary engagement by utilizing a strength-based and client-centered approach, as well as accessing an individual’s natural support system (i.e., family and friends). The program has been constructed to employ principles of recovery and wellness, and has a particular focus on community-based services and multiple opportunities for an individual to engage in voluntary treatment before a court order is requested. This is the fourth report for the AOT program in San Francisco and covers implementation and outcome data across the entire three years of operation. The AOT Team Primary program services are provided by the AOT Care Team, in accordance with San Francisco Health Code §§4111-4119, which is currently comprised of a Program Manager, a masters level clinician, and two Team Members to provide peer and family support. The AOT Care Team conducts extensive outreach to:    1 assess appropriateness of a referral and eligibility for the program, locate and engage Referred Individuals with local mental health resources, encourage voluntary participation in treatment and engagement among Referred Individuals, and https://www.treatmentadvocacycenter.org/a-promising-start 3  petition the court to order individuals into outpatient treatment when indicated. The Care Team works with Zuckerberg San Francisco General Hospital’s Division of Citywide Case Management, Citywide AOT Team, to coordinate treatment for individuals that are court ordered into treatment and support individuals who have voluntarily agreed to services in linking to long term care. Calls to the AOT Care Team Between November, 2015 and December, 2018 the San Francisco AOT program:  Received 616 calls: 295 calls for referrals—predominantly from service providers and family members—and 321 requests for information.  Of the 295 incoming referrals during the evaluation period, 129 unduplicated referred individuals were considered eligible for AOT participation and successfully contacted. The AOT team was unable to locate 8 individuals after multiple outreach attempts. Snapshot of All AOT Participants Individuals served by the AOT Care Team are at heightened risk of psychiatric hospitalization, incarceration, homelessness, and contact with Psychiatric Emergency Services (PES). In the 36 months (3 years) prior to being referred to the program:     82% had at least one known inpatient psychiatric hospitalization in San Francisco. 63% had at least one known incarceration in San Francisco. 54% experienced at least one period of homelessness. 81% had at least one known PES contact in San Francisco. Progress toward Outcomes for all AOT Participants Individuals in contact with AOT since the program’s implementation showed overall reductions in PES contacts, psychiatric hospitalization, and incarceration. Specifically:    74% of AOT participants were successful in reducing or avoiding PES contact. 91% were successful in reducing or avoiding time spent in inpatient psychiatric hospitalization. 88% were successful in reducing or avoiding time spent incarcerated. Well-Being, Social Functioning, and Independent Living Skills AOT Participants were surveyed, and respondents overwhelmingly reported a positive outlook on their future.   77% of respondents feel confident that they can reach their treatment goals. 86% of respondents feel hopeful about their future. Benefits of AOT engagement and case management were also reflected in the 4 survey responses of program participants2.    64% of respondents believed that regularly meeting with a case manager will help them to find or maintain stable housing. 67% of respondents believed that regularly meeting with a case manager will help them to maintain good physical health. 67% of respondents believed that regularly meeting with a case manager will help them live the kind of life they want. Engagement and Treatment Promotion Between November, 2015 and December, 2018, 89 of the 129 individuals served by the AOT program voluntarily engaged in services. In total, 85 of the 129 individuals (66 percent) remained connected to a treatment provider at the time of this evaluation. In addition, the Citywide AOT Team has provided clinical case management services to 43 of these individuals who have voluntarily agreed to treatment or have been court ordered to participate in treatment (26 voluntary and 17 court ordered). AOT Care Team members and the Citywide AOT Team support clients in identifying their own treatment goals and plans for the future, and then building a treatment plan to support those goals. When asked about their interactions with AOT staff in the participant survey:    82% of respondents reported that the AOT staff always treated them with respect. 81% reported that the AOT staff always listened to their concerns about treatment. 69% reported that AOT staff encouraged them to accept treatment voluntarily. Current Status and Disposition Over half (54%) of the individuals contacted by AOT whose cases are closed have achieved positive status outcomes—defined as being successfully connected to care and discharged—through their participation in the program. Additional dispositions include:     17% of AOT participants continue to be active clients and have voluntarily engaged in treatment. 2% are currently participating under a court order or settlement agreement. 5% were unable to be located by AOT staff. 15% were found to not meet the criteria for AOT participation upon assessment, but were offered support to access voluntary services. Snapshot of AOT Participants with Court Order Among the 17 court ordered AOT Participants during the evaluation period:    2 14 individuals experienced at least one PES contact in the 36 months prior to AOT contact. 15 individuals experienced at least one inpatient psychiatric hospitalization in the 36 months prior to AOT contact. 10 individuals were incarcerated at least once in the 36 months prior Surveys were administered throughout the three years of program implementation. 5  to AOT contact. 8 individuals had experienced homelessness in the 36 months prior to AOT contact. Progress toward Outcomes for AOT Participants with Court Orders The mixed methods evaluation detailed in this report is intended to fully address the regulatory program evaluation requirements of California Welfare and Institutions Code §5348(d). Findings in each of the fourteen mandated evaluation areas are summarized below, to the extent possible.  Over the course of the three-year evaluation period, a total of 17 individuals were court ordered to participate in outpatient treatment and served by the AOT program. Of the 13 court ordered individuals who have been discharged, 9 were in independent or supportive housing situations at discharge, 3 were living with family, and 1 was in a residential program. Of the 4 active court ordered individuals, 1 is in independent housing, 2 are living with family, and 1 is in a stabilization bed. At the time of this report, only 1 of the 17 individuals have had their court orders end without being successfully connected to care. (§5348(d)(1)).  Since engaging in AOT through the court process, 15 of 17 individuals have successfully reduced or avoided time spent incarcerated (§5348(d)(2)).  One court ordered participant has stabilized enough to return to the workforce, successfully securing employment, and two others have secured volunteer stipended positions with Zuckerberg San Francisco General Hospital (§5348(d)(3)).  14 of 17 individuals in the court process have successfully reduced or avoided time spent in inpatient psychiatric hospitalization since engagement in AOT (§5348(d)(4)).  The overall percentage of check-ins in which participants involved in the court process were in compliance with their treatment plans ranged from 8 percent to 100 percent, with 15 out of 17 individuals maintaining compliance levels at or above 50 percent (§5348(d)(5)).  Since the program’s implementation, the AOT Care Team has submitted a total of 11 court order renewals. At renewal, 9 out of the 11 participants volunteered to continue with the program (§5348(d)(6)).  Two individuals in the court ordered group were victims of violence during their time under court order (§5348(d)(7)).  3 of 17 individuals who are court ordered perpetrated at least one act of violence over the course of the court process (§5348(d)(8)).  6 of 17 individuals had instances of confirmed substance use over the course of the court process (§5348(d)(9)).  Throughout their participation in AOT, 100% of individuals who are court ordered had consistent contact with peer, clinical, and medical staff. Staff made at least one contact or attempted contact per week (§5348(d)(10)). 6  At times during the evaluation period, AOT Participants were supported via increased frequency of court contact. There were no other enforcement mechanisms employed by staff to encourage compliance (§5348(d)(11)). In the three years since the implementation of the AOT program, only one individual involved in the court process was asked to have increased contact with the court.  As a result of participation in AOT, individuals involved in the court process demonstrated an improved ability to function in their communities and build positive relationships with family members and others. Among the 17 participants with a court order, 11 out of 17 show improved social functioning and 10 out of the 17 show improved independent living skills (§5348(d)(12)).  With support from AOT staff, individuals involved in the court process are able to successfully access community-based services vital to maintaining stable and independent living situations (§5348(d)(13)). During the evaluation period, 49 of the voluntary participants and 7 participants with a court order or settlement agreement were successfully linked to long-term care.  The majority of participants and family members reported positive perspectives on the approach to engagement by the AOT Care Team, and felt supported by program staff. 89% of individuals involved in the court process believed that regularly meeting with a case manager will help them live the kind of life they want, while the majority of the 14 family members surveyed during this evaluation period reported feeling listed to and supported by AOT staff (§5348(d)(14)). Financial Analysis Overall, the implementation of AOT in San Francisco over the three-year evaluation period appears to have resulted in significant cost reductions related to psychiatric emergency services, inpatient psychiatric hospitalization, and incarceration, for the individuals who were served by the program. By examining average monthly utilization of these public systems for the 129 AOT participants included in this evaluation, and comparing those utilization rates for individuals in the time before and since engagement with AOT, implementation of the program appears to have resulted in an average estimated monthly savings of over $400,000. These savings are likely to have impacted both public city spending, as well as MediCal costs, depending on levels of individual coverage. 7 Introduction California State Assembly Bill 1421 (Thomson, Chapter 1017, Statutes of 2002) established the Assisted Outpatient Treatment Demonstration Project Act of 2002, more commonly referred to as “Laura’s Law” (Welfare and Institutions Code (WIC) §§5345-5349.5). The purpose and intent of Assisted Outpatient Treatment (AOT) is to:    identify persons with serious mental illness who are not engaged in treatment, assess if there is substantial risk for deterioration and/or involuntary detention (under WIC §5150) which could be mitigated by provision of appropriate services, and petition the court to order participation in such services if the individual is not able to be successfully engaged by other means. Although established through a state measure, counties can decide whether and how—outside of select goals and service requirements—to implement the AOT program in their respective areas. The present report describes San Francisco’s experience with AOT in the first three years of implementation, with a focus on the participants AOT engaged between November, 2015 and December, 2018. This is the fourth AOT Evaluation Report and provides information of use to local, state and regional government and law enforcement entities, as well as community, mental health, and other stakeholders. AOT Program Goals The Assisted Outpatient Treatment (AOT) model is designed to assist and support individuals with serious mental illness. The ultimate goals of the program are to improve the quality of life of participants and support them on their path to recovery and wellness, as well as prevent decompensation and cycling through acute services (e.g., psychiatric hospitalization) and incarceration. The Assisted Outpatient Treatment Program (AOT) In July 2014, San Francisco’s Board of Supervisors authorized Assisted Outpatient Treatment as a response to the late Mayor Ed Lee’s 2014 Care Task Force. Implemented November 2, 2015, and in operation now for over two years, the San Francisco Assisted Outpatient Treatment Model is utilized as an intervention and engagement tool designed to assist and support individuals with serious mental illness. The program has been constructed to employ principles of recovery and wellness, and has a particular focus on community-based services and multiple opportunities for an individual to engage in voluntary treatment. In San Francisco, the AOT program places an emphasis on promoting voluntary engagement by utilizing a strengths-based and client-centered approach, as well as accessing an individual’s natural support system (i.e., family and friends). If after 30+ days of engagement the staff is unable to successfully engage an individual in care, a petition to court order an individual into outpatient treatment may be pursued. This order uses the “black robe” effect—the symbolic weight of the court—to leverage an individual into care. Eligibility for AOT is initiated through a referral or petition request from a Qualified Requesting Party, as outlined in WIC §5346(b)(2). Qualified parties include an adult living with the individual, an individual’s immediate family, treatment providers, or a parole or probation officer. Eligibility of Referred Individuals is then assessed by the clinical staff from the AOT Care Team. Individuals that appear to initially meet AOT criteria are subsequently engaged by the AOT Care Team and offered voluntary services. San Francisco’s AOT model utilizes a multidisciplinary Care Team of clinical and peer-based services, as well as family support, to 8 Referred Individuals and their loved ones. The AOT Care Team conducts extensive outreach to:    locate and engage Referred Individuals with local mental health resources, encourage voluntary participation in treatment and engagement among Referred Individuals, and petition the court to order individuals into outpatient treatment when indicated. Exhibit 1 provides an overview of this model below. Exhibit 1. Overview of Assisted Outpatient Treatment Program The AOT Care Team is comprised of the Director (a psychologist with forensic experience) who oversees the general administration of the program, a licensed master’s level Program Manager who oversees the day to day operations of the program, a master’s level clinician, and two additional Team Members tasked with outreach, initial engagement and treatment, and client and family support. The AOT Care Team is housed within the San Francisco Department of Public Health’s Behavioral Health Services. In addition to the AOT Care Team, a Clinical Team from Zuckerberg San Francisco General Hospital Division of Citywide Case Management (“Citywide AOT Team”) provides intensive clinical case management services, including psychiatric assessment and treatment coordination, for individuals courtordered into treatment through AOT, and supports individuals who have voluntarily agreed to services in linking to long term care. The current Citywide AOT Team includes a Clinical Supervisor, one full-time Clinical Case Manager, a half-time Peer Specialist, and a part time Nurse and Psychiatrist. San Francisco relies on a highly collaborative model of service delivery and program improvement. Citywide AOT is in the process of hiring two additional full time clinical case managers. AOT staff work closely with service providers throughout the city, strengthening the referral network and modeling best practices. In order to remain accountable to the City of San Francisco and various stakeholder groups, AOT provides the San Francisco Health Commission and the AOT Implementation Committee with an annual update on the progress of the program, and a review of any related outcome data. Further, AOT contributes program data 9 to the Board of Supervisors via the MHSA 3-Year Integrated Plan and Annual Update, and provides any additional updates to the Board of Supervisors upon request and frequent updates to the Director of Behavioral Health Services. Outside of the city, the San Francisco AOT Team is building a network of family supports nationwide, has been instrumental in partnering with other counties that have adopted AOT, and has worked to initiate and sustain a quarterly conference call to share information and expertise. AOT Partners The following terms will be used to refer to the roles or relationships of various partners in San Francisco’s AOT program:  AOT Staff: The AOT Care Team and Citywide AOT Team defined below.  AOT Care Team: The AOT Care Team within the Department of Public Health’s Behavioral Health Services, focuses on program administration/reporting, intake, engagement and treatment of individuals, linkage to services, petitioning the court when indicated, and support through the legal process. The core AOT Care Team consists of the AOT Program Manager, clinician, and two Team Members to provide peer and family support.  Citywide AOT Team: The Department of Public Health contracts with Zuckerberg San Francisco General Hospital Division of Citywide Case Management. This organization employs a team of mental health providers and a peer specialist to provide intensive clinical case management services, which include engaging individuals that are court ordered into outpatient treatment and supporting individuals who have voluntarily agreed to services in linking to long term care. This team is a Full Service Partnership funded through the Mental Health Services Act and provides a wide range of services to meet the unique needs of each individual (the services offered exceed the requirements outlined in WIC §5348).  AOT Participants/Contacted Individuals: All individuals contacted by the AOT Care Team, regardless of whether an appointment was missed. For the purposes of this report AOT Participants are classified as:      Connected to Care and Discharged from AOT: Participants who have been successfully connected to care and are actively seeing a provider for treatment support. Accepted Treatment: Participants who have been connected to treatment and are in the process of ongoing AOT engagement. Ongoing Outreach: Participants who have yet to voluntarily accept treatment, for which the AOT Care Team continues outreach and engagement efforts. Petition Filed: Participants for whom the AOT Care Team has petitioned the court for engagement in outpatient treatment. AOT Families: Family, friends, and other close individuals who AOT Participants identify as providing them with support or assistance to AOT. Program Evaluation Behavioral Health Services (BHS) contracted with Harder+Company Community Research, in compliance with San Francisco Health Code §4118(c), to conduct an evaluation and in-depth analysis of individuals referred to AOT, the nature of 10 engagement in AOT activities, and the impact of AOT on mental health service utilization and other outcomes. The evaluation is intended to fully address regulatory program evaluation requirements and is summarized in an annual report each year of the program evaluation.3 Now, having completed three full years of implementation, this final report seeks to provide insight into the program’s longterm impact on all participants seen since the onset of the program. To fully understand the program’s impact, Harder+Company and the AOT Care Team chose a multi-stakeholder, mixed methods evaluation which draws on the following data sources: 3  Program Data: Data collected for all AOT Contacted Individuals since November 2015. This information includes AOT-related intake, initial, ongoing, and attempted contacts, background information, service linkage and use, and key events tracking.  Participant Surveys: A paper-based questionnaire assessing thoughts about and experiences with AOT engagement and service linkage has been distributed to participants regularly throughout their time in AOT. Questionnaire completion was voluntary and no financial incentive was provided. AOT Participants submitted questionnaires to their case managers in a confidential and sealed envelope, who then passed them along to the evaluators at Harder+Company.  Participant Interviews: A 20 minute interview conducted with AOT Participants with a court order. Interviews explored individuals’ experiences with the AOT program, engagement, and service linkage, as well as thoughts on how the AOT program has impacted their lives. A total of three interviews were completed by Harder+Co and all were given a $25 incentive for their time.  Family Surveys: An eight item survey was distributed to members of the AOT Participants’ families. The survey asked respondents about their satisfaction with AOT support, perceived benefits and challenges, as well as recommendations for the program. The Family Survey was administered online and via mail to consenting families in January 2018. No incentives were provided, and a total of 12 respondents completed the survey.  Staff Interviews: A 30-minute interview conducted with members of the AOT Care Team and Citywide AOT Team to gather perspectives and reflections on program implementation, effectiveness, and larger social and organizational impacts. A total of 10 interviews were completed.  Stakeholder Survey. A stakeholder survey was launched in November 2017 to collect feedback on program implementation. The AOT Care Team helped identify partners who they felt could speak to the successes and challenges faced by the AOT program to-date, as well as any general recommendations they could share. A total of 22 stakeholders submitted a completed survey.  Financial Analysis. In order to better understand the long-term financial implications of AOT in San Francisco, financial analysis was conducted to determine changes in the amount of public funding utilized for participants’ incarceration, inpatient psychiatric hospitalization, and psychiatric emergency service use since the implementation of AOT. California Welfare and Institutions Code §5348(d) 11 Snapshot of AOT Participants The AOT program has established various methods by which Qualified Requesting Parties, as defined by WIC §5346 (b)(2) (e.g., an adult living with the individual, immediate family members, treatment providers, or a parole/probation officer), can make referrals and request information about the program. The program offers a toll-free number, local number, referral form and website, and general email address, and conducts outreach presentations to multiple stakeholders. Eligibility of referred individuals is then assessed by the clinicians on the AOT Care Team. This section of the report details information on all the individuals the AOT Care Team engaged throughout the entirety of the program – starting November 5, 2015 to December 31, 2018. Section Overview This section of the report details the following information:  Information on all calls received by the AOT Care Team during the evaluation period.  Demographic information on all active cases during the evaluation period.  Risk factors for all active cases during the evaluation period. Calls to the AOT Care Team Since November 2015 the AOT Care Team received 616 calls regarding the program. Of the 616 calls received, 321 were information only calls (52 percent) while the remaining 295 calls (48 percent) were referral requests, as detailed in the following exhibit. Exhibit 2. Type of AOT Calls (n=616) 52% Information Only 48% Referral The most common source of referral calls was providers (53 percent), while 39 percent of the calls were from family members. Approximately 5 percent of referral calls were from persons who were not determined to be a Qualified Requesting Party (QRP). In these cases, the AOT Care Team worked with callers to identify potential QRPs to make the referral. 12 Exhibit 3. AOT Referring Parties (n=295) 53% 39% 5% Provider Family Non QRP 2% 0% Adult Living with RI Parole The majority of the calls made to the AOT program originated in San Francisco (76 percent), while 16 percent of referrals originated from other counties in California. An additional 8 percent of referral calls originated from outside of California. Exhibit 4. Origin of Referral Calls by California County (n=295) 79% San Francisco Out of State 8% Contra Costa 3% San Mateo Alameda 3% 2% Marin 1% Solano 1% Unknown 1% Los Angeles 1% Santa Clara 1% Sonoma 1% Sacramento 0% San Luis Obispo 0% Santa Cruz 0% Demographics of AOT Cases Of the 295 individuals referred to AOT since the onset of the program, 129 unduplicated referred individuals were considered eligible for AOT participation and successfully contacted; AOT was unable to locate 8 individuals after multiple outreach attempts.4 A small number of individuals were referred to the program multiple times5, in which case only their most recent engagement with AOT was included in the analysis. These AOT participants demonstrate considerable demographic diversity, as illustrated below. Since 2015 the AOT program served a majority of individuals who identified as 4 In cases where an individual was determined not to be an eligible candidate, the referral source was provided with a consultation regarding alternative resources and, when indicated, family support was provided. 5 8 individuals have two referrals to the AOT program, while one individual has 3. 13 male (59%) and 40 percent identified as female. Approximately 1 percent identified as non-binary and another 1 percent as transgender. Exhibit 5. Gender of Contacted Individuals (n=129) 59% 40% Female Male 1% 1% Non binary Transgender Male The greatest proportion (27 percent) of contacted individuals is between 26 and 35 years of age. Overall, most program participants are under 46 years of age. Exhibit 6. Age Range of Contacted Individuals (n=129) 27% 26% 19% 18% 18-25 26-35 36-45 46-55 5% 5% 56-65 65+ The majority of individuals contacted identify as White (42 percent), followed by 23 percent of individuals identifying as Black/African American, 16 percent as Asian, and 11 percent as Latinx. 14 Exhibit 7. Race of Contacted Individuals (n=129) White 42% Black/African American 23% Asian 16% Latinx 11% Multi-Ethnic 7% Native American 1% Unknown 1% Risk Factors Individuals in contact with AOT demonstrate significant needs, and are at heightened risk of psychiatric hospitalization, contact with Psychiatric Emergency Services (PES), incarceration, and homelessness. The exhibit below shows the percentage of program participants who experienced the above at least once during the 36 months prior to their participation in the AOT program. Most participants (82 percent) experienced at least one inpatient psychiatric hospitalization, and 81 percent of program participants experienced at least one PES contact in the 36 months prior to intake. Incarceration and homelessness were also common among the majority of participants. Exhibit 8. Psychiatric Hospitalization, PES Contact, Incarceration, and Homelessness in the 36 Months Prior to AOT Contact (n=129) Homelessness Incarceration 54% 63% Psych Hospitalization 82% PES 81% 15 Progress Toward Outcomes for All AOT Participants Section Overview To assess how the AOT program has impacted its participants, the following section compares participants’ interactions with Psychiatric Emergency Services (PES), psychiatric inpatient hospitalization, and incarceration before and after their intake into the AOT program. The goal of this analysis is to track any changes in participants’ contact with each of the three over time, before, during, and after each participant’s engagement with the program and the AOT Care Team to better understand the long term impact of AOT as an intervention. This section of the report details the following information:  Outcome information on the number of incidents and any changes in the average number of visits/days per month over time  Information on participants’ wellbeing, social functioning, and independent living skills, as detailed by program participants  Information and quotes on the Citywide AOT Team’s engagement with program participants  Information on all participants’ current status and disposition In order to provide a holistic analysis of the program’s impact, this section looks at the outcomes for all AOT participants since the beginning of the program’s implementation, a total of 129 unduplicated individuals or 139 duplicated individuals.6 Of these 129 individuals, 89 have voluntarily engaged in services. Psychiatric Emergency Services (PES) Among the participants analyzed, the number of individuals who experienced PES visits dropped from 105 individuals (81 percent) in the 36 months prior to their AOT participation, to 60 (47 percent) since contact with AOT. Exhibit 9. PES Occurrence 36 Months Prior and Since AOT Contact (n=129) 81% 47% 36 Months Prior Since AOT In addition, average PES visits per month—for all participants—fell from 0.18 visits per month in the 36 months prior to AOT participation, to 0.07 after ending participation in AOT. This reduction in PES visits per month after exiting the program was also found to be statistically significant7, offering strong evidence that the observed change may be related to AOT participation. 6 The unduplicated count is the number of unique individuals that participated in the AOT program. The duplicated count is the number of cases eligible to participate in the AOT program, including instances in which individuals participated multiple times in the program. 7 Paired t (df)=128, p-value<0.05 Exhibit 10. Average PES Visits per Month (n=129) 0.20 0.18 0.07 Before During After Psychiatric Inpatient Hospitalization The number of individuals experiencing at least one psychiatric inpatient hospitalization in the 36 months prior to intake dropped from 106 (82 percent) before AOT contact, to 40 (47 percent) since. Exhibit 11. Hospitalization 36 Months Prior, and Since AOT Contact (n=129) 82% 47% 36 Months Prior Since AOT Although the total number of individuals experiencing an inpatient hospitalization decreased, average days per month hospitalized as a whole increased for the population of participants while they were actively engaged in AOT. However, this average increase from 0.96 days per month in the 36 months prior to AOT participation, to 2.27 days while actively engaged decreases to an average of 0.29 days per month after participants have left the program. It is likely that these changes in the average number of days hospitalized over time is attributable to the participants’ high need for services and long delayed access to much needed care, as well as the AOT Team’s active assessment and advocacy with hospital staff and the court to ensure that participants receive the clinically appropriate level of care. Additionally, participants are encouraged to seek support and services when needed. Similar to the outcomes for PES, the decrease in the average days hospitalized per month after exiting the program was also found to be statistically significant8 and further indicate the program’s long-term effects on participants. 8 Paired t (df)=126, p-value<0.05 17 Exhibit 12. Average Days Hospitalized per Month (n=129) 2.27 0.96 0.29 Before During After Incarceration Lastly, when looking at participants’ experiences with the justice system, the number of individuals with a recent incarceration dropped from 81 individuals (63 percent) prior to AOT contact, to 51 (40 percent) since intake. Exhibit 13. Incarceration 36 Months Prior, and Since AOT Contact (n=129) 63% 40% 36 Months Prior Since AOT Average days incarcerated per month shows the biggest long-term decrease over time, from 2.31 days per month in the 36 months prior to intake to 1.34 days per month after exiting the program. While average days per month increased to 3.13 while actively engaged in the program, this trend in average days per month is in line with the trend seen in days hospitalized per month and likely related to coordination of care and advocacy to ensure that legal dispositions take mental health needs into account. This decrease in average days per month after exiting the program was also found to be statistically significant9, again, offering strong evidence that these decreases can be attributed to participation in AOT. 9 Paired t (df)=127, p-value<0.05 18 Exhibit 14. Average Days Incarcerated per Month (n=129) 3.13 2.31 1.34 Before During After Summary of Outcome Attainment Exhibit 15, below, summarizes all of the above outcomes, including the observed changes over time in average PES visits per month, average days hospitalized per month, and average days incarcerated per month, before, during, and after engagement with the program and the AOT Care Team. Exhibit 15. Changes in Outcomes, 36 Months Prior, and Since AOT Contact 36 Months Prior 0.18 During Contact 0.20 Average Days Hospitalized per Month 0.96 2.27 0.29 Average Days Incarcerated per Month 2.31 3.13 1.34 Average PES Visits per Month After Exiting 0.07 The scope of AOT’s role in reducing negative outcomes is also evident when the averages referenced above are disaggregated. When each individual’s monthly averages are compared pre and post-AOT contact, findings continue to suggest successful outcome attainment among the group. Individuals are considered to have attained a successful outcome by either reducing their monthly average of negative occurrences or—for those with no occurrences of a negative outcome prior to AOT contact—experiencing no occurrences of the negative outcome since AOT contact. Since the beginning of the program, 74 percent of AOT participants were successful in reducing or avoiding PES contact, 91 percent were successful in reducing or avoiding time spent in inpatient psychiatric hospitalization, and 88 percent were successful in reducing or avoiding time spent incarcerated. Additionally, at the end of December 2018, 68 unique program participants (53 percent) were closed and remained engaged in treatment. 19 Well-Being, Social Functioning, and Independent Living Skills The AOT program supports the basic needs and well-being of participants through provision of supportive resources, such as access to food, assistance applying for state identification cards, Supplemental Security Income (SSI), CalFresh (food stamps), and transportation assistance (e.g., Clipper Card). Peer staff provide additional support around activities of daily living, employment coaching, and other independent living skills as needed. Participants who were not able to function successfully in the community prior to AOT are often able to navigate complex processes with the additional support of the Care Team and Citywide AOT Team. When asked about their future via an AOT Participant Questionnaire, most participants had a positive outlook. Eighty-six percent of survey respondents reported that they feel hopeful about their future, and 77 percent reported feeling confident that they can reach their treatment goals. Exhibit 16. Outlook on Treatment Goals and Future (n=66) I am confident that I can reach my treatment goals n=51 (77%) I feel hopeful about my future n=57 (86%) For participants working with a case manager, the perceived impact of treatment on quality of life is measureable. Benefits of AOT engagement and case management were reflected in the survey responses of program participants. 64 percent of respondents believed that regularly meeting with a case manager will help them to find or maintain stable housing, 67 percent believed that regularly meeting with a case manager will help them to maintain good physical health, and 67 percent believed that regularly meeting with a case manager will help them live the kind of life they want. Exhibit 17. Perspectives on Meeting with a Case Manager (n=66) How much will regularly meeting with a case manager help you to… A Lot A Little/Not at All …find or maintain stable housing? n=42 (64%) n=13 (20%) …maintain good physical health? n=44 (67%) n=13 (20%) …live the kind of life you want? n=44 (67%) n=13 (20%) Engagement and Treatment Promotion The Citywide AOT Team provides intensive clinical case management for courtordered AOT Participants, and provides support for individuals who agree to voluntary services, linking them to long term care. This support is critical to clients’ long term success. As one Care Team member explains, “[the Citywide AOT Team] really meets each person they work with wherever they are at to engage and stabilize them… Once stable in the community, the team then coordinates a smooth and successful transition to a long term provider.” According to the Citywide AOT Team, achieving benefits for program participants requires being persistent and client centered. One staff member shared how the 20 unique nature of AOT allows them to successfully engage people who would otherwise remain disconnected: “For a lot of our clients the systems have already tried, they've been trying for years to engage these individuals and reduce hospitalization, reduce incarceration, and reduce crisis contacts. But I think the uniqueness of this program, having additional peer support, having low caseloads, the flexibility to do the amount of outreach that we're doing, I really think this program is changing and improving the lives of these individuals.” Since the beginning of AOT implementation, the Citywide AOT Team has provided case management support to 43 individuals who have voluntarily agreed to treatment, 17 of whom have been court ordered court ordered. Of these, 9 voluntary and 4 court ordered individuals are still receiving services (as of December 2018) from the Citywide Team that are designed to stabilize them in preparation for linkage to long term services. Fourteen of the voluntary participants and 5 court ordered participants were successfully linked to long-term care. Only 1 court ordered individual was not successfully linked to long-term care due to refusing ongoing services and referrals. Besides persistence, respect for participants was mentioned by staff as central to the success of the program. AOT Care Team members emphasized that client engagement begins with supporting the client in identifying their own treatment goals and plans for the future, and then building a treatment plan to support those goals. This emphasis on respect is validated by participants’ responses in the participant survey. Eighty-two percent of respondents reported that staff have always treated them with respect, and 81 percent reported that the staff always listened to their concerns about treatment. Sixty-nine percent also noted that they were always encouraged to accept treatment voluntarily, which may be related to the open conversations that the AOT Team has with participants regarding the possibility of a court order. Exhibit 18. Interactions with the AOT Team The AOT Team always… …treated me with respect n=23 (82%) …listened to my concerns about treatment n=21 (81%) …encouraged me to accept treatment voluntarily n=18 (69%) Current Status and Disposition As of the end of 2018, when looking at non-active cases, 54 percent of individuals in contact with AOT achieved positive status outcomes—defined as being successfully connected to care and discharged—through their participation in the program. This total includes 12 individuals who were court ordered and successfully connected to care. Twenty-five individuals (19 percent) were active AOT cases at the time of data collection. In cases where an individual does not meet the legal eligibility for AOT, staff support the referral entity in identifying other appropriate resources. In total, 15 percent of individuals were found to not meet the criteria of participation for the program after initial contact was made. A full accounting of dispositions for the 129 individuals included in the evaluation is presented in Exhibit 19 below. 21 Exhibit 19. Disposition of Contacted Individuals (as of 12/31/18) Disposition Closed – court order not granted n=2 (2%) Closed – court order ended / withdrawn n=12 (9%) Closed - other n=8 (6%) Closed – successfully connected / re-engaged in services n=49 (38%) Closed – criteria not met – client outreached n=19 (15%) Closed – referral withdrawn n=7 (5%) Closed – unable to locate n=7 (5%) Open – active case n=21 (16%) Open – court ordered / settlement agreement n=4 (3%) Conservatorship Sixteen individuals (12 percent) that AOT outreached and engaged with since the beginning of the program were determined to need a higher level of care than the AOT program provides. With the support of AOT staff, the Department of Aging and Adult Services, and the courts, these 16 individuals were placed on an LPS Conservatorship, providing them the safety and stability needed to ensure their recovery. Conservatorship services are provided only for individuals who meet the definition of grave disability, which includes being unable to provide for their basic needs (i.e. food, clothing, and shelter) due to a mental disorder. These services may include individualized treatment, supervision, and an involuntary medication order. Of the 16 individuals conserved, 13 remain housed in the community, while 3 now reside in locked facilities which provide them with a higher level of care and personal safety appropriate for their needs. 22 Snapshot of Court Ordered AOT Participants In cases where the AOT Care Team is unable to successfully engage individuals in voluntary treatment, petitions may be filed for court ordered treatment. These individuals who are court ordered are subject to enhanced monitoring by the AOT Care Team, and present significant needs and challenges. From the beginning of the program to present, the AOT Care Team has filed a total of 31 petitions for 20 unique individuals, 17 of which moved forward either as a settlement agreement10 or a court order to participate in the AOT program.11 This snapshot and all the exhibits subsequently shown in this section of the report focus on the 17 individuals who moved forward with a settlement agreement or were court ordered to participate in the program, henceforward referred to as individuals who were court ordered into the program. Section Overview This section of the report details the following information:  Demographic information on all court ordered individuals during the evaluation period  Risk factors for all court ordered individuals during the evaluation period Exhibit 20. Contacted Individuals and Court Orders (n=129) 87% 13% Not Court Ordered Court Ordered Additionally, since the beginning of implementation, the AOT Care Team has submitted a total of 11 court order renewals. At renewal, 9 out of the 11 participants volunteered to continue with the program, and one renewal was withdrawn. Only one individual had to be court ordered into treatment upon renewal. 10 A Settlement Agreement is when an individual accepts services once a petition is filed with the court. 11 Of the petitions filed that did not move forward, one case withdrew and later voluntarily engaged in services, and the other had their court order dismissed. 23 Demographics Among court ordered AOT Participants, 8 (47 percent) identify as female, and 9 (53 percent) identify as male. Exhibit 21. Gender of Individuals who are court ordered (n=17) 53% 47% Female Male Since the beginning of AOT implementation, the majority (71 percent) of court ordered participants have been younger than 45 years of age. Three participants are between 46 and 55 years of age, one is between 56 and 65 years of age, and one is older than 65 years of age. Exhibit 22. Age of Individuals who are court ordered (n=17) 35% 29% 18% 6% 18-25 26-35 36-45 46-55 6% 6% 56-65 65+ The highest percentage of court ordered individuals identified as White (35 percent), while four court ordered participants identify as Black/African American (24 percent) and two as Latinx (12 percent). 24 Exhibit 23. Race/Ethnicity of Individuals who are court ordered (n=17) Multiracial 18% Native American 6% Asian 6% Latinx 12% Black/African American 24% White 35% Risk Factors Fourteen of the 17 individuals who were court ordered during the evaluation period experienced at least one PES contact in the 36 months prior to AOT contact, and 15 experienced at least one inpatient psychiatric hospitalization. Ten individuals who are court ordered were incarcerated at least once in the 36 months prior to AOT contact, and 8 had experienced homelessness. Exhibit 24. PES Contact, Psychiatric Hospitalization, Incarceration, and Homelessness for Individuals who are court ordered in the 36 Months Prior to AOT Contact (n=17) PES 82% Psych Hospitalization 88% Incarceration Homelessness 59% 47% 25 Of the 17 court ordered cases included during the evaluation period, the majority were either living independently, living in supportive housing, or living with family (88 percent). Only one individual is in a stabilization bed. Exhibit 25. Housing for Individuals who are Court Ordered (Current or at Time of Discharge) Housing Status Total Independent 6 Supportive Housing 4 With Family 5 Stabilization Bed/Room 1 Residential Program 1 26 Progress Toward Outcomes for Court Ordered AOT Participants As with outcomes for all AOT Participants in the previous section, outcome data for all 17 individuals who are court ordered or reached a settlement agreement were examined across the three years since the program’s implementation and are included as a part of the following analysis. Due to the small number of individuals who are court ordered, the subsequent analysis cannot show statistical significance. However, notable reductions in negative outcomes are highlighted where relevant. Psychiatric Emergency Services, Inpatient Psychiatric Hospitalization, and Incarceration The number of individuals who experienced PES contact dropped from 14 in the 36 months prior to AOT participation, to 10 since being court ordered. Exhibit 26. PES Occurrence 36 Months Prior, and Since AOT Court Order (n=17) 82% 59% 36 Months Prior Since Court Order The number of individuals with a recent inpatient psychiatric hospitalization dropped moderately from 15 in the 36 months prior to AOT participation, to 14 since being court ordered. Exhibit 27. Hospitalization 36 Months Prior, and Since AOT Court Order (n=17) 88% 82% 36 Months Prior Since Court Order 27 Finally, the number of individuals with a recent incident of incarceration changed from 6 in the 36 months prior to AOT participation to 3 since being court ordered. Exhibit 28. Incarceration 36 Months Prior, and Since AOT Court Order (n=17) 59% 29% 36 Months Prior Since Court Order Due to the small size of the court ordered AOT population, exploring changes in the averages of PES visits, days hospitalized, and days incarcerated is not particularly meaningful at the aggregate level. Instead, it is helpful to examine changes in these occurrences over time at the individual level. Exhibit 30, below, highlights each of the court ordered participants’ changes in average occurrences per month, from 36 months prior to AOT contact, to post court order. Exhibit 29. Changes in Average PES Visits per Month, Days Hospitalized per Month, and Days Incarcerated per Month, by Participant PES Visits per Month Pre Post Participant 1 0.08 0.00 Positive Outcome  Participant 2 0.19 0.05  Days Hospitalized per Month Positive Pre Post Outcome  0.39 0.00 Days Incarcerated per Month Positive Pre Post Outcome  0.00 0.00 0.60  0.03 2.15 1.36 0.00  0.00 0.00   0.17 0.00  5.03 0.00  0.00 0.00  6.44 0.00  0.00 0.00  5.50 Participant 3 0.19 0.00  Participant 4 0.94 0.00  0.11 0.00 Participant 5 0.08 0.07  0.44 0.86 0.00 0.00 1.00 8.00 Participant 6 0.00 0.00  Participant 7 0.33 0.23  1.50 0.00  Participant 8 0.08 0.00   Participant 9 0.11 0.00  1.03 1.40 0.11 13.47 Participant 10 0.22 0.00  0.67 0.00  5.44 0.00  0.83 0.00  0.50 0.00  0.00 0.00  17.86 0.00  0.00 0.00  Participant 11 0.08 0.00  Participant 12 0.00 0.00  0.89 0.00  Participant 13 0.11 0.00  Participant 14 0.00 0.00  1.22 0.00  0.00 0.00  Participant 15 0.06 0.00  1.67 0.00  0.11 0.00  3.28 0.00  0.00 0.00  2.89 0.00  0.56 0.00  Participant 16 0.03 0.00  Participant 17 0.08 0.00  As defined in a previous section of this report, individuals are considered to have attained a successful outcome by either reducing their monthly average of negative occurrences or—for those with no occurrences of a negative outcome prior to AOT contact—experiencing no occurrences of the negative outcome since AOT court order. During the evaluation period, all 17 ordered participants were successful in reducing or avoiding PES contact, 14 were successful in reducing or avoiding time 28 spent in inpatient psychiatric hospitalization, and 15 were successful in reducing or avoiding time spent incarcerated. Treatment Plan Progress and Adherence The following exhibits detail various outcomes for individuals who are court ordered as tracked by the AOT Care Team since the program’s implementation. Exhibit 30. Court Ordered Treatment Plan Adherence % of Check-Ins Compliant with Treatment Confirmed Substance Use Violence Victim Violence Perpetrator Participant 1 98% - - - Participant 2 32%  -  Participant 3 62%  -  Participant 4 92% - - - Participant 5 89%  - - Participant 6 58% - - - Participant 7 62% -   Participant 8 64%  - - Participant 9 8% - - - Participant 10 100% - - - Participant 11 100% - - - Participant 12 100%  - - Participant 13 100%   - Participant 14 50% - - - Participant 15 88% - - - Participant 16 75% - - - Participant 17 100% - - - 29 Exhibit 31. Court Ordered Enforcement Mechanisms Participant 1  Increased contact with the court - Participant 2     Participant 3  -  - Participant 4 - - - Participant 5 - -  Participant 6  -  - Participant 7 - - -  Participant 8  -   Participant 9  - - Participant 10 - -  Participant 11 - -  Participant 12 - -  - Participant 13 - -   Participant 14 - -   Participant 15 - - - - Participant 16  - -  Participant 17 - -  - Order to appear Judge feedback Court order extended   Exhibit 32. Court Ordered Social and Living Skills Improved Social Functioning Improved Independent Living Skills - - Participant 2 - Participant 3 Participant 5    Participant 6 -      Participant 7 - Participant 8  - Participant 9 - - Participant 10      Participant 15       Participant 16 - - Participant 17  - Participant 1 Participant 4 Participant 11 Participant 12 Participant 13 Participant 14 - - 30 Exhibit 33. Court Ordered Participant Survey Results Participant is confident they can reach their treatment goals Participant is hopeful about their future Participant 4     Participant 5 - Participant 6 Participant 8    Participant 9 N/A12 N/A N/A Participant 10  - - Participant 11 N/A N/A N/A Participant 12 N/A N/A N/A Participant 1 Participant 2 Participant 3 Participant 7         Participant received help to navigate the AOT program and courts when they needed it         Participant 13   - Participant 14 N/A N/A N/A Participant 15 N/A N/A N/A Participant 16 N/A N/A N/A Participant 17 N/A N/A N/A Well-Being, Social Functioning, and Independent Living Skills Before participants enter AOT, many are in need of extensive support to stabilize. However, interventions for hard-to-reach individuals often require a prolonged outreach process, as was described by one family member who participated in the family survey: “AOT and I kept in contact and repeatedly tried to get [my child engaged in services]. [AOT Care Team staff] set aside a time every week hoping he would show up. He finally went in and they began working with him and got him back on his medicine. It took us 2 years but without the AOT team he could have been killed.” Interventions, like the one described above, are only possible because of the way AOT is structured, with an emphasis on client-centered outreach, flexibility, and meeting individuals where they are. As a result of this consistent outreach, the AOT Team is able to successfully engage and stabilize individuals in a way that positively impacts their well-being. As mentioned previously, AOT staff also support participants as they navigate different systems of care and establish a foundation for a more normalized life. One court ordered participant explained how helpful AOT Care Staff were in helping them accomplish several things, specifically, “[AOT Care Team staff] made me feel safe. They were worried about me being on the streets. [Since being court ordered] I haven't been on the street as much, I haven't been homeless… [They] helped me get my ID, and [they’re] working on helping me get my SSI.” This individual’s 12 N/A is indicated for individuals who have not completed a participant questionnaire 31 experience illustrates the gradual but impactful ways that stabilization can occur and the client-centered nature of outreach and engagement. In the time since this interview, this individual has remained housed and actively engaged in services. Both their social functioning and independent living skills have shown a marked increase since engagement began, and continue to improve. “AOT’s diligence and care has literally been life saving for my sister.” This sort of stabilization is most easily achieved when participants willingly engage in services, but not all do. Nonetheless, AOT’s practice of consistent and persistent outreach and engagement allows them to reach otherwise disconnected individuals. –Family Member Benefits of AOT’s unique engagement structure for individuals who are court ordered were also reflected in participants’ survey responses. Nearly all (88 percent) of respondents who were court ordered believed that regularly meeting with a case manager will help them to find or maintain stable housing. 89 percent believed that regularly meeting with a case manager will help them to maintain good physical health, and 89 percent believed that regularly meeting with a case manager will help them live the kind of life they want. Exhibit 34. Court Ordered Perspectives on Meeting with a Case Manager How much will regularly meeting with a case manager help you to… A Lot A Little/Not at All …find or maintain stable housing? 88% 12% …maintain good physical health? 89% 11% …live the kind of life you want? 89% 11% Consistent AOT engagement with individuals who are court ordered appears to also positively impact participants’ outlooks on their future. When surveyed, nearly all (90 percent) respondents who were court ordered reported feeling confident that they can reach their treatment goals, and 90 percent reported that they feel hopeful about their future. When interviewed, one court-ordered individual discussed their desire to be able to work again in the future, and another discussed concrete employment goals that they are working toward. To date, one court ordered participant has stabilized enough to return to the workforce, successfully securing employment. 32 Financial Analysis Overview and Method As previously mentioned, the overarching goals of AOT implementation include the reduction of particular client outcomes, including use of psychiatric emergency services, inpatient psychiatric hospitalization, and incarceration (§5348(d)). In addition to their impacts on AOT participants and their families, these outcomes also have financial implications for the public systems that support them. At the time of this report, a single instance of PES contact is associated with a cost of $2,962, one day of inpatient psychiatric hospitalization carries a cost of $2,837, and a day spent in San Francisco jail comes at a cost of $245. In the case of PES contact and inpatient psychiatric hospitalization, these costs are at times absorbed by MediCal coverage, if the individual is a beneficiary. Whether the impact is on MediCal or city spending, even slight reductions in AOT participants’ contact with these systems can have significant financial implications. In order to better understand these financial implications, and in compliance with San Francisco health code (§4118(c)), this three-year evaluation report includes an analysis of the total estimated costs associated with the three outcomes noted above, for all 129 AOT participants. Data on each of these individuals’ contact with PES, psychiatric hospitalization, and the jails over time was used to estimate the total average monthly cost of service for each. For the purposes of analysis, each participant’s average monthly utilization of these services was calculated both for the time before their engagement with AOT, and the time since their engagement. These monthly averages were then summed across the entire AOT participant pool, creating an estimate of overall monthly costs before clients engage with AOT, and after. This calculation serves as a reliable estimate of the actual cost implications for the public systems supporting AOT participants. Financial Implications Overall, the implementation of AOT in San Francisco over the three-year evaluation period appears to have resulted in significant cost reductions related to psychiatric emergency services, inpatient psychiatric hospitalization, and incarceration, for the individuals who were served by the program. Prior to their engagement with AOT, these 129 individuals combined carried a total average cost of $67,139 per month for psychiatric emergency services (22.7 instances per month total or 0.18 instances per person), $345,799 per month for inpatient psychiatric hospitalization (121.9 days per month total or 0.94 days per person), and $72,422 per month for time spent incarcerated (295.4 days per month total or 2.29 days per person). In total, utilization of these three systems averaged over $485,000 per month for the pool of 129 participants, before their engagement in AOT. When examining average costs in the time since these participants engaged in AOT (including both their time during the program and after case closure), these 129 individuals carried a total average cost of $50,286 per month for psychiatric emergency services, $29,452 per month for inpatient psychiatric hospitalization, and $2,007 per month for time spent incarcerated. In total, utilization of these three systems averaged $81,745 per month for the pool of AOT participants, when looking at the time since they became engaged with AOT. This represents an average estimated monthly savings of $403,614, (approximately 83 percent). Exhibit 35. Average estimated monthly cost of psychiatric emergency services before and since engagement with AOT $67,139 $50,286 Monthly Cost Pre-AOT Monthly Cost Since AOT Exhibit 36. Average estimated monthly cost of inpatient psychiatric hospitalization before and since engagement with AOT $345,799 $29,452 Monthly Cost Pre-AOT Monthly Cost Since AOT Exhibit 37. Average estimated monthly cost of incarceration before and since engagement with AOT $72,422 $2,007 Monthly Cost Pre-AOT Monthly Cost Since AOT 34 Exhibit 38. Average total estimated monthly cost of psychiatric emergency services, inpatient psychiatric hospitalization, and incarceration before and since engagement with AOT $485,360 $81,745 Monthly Cost Pre-AOT Monthly Cost Since AOT 35 Benefits for Family Members With the understanding that participants benefit from strong family support during their journey to recovery and wellness, San Francisco prioritized family engagement when planning and implementing AOT. Consequently, as part of the AOT process, families are engaged and offered education and support, including information about eligibility, benefits, limitations, and opportunities of the program. The fourteen family members surveyed during this evaluation period learned about AOT and were referred to the program through multiple avenues, including a therapist/case manager/mental health agency, an AOT-referred family member, an attorney/judge, the National Alliance on Mental Illness (NAMI), homeless services, and even an article about AOT. As a result of the support and education AOT has provided, many families reported that they had more or better access to resources and a better understanding of the symptoms their loved ones experience. Half reported a better understanding of what AOT is and how it can help their family member, and a better understanding of the mental health system as a whole. Some reported appreciating the way the AOT Team listened and talked to them. For example, one family member described how AOT has supported them and helped them to support their loved one, saying “Dealing with my [loved one’s] situation is hands down, the most difficult challenge /obstacle in my life… For me personally, it has been a tremendous help to be able to talk to someone [from the AOT staff]. I am eternally grateful.” AOT staff repeatedly emphasized two ways that AOT is uniquely situated to support families—the program’s ability to empower family members of at-risk individuals and the support it provides directly to family members. One of the biggest benefits for families, as reported by both family members and AOT staff, is the ability for families to directly refer their loved ones to the program. As one AOT staff member explained, it empowers families “to feel like they have agency, they have a voice, and they have an avenue to get their loved ones support when nothing else they've tried has really worked.” Additionally, reflecting on their experiences working with clients and their families, AOT staff acknowledged the stress that families face when supporting the long term recovery and success of program participants, highlighting the second major benefit for families. As one AOT staff member explained: “AOT saved my family in a way I can’t explain. I’m very grateful to this amazing program and team.” –Family Member “[AOT] supports family members, helping them understand their loved ones' diagnosis and that this is an ongoing journey. We teach them new coping skill tools to use… we find out what their needs are and what their concerns are and we try to accommodate them with supportive help, with agencies, and with referrals out to other family support groups.” This support, staff said, is critical, because investing time to educate family members on mental health, what it means to have a mental illness, and how to strengthen participants’ support systems can result in improved outcomes for program participants. 36 Strengthening Partnerships By strengthening the network of providers that AOT works with, the AOT Care Team is able to better identify and connect participants with appropriate treatment options. In November 2017, the AOT Care Team identified a number of providers and other stakeholders they felt could speak to the successes and challenges faced by the AOT program to date. These stakeholders were asked to complete a survey where they shared program feedback, their observations about the impact of AOT on Referred Individuals, and general feedback about AOT’s role as part of the larger mental health system in San Francisco. When asked, most stakeholders reported observing numerous positive changes in Referred Individuals. Eighty-six percent of respondents reported at least some increased interaction with mental health services, 73 percent reported at least some increase in more positive interactions with mental health services, and 68 percent reported at least some increase in mental health stability. 68 percent also reported at least some increase in housing stability, while 64 percent of stakeholders surveyed observed at least some improvement in Referred Individuals’ functioning. Exhibit 39. Positive Changes Observed in Referred Individuals To what extent have you observed the following positive changes in Referred Individuals since beginning AOT? At least some Increased interaction with mental health services n=19 (86%) More positive interactions with mental health services n=16 (73%) More stable mental health n=15 (68%) More stable housing n=15 (68%) Improved functioning n=14 (64%) Through strong collaboration with providers across San Francisco, AOT offers access to care and support tailored to each participant’s unique needs, while simultaneously strengthening partnerships throughout San Francisco’s system of care. One stakeholder shared that collaboration begins during outreach, saying “AOT Case Managers have increased relationships with multiple partners, including street medicine and crisis services, and get folks medicated and evaluated quickly.” Collaboration extends beyond San Francisco’s network of providers, as well, with one stakeholder explaining “I think the collaboration between DPH, the City Attorney's Office, the clinicians, the Public Defender's Office and the courts is very successful. I think we all feel we are working to together for the best interest of the individual.” Both AOT staff and stakeholders discussed a desire to expand collaboration to better support participants on their journey to recovery and wellness, something AOT is actively pursuing through increased community outreach and education. One AOT staff member explained, “we're continuing to do presentations in the community, making sure to really get that information out there to people about what AOT is, what we can do, how we're here to support both referred individuals as well as community providers, and that we really want to work together.” “The AOT program has brought agencies together and made for better working relationships that were not there prior.” –Stakeholder 37 Conclusion Summary of Findings During the three-year evaluation period, AOT Participants demonstrated overall reductions in PES visits, inpatient psychiatric hospitalization, and incarceration. Among all AOT Participants to-date, 74% reduced or avoided PES contact through participation in AOT, 91% reduced or avoided time spent hospitalized, and 88% reduced or avoided time spent incarcerated. Among the group of 17 court ordered AOT participants, all 17 reduced or avoided PES contact, 14 reduced or avoided time spent hospitalized, and 15 reduced or avoided time spent incarcerated. Before participants enter AOT, many are in need of extensive support to stabilize. This is most easily achieved when participants willingly engage in services, but not all do. Nonetheless, AOT’s practice of consistent and persistent outreach and engagement appears to have a positive effect on individuals, stabilizing them even when they are not engaged in treatment. Participants who were not able to function successfully in the community prior to AOT are often able to with the additional support of the AOT Team, and the majority of AOT Participants surveyed feel confident that they can reach their treatment goals, and feel hopeful about their future. The findings detailed in this report suggest overall reductions in negative outcomes for voluntary and court-ordered AOT Participants, as well as improvements in functioning and well-being. Given the stated goals of the AOT program to prevent decompensation and reduce cycling through acute services and incarceration, as well as the significant decrease in negative outcomes following an individual’s exit from the program shown in this report, there is substantial evidence that indicates that implementation to date can be considered a success. In addition, analysis of the public costs associated with utilization of psychiatric emergency services, inpatient psychiatric hospitalization, and incarceration indicates significant estimated savings as a result of AOT implementation. Lessons Learned At the closing of this evaluation cycle, there are a number of promising tools and practices, employed by Care Team and the Citywide AOT Team alike, that have emerged since the implementation of AOT began in November 2015. Many of these can serve as models for best practices throughout the system of care, and they, along with other key insights and lessons learned, are summarized below. Engagement and Treatment Promotion  Persistence pays off. Staff, stakeholders, family members, and AOT participants all noted that consistent and persistent outreach by AOT staff leads to a higher likelihood of engagement, including with individuals who are initially unwilling to participate in treatment. This allows AOT to engage some of the hardest to serve individuals, many of whom the system of care has been unsuccessful in engaging in the past. This persistent outreach and high-touch service model is designed specifically to serve the small number of individuals who meet the legal eligibility criteria and are not otherwise consenting to necessary services. 38  Meet people where they are. AOT staff spend a significant amount of time outreaching individuals on the street, in their residences, in hospitals, and jail. The AOT Staff’s extensive outreach and engagement efforts allow for participants to be engaged at times and places when they are most readily able to engage, positively impacting participation in services and treatment.  High touch case management increases the likelihood of stabilization and successful engagement. The provision of intensive case management support for voluntary participants is critical for successful stabilization and long term engagement in treatment, providing consistent support to individuals when they need it most.  Let participants take the lead when possible. Engagement focused on participants’ own treatment goals and future plans allows for respectful and client-centered support. However, in instances where individuals are not successful in engaging in voluntary care, AOT’s provision of the court order option is vital for promoting stabilization. Family Support  Strengthening families can improve outcomes. Educating family members on mental health, what it means to have a mental illness, and how to strengthen participants’ support systems can result in improved outcomes for program participants.  Allowing family members to refer their loved ones directly to AOT helps connect hard to engage individuals to services. Many family members reported feeling discouraged and even let down over their previous inability to help their loved ones access appropriate services throughout the system of care. By empowering family members to refer directly, individuals who may otherwise be disconnected from the system of care are more likely to be identified, outreached, and served. Relationship Building  Building a network of strong partnerships enhances the system of care. Strengthening relationships between agencies and organizations throughout the system of care allows for information and resource sharing, and enhances service provision throughout.  Collaboration throughout the system of care allows for the provision of individualized care and services and increases the likelihood of successful engagement. Through strong collaboration with providers across San Francisco, AOT staff have been able to efficiently connect participants to services tailored to their specific needs. When participants are supported by a collaborative network of providers in this way, they are less likely to disconnect from services, increasing the likelihood of stabilization, as well as successful and consistent engagement during recovery. Contact for More Information For more information on San Francisco’s AOT program, or to make a referral, please visit www.sfdph.org/aot. To refer an individual by phone, please dial (415) 255-3936 or, toll free, (844) 255-4097. 39 Appendix A San Francisco’s Assisted Outpatient Treatment Fact Sheet Summary Assisted Outpatient Treatment, also referred to as “AOT or” “Laura’s Law,” was enacted in 2002 by California Assembly Bill 1421 and refers to court-ordered outpatient treatment for individuals who have a severe mental illness. In counties that choose to adopt the program, AOT allows for adults who meet certain requirements to request that the county mental health director petition the court to mandate treatment for individuals who have previously refused care and meet strict eligibility requirements. Overview of the Process 40 Who is Eligible for AOT? An individual must meet all of the following criteria to qualify for AOT: 1) Be at least 18 years of age; 2) Suffer from a serious mental disorder (defined by W&I §5600.3 (b)(2) and (3)); 3) Be unlikely to survive in the community without supervision, per clinical determination; 4) Demonstrate a history of failing to comply with treatment (one of the following must be true): a) The person’s mental illness has been a key factor in necessitating psychiatric hospitalization or mental health services while incarcerated at least twice within the last 36 months, not including the period immediately preceding the petition for AOT, or b) The person’s mental illness has resulted in one or more incidents of serious and violent behavior toward himself or another in the last 48 months, not including the period immediately preceding the petition for AOT; 5) Have been offered the opportunity to participate in treatment but failed to engage; 6) Be substantially deteriorating; 7) Be an appropriate match for AOT, meaning that AOT offers the least restrictive placement needed to ensure recovery and stability; 8) Be unlikely to relapse or be subject to an involuntary psychiatric hold (5150) with AOT; and 9) Likely benefit from AOT. Who Can Request a Petition for AOT? A request for AOT may be initiated by the following adults (age 18+): 1) Any adult who lives with the individual with mental illness; 2) A parent, spouse, sibling, or adult child of the individual with mental illness; 3) The director of a mental health institution in which the individual with mental illness lives; 4) The director of a hospital where the individual with mental illness is hospitalized; 5) A licensed mental health provider supervising the treatment of the individual; or 6) A peace, parole, or probation officer assigned to supervise the individual. 41 FAQs Who is AOT designed to help? AOT is designed to assist individuals who have a documented severe mental illness, who are not actively engaged in care, are in deteriorating condition, and have a history of failing to comply with treatment. AOT requires that individuals meet strict eligibility guidelines, as outlined above. How many people are expected to be eligible for AOT in San Francisco? SFDPH estimates participation to be fewer than 100 annually. (SFDPH currently provides mental health care for approximately 31,000 San Franciscans at 23 SFDPH mental health clinics and programs, and 300 contracted programs in the community. About 7,200 patients are treated each year at psychiatric emergency department at San Francisco General Hospital and Trauma Center. Does AOT help provide care for people with mental illness who are homeless? In some cases, homeless people will be eligible for AOT; in other cases they will not. AOT has strict eligibility criteria that apply regardless of whether an individual is housed. These criteria include the requirement that AOT be initiated by someone who knows the individual, either personally (family member or co-habitant) or professionally (mental health provider or peace, parole or probation officer assigned to supervise the individual), and that the individual not be actively engaged in mental health treatment. What are the individual’s rights in the process? AOT strictly defines patient eligibility criteria in an effort to ensure appropriate application of the law and to protect individual rights. AOT provides at least two opportunities to engage patients in voluntary treatment prior to a court hearing. Additionally, AOT specifically defines the rights of the individual with mental illness who is subject to AOT, including adequate notice of hearings, to receive a copy of the court-ordered evaluation, to a court appointed public defender in the absence of private counsel, to be present at the hearing, to present evidence and call and/or cross-examine witnesses, and to appeal decisions. What is the difference between AOT and a 5150? A “5150” refers to Section 5150 of the California Welfare & Institutions Code and is an emergency hold in response to a psychiatric crisis, allowing for up to 72 hours of involuntary psychiatric evaluation and treatment of persons believed to be a danger to self, a danger to others, or gravely disabled by mental illness. AOT is a non-crisis process that allows for an adult that meets AOT criteria and declines voluntary treatment to be compelled by a civil court process to receive mental health care in the community. The goal of AOT is to support individuals with mental illness in the community in an effort to prevent future crisis. If someone does not comply with court-ordered AOT are they automatically subject to a 5150? No. Failure to comply with AOT alone may not be grounds for a 5150 involuntary hold or for a contempt of court finding. The criteria for a 5150 involuntary hold are already prescribed in state law and are no different for AOT Participants than for any other individual. In order to meet the criteria for a 5150 an individual must be at imminent risk for danger to self, at imminent risk for danger to others, or be gravely disabled (unable to care for basic needs such as food, shelter, and clothing) due to a mental illness. What are consequences of noncompliance with court-ordered AOT? If the treating mental health treatment provider believes that the individual is a danger to self, a danger to others, or gravely disabled and in need of involuntary treatment, the provider may initiate the 5150 process. There are no additional enforcement mechanisms for individuals who do not meet 5150 criteria. However, some jurisdictions that have implemented AOT have noted that court involvement itself can prompt some 42 patients to choose treatment, including medication. This has been called the “Black Robe Effect.” Would AOT reduce the number of 5150s? The impact of AOT on 5150 involuntary holds is unknown. Once implemented, the data collection, reporting, and evaluation requirements under AOT would likely answer this question. How does AOT differ from SFDPH’s existing Community Independence Placement Program? The Community Independence Placement Program (CIPP) is a voluntary program for individuals who have been subject to a 5150 involuntary hold and who meet the grave disability criteria required for conservatorship. Participation in the program is initiated in the hospital and participants are transitioned to community-based care. Participants agree to allow a conservator and the mental health court to work on their behalf to ensure that they adhere to their prescribed treatment plans, including medication adherence. AOT is court-ordered treatment initiated while the individual resides in the community. AOT provides a mechanism for family members and others who know the individual well to help engage an individual into treatment without requiring hospitalization or law enforcement. Individuals who meet strict eligibility requirements may be ordered by the court to receive mental health treatment. Can AOT mandate medication? No. State statute specifies that involuntary medication shall not be allowed absent a separate court order available only for individuals who are conserved due to their grave disability. How much will AOT cost? Other communities that have implemented AOT (Orange County, Nevada County) estimate the mental health treatment costs at $35,000-$40,000 per person per year. This does not include costs associated with the judicial system. Per State statute, no voluntary mental health programs may be reduced as a result of the implementation of AOT. How will the effectiveness of AOT be evaluated? Counties that implement AOT are required to collect and report key data to the State Department of Health Care Services for evaluation. At minimum the evaluation is required to include data that relates to number of individuals that receive services through AOT, engagement of AOT Participants in services, and key data points to measure the effectiveness of AOT as an intervention and engagement tool (e.g., hospitalization, contact with law enforcement, social functioning, and employment). 43