Department of Veterans Affairs Office of Inspector General Office of Healthcare Inspections Report No. 16-03985-181 Healthcare Inspection Evaluation of the Veterans Health Administration Veterans Crisis Line March 20, 2017 Washington, DC 20420 In addition to general privacy laws that govern release of medical information, disclosure of certain veteran health or other private information may be prohibited by various Federal statutes including, but not limited to, 38 U.S.C. §§ 5701, 5705, and 7332, absent an exemption or other specified circumstances. As mandated by law, OIG adheres to privacy and confidentiality laws and regulations protecting veteran health or other private information in this report. To Report Suspected Wrongdoing in VA Programs and Operations: Telephone: 1-800-488-8244 E-Mail: vaoighotline@va.gov Web site: www.va.gov/oig Table of Contents Executive Summary ................................................................................................... i Purpose....................................................................................................................... 1 Background ................................................................................................................ 1 Scope and Methodology............................................................................................ 9 Inspection Results ..................................................................................................... Objective 1. Veterans Crisis Line Failure to Respond Adequately to a Veteran Caller.................................................................................................. Objective 2. Veterans Crisis Line Review .............................................................. Objective 3. Status of VHA’s Actions for Implementation Issued in Response to Office of Inspector General February 2016 Report ............................ Objective 4. Office of Special Counsel Complaint .................................................. 11 11 12 37 42 Conclusions................................................................................................................ 47 Recommendation(s)................................................................................................... 50 Appendices A. Timeline: History of Veterans Crisis Line, Chronology of Office of Inspector 52 General Reviews, and Publication of GAO Report ........................................ B. Veterans Crisis Line Organizational Chart June 2016 ....................................... 53 C. Veterans Crisis Line Organizational Chart July 2016 ........................................ 54 D. Veterans Crisis Line Organizational Chart September 2016 ............................. 55 E. Silent Monitors for Quality Assurance Monthly Report ...................................... 56 F. Acting Under Secretary for Health Comments ................................................... 57 G. Office of Inspector General Contact and Staff Acknowledgments ..................... 68 H. Report Distribution............................................................................................. 69 Evaluation of the Veterans Health Administration Veterans Crisis Line List of Abbreviations AAS American Association of Suicidology CAB Clinical Advisory Board COR Contracting Officer’s Representative EHR electronic health record ELC Executive Leadership Council GAO U.S. Government Accountability Office HEC Health Eligibility Center L2HS Link2Health Solutions, Inc. MIRECC Mental Illness Research, Education and Clinical Centers MHS Mental Health Services NEO New Employee Orientation NSPL National Suicide Prevention Line OIG Office of Inspector General OMHO Office of Mental Health Operations OSC Office of Special Counsel OSP Office of Suicide Prevention QACO Quality Assurance Clinical Officer QASP Quality Assurance Surveillance Plan QM quality management QMO Quality Management Officer SPC Suicide Prevention Coordinator SSA Social Service Assistants VCL Veterans Crisis Line VCL Atlanta Veterans Crisis Line located in Atlanta, GA VCL Canandaigua Veterans Crisis Line located in Canandaigua, NY VHA Veterans Health Administration VA OIG Office of Healthcare Inspections Evaluation of the Veterans Health Administration Veterans Crisis Line Executive Summary Suicide, the act of taking one's own life,1 is a serious public health concern. As a medical, psychiatric, and social issue, it is accentuated in the veteran population. On July 7, 2016, the Veterans Health Administration (VHA) announced the results of a review of 55 million veteran records covering the period from 1979-2014. Based on 2012 data, VA estimated that the number of veteran deaths by suicide averaged 22 per day. There was a slight decrease in 2014, when an average of 20 veterans a day died from suicide.2 Either figure, 22 veteran suicides per day or 20 veteran suicides per day, significantly exceeds the national average. For example, in 2014, the rate of suicide among U.S. civilian adults was 15.2 per 100,000, while the rate of suicide among veterans was 35.3 per 100,000.3 The American Foundation for Suicide Prevention4 notes that suicide has no single cause, but “most often occurs when stressors exceed the current coping abilities of an individual suffering from a mental health condition.”5 Various treatments exist for many of the underlying mental health conditions that increase the risk of suicide. However, many in emotional distress cannot, will not, or do not have the ability to contact a mental health provider or other caregiver directly. “Crisis lines” help to fill this breach. The essential concept of a crisis line is that competent and compassionate individuals are available to provide around-the-clock resources to any individual in distress, to include help, support, and referrals, and even to arrange for immediate evaluation if necessary. In 2007, VHA established a telephone suicide crisis hotline. Initially called the National Veterans Suicide Prevention Hotline, its name changed to the Veterans Crisis Line (VCL) in 2011.6 The primary mission of the VCL is “to provide 24/7, world class, suicide prevention and crisis intervention services to veterans, service members, and their family members.”7 Since its launch in 2007, through September 2016, VCL staff have answered over 2.5 million calls and initiated the dispatch of emergency services to callers in crisis over 66,000 times. The VCL faces two major challenges. First is to meet the operational and business demands of responding to over 500,000 calls per year, along with thousands of 1 Lippincott Williams & Wilkins. Stedman's Medical Dictionary, Twenty-Eighth Edition. December 6, 2005. VA Office of Public and Intergovernmental Affairs, “VA Conducts Nation’s Largest Analysis of Veteran Suicide,” July 7, 2016, 09:56:00 AM, http://www.va.gov/opa/pressrel/pressrelease.cfm?id=2801. Accessed December 12, 2016. 3 Facts about Veteran Suicide, July 2016, VA Suicide Prevention Program, http://www.va.gov/opa/publications/factsheets/Suicide_Prevention_FactSheet_New_VA_Stats_070616_1400.pdf. Accessed December 5, 2016. 4 Established in 1987, the American Foundation for Suicide Prevention (AFSP) is a voluntary health organization that gives those affected by suicide access to a nationwide community empowered by research, education and advocacy. 5 American Foundation for Suicide Prevention, 120 Wall Street, 29th Floor, New York, NY 10005. Accessed December 4, 2016. 6 Veterans Crisis Line 1-800-273-8255 Press 1, https://www.veteranscrisisline.net/About/AboutVeteransCrisisLine.aspx. Accessed December 4, 2016. 7 VCL Mission Statement. 2 VA Office of Inspector General i Evaluation of the Veterans Health Administration Veterans Crisis Line electronic chats and text messages, and initiating rescue processes when indicated. Second is to train staff to respond to veterans and their family members in individual encounters during which a responder8 must make an accurate assessment of the needs of the caller under stressful, time-sensitive conditions. The responder must be able to competently assess whether a caller is an imminent danger to self or others and needs emergency rescue services; whether the caller is not suicidal or homicidal but nonetheless is distressed and needs to be matched with mental health services in VA or the community; or whether the call is not appropriate for a crisis line such as a routine call for information, a prank call, or a wrong number. These complex and difficult challenges are not unique to the VCL as we observed other crisis hotlines that face similar issues. Although we made findings and recommendations concerning the VCL, we note an unwavering and impressive commitment by VCL staff who compassionately assist veterans in crisis. The VA Office of Inspector General Office (OIG) conducted a healthcare inspection of the VCL. The inspection had four primary objectives:  To respond to a complaint alleging that the VCL did not respond adequately to a veteran’s urgent needs.  To perform a detailed review of the VCL’s governance structure, operations, and quality assurance functions in order to assess whether the VCL was effectively serving the needs of veterans.  To evaluate whether VHA completed planned actions in response to VA OIG recommendations for the VCL, published on February 11, 2016, in our report titled Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (Report No. 14-03540-123).  To address complaints received from the U.S. Office of Special Counsel (OSC) alleging inadequate training of VCL Social Service Assistants (SSAs),9 resulting in deficiencies in coordinating immediate emergency rescue services needed to prevent harm. Prior to the initiation of this review, we received a request from Senator Bill Nelson (FL) to evaluate aspects of suicide prevention, including the VCL. We address his concerns related to the VCL in this report. Our review of other aspects of suicide prevention will be addressed in future oversight. 8 Responders answer the calls, texts, and chats that are received by the VCL. Responsibilities of the SSA include attempting to locate the caller based on information obtained by the responder, communicating with the responder and VCL supervisors, and obtaining assistance from emergency rescue services. The SSA has access to and utilizes a database of local emergency resources pertinent to the caller’s location. 9 VA Office of Inspector General ii Evaluation of the Veterans Health Administration Veterans Crisis Line Findings to Objective 1: VCL Failure to Respond Adequately to a Veteran Caller. We found that VCL staff did not respond adequately to a veteran’s urgent needs during multiple calls to the VCL and its backup call centers. The failure to respond adequately to the veteran’s urgent needs resulted in missed opportunities to provide crisis intervention services. In addition to the failure to provide crisis intervention during the calls, VCL supervisory staff did not identify the deficiencies in their internal review of the matter. Due to privacy concerns and to avoid discussion of protected information, we are unable to further discuss the veteran’s interaction with the VCL. Findings to Objective 2: VCL Governance Structure, Operations, and Quality Assurance Functions Have a Number of Deficiencies. Our inspection of the VCL governance structure, operations, and quality assurance functions identified a number of deficiencies. The more significant findings included the following: We found deficiencies in the VCL’s processes for managing incoming telephone calls. Callers may decide to remain anonymous, but in every case, responders document the incoming telephone number. However, responders must manually enter the number into the electronic documentation system, which increases the risks of human error. While reviewing responders’ call documentation, we found that the documentation was often lacking in sufficient detail to facilitate retrospective assessment of the interaction between the caller and responder. We also found deficiencies in governance and oversight of VCL operations. Both administrative and clinical leaders are vital to running a crisis line. In February 2016, the VCL was realigned under VHA Member Services (Member Services), whose responsibilities are administrative in nature and do not include providing clinical services. To address this, VA leadership stated that the VCL would remain closely tethered to the VA Office of Mental Health Operations and VHA Clinical Operations. Despite this expectation that Member Services and the subject matter experts on suicide prevention work closely together, we found substantial disagreement about key decisions and oversight between the two groups. For example, the Clinical Advisory Board (CAB) was described as the major vehicle for clinical input, but during the review, we found low attendance at CAB meetings and that the role of the CAB was changed to a consultative body. Clinical staff expressed concern that the CAB was not effectively used to obtain clinical input for policy decisions. The lack of effective utilization of clinical decision makers at the highest level of VCL governance resulted in the failure to fully include clinical perspectives impacting the operations of the VCL. Another example of deficient governance was lack of permanent VCL leadership. During 2015, the VCL was without a permanent director for 10 months. At the end of 2015, a permanent director was chosen. The new permanent director resigned his position in June 2016. As of December 2016, the VCL continued to operate without a permanent director. VA Office of Inspector General iii Evaluation of the Veterans Health Administration Veterans Crisis Line Member Services leaders collected VCL call performance data but did not collect data regarding attempted or completed suicides following a veteran’s contact with the VCL. Because VCL leaders did not collect data regarding attempted or completed suicides following a veteran’s contact with the VCL, they could not demonstrate whether the performance data improved the ability to evaluate the efficacy of the clinical mission. Moreover, a number of VCL staff expressed concerns that under Member Service leadership, the emphasis is on business and efficacy metrics with less emphasis on clinical quality measurement. The Suicide Prevention Office leads suicide prevention efforts for VHA and coordinates and disseminates evidence-based findings related to suicide prevention. The Director of the VHA Suicide Prevention Office stated that the Acting Director of the VCL informed her that she had no authority over the VCL. We found limited communication between the VHA Suicide Prevention Office and VHA Member Services about suicide prevention and the VCL. The VCL staff did not have the capacity to answer all calls received, requiring VHA contract with four backup call centers not otherwise affiliated with VA to handle the overflow. VHA contracted with an external vendor to manage backup center performance and report to VCL, with administrative and clinical oversight by VCL managers. We found that VHA contracting staff and Member Services and VCL leaders lacked an understanding of the contract terms and did not verify quality control aspects of contractor performance, resulting in deficient oversight. A component of the VCL long-term plan was to expand to a second site to ensure geographic redundancy and to expand coverage, with the goal of achieving zero rollover calls to backup call centers. Atlanta was chosen as the site for the second call center, given a pre-existing call center infrastructure and a large pool of potential employees living in the Atlanta metropolitan area. Planning for the new call center started on July 21, 2016, with a timeline of full telephonic operation by December 31, 2016. This schedule was considered ambitious by a number of VCL staff. The Atlanta Call Center accepted its first call on October 10, 2016, but did not meet its primary target date of zero rollover calls to backup call centers by November 21, 2016, or by its secondary target date of December 12, 2016. Member Services leaders did not establish on-site leadership for the Atlanta Call Center, or develop and implement a formal business plan to achieve strategic priorities and communicate key project objectives and goals. We were also informed during interviews that Member Services leaders did not heed warnings from VCL clinical staff who predicted obstacles to achieving proposed operational targets in the intended timeline (such as inadequate staffing and training resources). Lack of formal planning and inaccurate forecasting resulted in more than 16,000 hours of Canandaigua FTE (full–time equivalent) employees being temporarily redirected to the Atlanta Call Center for training and operations. This led to an increase in the number of calls that rolled over to backup centers and delays in the development and implementation of VCL processes, policies, and procedures. VA Office of Inspector General iv Evaluation of the Veterans Health Administration Veterans Crisis Line VCL Quality Management (QM) focuses on making and measuring improvements to a program with the prevention of problems being the primary objective. In our February 2016 report, we recommended the establishment of a formal quality assurance process. VCL leaders concurred with the recommendation and submitted an action plan with a target date for completion of September 30, 2016. VCL did not meet this timeframe and requested a March 2017 deadline extension. We found continued deficiencies in the VCL QM program. VHA provides a framework for QM leadership and program structure to ensure delivery of safe and effective care,10 yet VCL QM staff lacked background or training in quality management principles and processes. Moreover, VCL staff collected data for clinical performance measures but lacked a reporting structure for communication of data analysis, development of action plans, or follow-up for assessment of process improvement. We did not find evidence that QM staff reviewed data until August 2016, following an OIG site visit. Starting in September 2016, this data was used to evaluate process improvement. VCL leaders defined the success of the call center partly in terms of suicide reduction. However, the VCL had no process in place for routinely obtaining or reviewing data on serious outcomes, such as attempted or completed suicides by veterans who made contact with the VCL prior to the event. By not reviewing serious adverse outcomes, VCL QM managers missed opportunities for quality improvement. VCL managers established a process to track caller complaints. The caller complaint data included complaints related to time “on hold.” We found that some backup call centers use a queuing process that may lead callers to perceive they are on hold. A queued call is one that has been routed to a call center, but not yet answered. During this time, the caller waits for a responder to answer. The caller’s only option is to abandon the call (hang up) and call back, or continue to wait for the responder to pick up. VCL staff described that queuing a call before a responder answered was not the same process as placing a call on hold after a responder answered. We found that VCL leadership had not established expectations or targets for queued call times or thresholds for taking action on queue times. A veteran could be queued for 30 minutes, for example, and that wait time might not be reflected in hold time data; however, the result of the delay is the same, whether the veteran was in a queue or on hold. We found several challenges in VCL QM staff’s ability to collect, analyze, and effectively review relevant QM data in order to make changes to prevent problems and improve outcomes for callers. After reviewing the number and types of QM roles in the VCL, as well as QM staff experience and background, we determined that the challenges likely stemmed from the QM staff’s lack of training in QM principles. The QM staff had not been provided with training in the skills needed to provide leadership to promote quality and safety of care, leading to deficiencies in the QM program. 10 VHA Directive 1026, VHA Enterprise Framework for Quality, Safety, and Value, August 2, 2013. VA Office of Inspector General v Evaluation of the Veterans Health Administration Veterans Crisis Line VCL policies were not consistent with existing VHA policies for veteran safety11 or risk management12 and did not incorporate techniques for evaluating available data to improve quality, safety, or value, to veterans. For example, the VCL could enhance performance improvement evaluations by using call recordings to monitor the quality of interactions between responders and callers, and by collecting and analyzing performance data from the new Atlanta Call Center separately from the Canandaigua Call Center. The new call center in Atlanta could potentially have QM concerns that are no different from its Canandaigua partner, but the ability to recognize site-specific issues, especially in a new program, may be facilitated by separating quality data elements by site. As of December 15, 2016, VCL staff reported technological capability to listen to call recordings at the Canandaigua location. VCL staff reported that the Atlanta location did not have call recording capability, but had plans to have call recording by the end of December 2016. Neither location had procedures, protocols, or policies that provided guidance for listening to or using call recording information. Findings to Objective 3: VHA Has Not Implemented Actions Plans Submitted in Response to the February 2016 OIG Report. We made seven recommendations in a previously published report, Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (Report No. 14-03540-123, February 11, 2016). VHA concurred with the recommendations and agreed to a completion deadline of September 30, 2016. We consider all prior recommendations to remain open as of the publication of this report. Our specific findings, relative to the February 2016 recommendations, are below: Recommendation 1. We recommended that the OMHO (now VHA Member 13 Services) Executive Director ensure that issues regarding response hold times when callers are routed to backup crisis centers are addressed and that data is collected, analyzed, tracked, and trended on an ongoing basis to identify system issues. Appropriate VHA leaders did not demonstrate the use of the Link2Health14 data to improve performance. Although we found evidence that VCL staff reviewed Link2Health data, we did not find that they used the data systematically to provide feedback to backup centers regarding performance parameters such as queue times, abandonment rate, and call answer rate. At the time of this report, VHA had not completed the necessary actions to close this recommendation. 11 VHA Handbook 1050.01, VHA National Patient Safety Improvement Handbook, March 4, 2011. This Handbook was scheduled for recertification on or before the last working date of March 2016 and has not yet been recertified. 12 VHA Directive 2010-025, Peer Review for Quality Management, June 3, 2010. This Directive expired June 30, 2015 and has not yet been updated. 13 The VCL was realigned under VHA Member Services in the spring of 2016. At the time the February 2016 OIG report regarding the VCL was published, the Office of Mental Health Operations was responsible for the VCL. 14 Link2Health is an incorporated subsidiary of the non-profit Mental Health Association of New York City, which has a contract with VHA to provide a platform for routing telephone calls from the National Suicide Prevention Line toll-free number to the VCL. VA Office of Inspector General vi Evaluation of the Veterans Health Administration Veterans Crisis Line Recommendation 2. We recommended that the Member Services Executive Director ensure that orientation and ongoing training for all VCL staff is completed and documented. We found that VHA did not ensure that orientation and ongoing training for all VCL staff was completed and documented. At the time of this report, VHA had not completed the necessary actions to close this recommendation. Recommendation 3. We recommended that the Member Services Executive Director ensure that silent monitoring frequency meets the VCL and American Association of Suicidology requirements and that compliance is monitored. We found that VHA had not yet ensured that the VCL establish a requirement for silent monitoring frequency of VCL calls and therefore could not have monitored compliance with such a requirement.15 VHA requested an extension until March 2017, and this recommendation remains open. Recommendation 4. We recommended that the Member Services Executive Director establish a formal quality assurance process, as required by VHA, to identify system issues by collecting, analyzing, tracking, and trending data from the VCL routing system and backup centers, and that subsequent actions are implemented and tracked to resolution. The VCL lacked key components of a formal quality assurance process necessary to comply with VHA requirements. Specifically, the VCL had not designated individuals with appropriate background and skills to provide leadership to promote quality and safety of care. VCL policies did not incorporate relevant existing VHA directives outlining the key elements of a successful QM program. The VCL also lacked a committee that regularly reviewed data, information, or risk intelligence, and that ensured that key quality, safety, and value functions were discussed and integrated into VCL processes. VHA requested a deadline extension of March 2017 to implement this recommendation. Recommendation 5. We recommended that the Member Services Executive Director consider the development of a VHA directive or handbook for the VCL. VHA concurred in the original response to this recommendation and stated it would establish a VHA directive for the VCL. We found that while VHA had made significant progress, it had not completed a VHA directive for operating the VCL (such as outlining the VCL purpose, roles and responsibilities). VHA requested a deadline extension of March 2017 to complete the VHA directive for the VCL. 15 The VCL QM manual assigned Silent Monitor staff to attempt to monitor each responder at least once per pay period. The QM manual acknowledged the unenforceable nature of this objective in stating, “At times, this is not possible due to employee’s assignment to other duties which take them off the phones, such as precepting new hires or participating in other special projects.” VA Office of Inspector General vii Evaluation of the Veterans Health Administration Veterans Crisis Line Recommendation 6. We recommended that the Member Services Executive Director ensure that contractual arrangements concerning the VCL include specific language regarding training compliance, supervision, comprehensiveness of information provided in contact and disposition emails, and quality assurance tasks. We found that while VHA established a new contract in April 2016 with a goal of improved quality assurance monitoring, VHA did not ensure compliance with the quality assurance surveillance plan for VCL backup center performance. VHA planned to extend the current contract for 6 months and requested an extension for implementation of this recommendation until September 2017. Recommendation 7. We recommended that the Member Services Executive Director consider the development of algorithms or progressive situation-specific stepwise processes to provide guidance in the rescue process.16 VHA stated that they were developing standard work processes for all caller types including SSA responsibilities during emergency dispatch. We found that VCL managers developed standard situation-specific progressive stepwise processes to provide guidance in the rescue processes. However, VHA needs to demonstrate the training and competence of SSAs on the current protocol to achieve closure. VHA requested a deadline extension of March 2017 to implement this recommendation. Findings to Objective 4: A Number of Issues Raised by a Complainant and Referred by the Office of Special Counsel Were Substantiated. The OSC referred a complaint to VA on August 25, 2016 alleging inadequate training of VCL SSAs that resulted in deficiencies in coordinating immediate emergency services needed to prevent harm. OIG accepted the referral and did a review of the issues related to the complaint. We found that the VCL developed a monitoring process for responders; however, the VCL lacked a process for monitoring the quality of performance by SSAs. It was not stated when VCL policies for training responders or SSAs, or policies for QM, were changed or disseminated to staff, and VCL managers lacked the ability to monitor whether staff received updated policies. We substantiated the OSC complainant’s allegations that SSAs were allowed to coordinate emergency rescue responses independently after the end of a 2-week training period, without supervision, and regardless of performance or final evaluation. We substantiated the OSC complainant’s allegations that in mid-2016, a newly trained SSA contacted a caller in crisis by telephone to solicit the veteran's location. We found that no harm resulted from the interaction. VCL management reviewed this event, confirmed a violation of VCL policy, and took administrative action against the SSA. Because of this case, supervisors reminded VCL staff of the VA Privacy and Security Policy via email and SharePoint website notification. 16 VCL staff consider rescues, welfare checks, and dispatch of emergency services to be equivalent terms. VA Office of Inspector General viii Evaluation of the Veterans Health Administration Veterans Crisis Line We substantiated the OSC complainant’s allegations regarding a lack of documentation by an SSA when closing out a veteran’s case in mid- 2016. A supervisor counseled the SSA on documentation requirements. We verified that the documentation concern was not a training issue but a specific personnel concern. We could not substantiate the OSC complainant’s allegations that in mid-2016, documentation entered by an SSA resulted in conflicting information on whether a veteran in crisis was contacted within 24 business hours. Because of the complainant’s anonymity, we could not interview the complainant to obtain information that identified the caller, and the VCL’s managers could not locate the record without any caller identifiers. VCL managers described the potential for an SSA to document initially that the veteran could not be contacted, and if that was the case, enter a referral to the SPC. Once the SPC reached the veteran, the coding could be changed to a “contacted” status, resulting in an apparent conflict. Recommendations 1. We recommended that the Under Secretary for Health implement an automated transcription function for callers’ phone numbers in the Veterans Crisis Line call documentation recording system. 2. We recommended that the Under Secretary for Health ensure that Veterans Crisis Line policies and procedures, staff education, Information Technology support, and monitoring are in place for audio call recording. 3. We recommended that the Under Secretary for Health implement a Veterans Crisis Line governance structure that ensures cooperation and collaboration between VHA Member Services and the Office of Suicide Prevention. 4. We recommended that the Under Secretary for Health develop clear guidelines that delineate clinical and administrative decision-making,  assuring that clinical staff make decisions directly affecting clinical care of veterans in accordance with sound clinical practice. 5. We recommended that the Under Secretary for Health ensure processes are in place for routine reviewing of backup call center data, establish wait-time targets for call queuing and rollover, and ensure plans are in place for corrective action when wait-time targets are exceeded. 6. We recommended that the Under Secretary for Health ensure processes are in place to require contracted backup centers to have the same standards as the Veterans Crisis Line related to call queuing and wait-time targets. 7. We recommended that the Under Secretary for Health ensure that VHA Member Services leadership, Veterans Crisis Line leadership, VHA Contracting Officers, and Contracting Officer Representatives implement the quality control plan and conduct ongoing oversight to ensure contractor accountability in accordance with their roles as specified in the contract with backup call centers. VA Office of Inspector General ix Evaluation of the Veterans Health Administration Veterans Crisis Line 8. We recommended that the Under Secretary for Health ensure that training is provided to Veterans Crisis Line quality management staff in the skills needed to provide leadership to promote quality and safety of care. 9. We recommended that the Under Secretary for Health ensure the development of structured oversight processes for tracking, trending, and reporting of clinical quality performance measures. 10. We recommended that the Under Secretary for Health ensure processes for Veterans Crisis Line quality management staff to collect and review adverse outcomes so that established cohorts of severe adverse outcomes are analyzed. 11. We recommended that the Under Secretary for Health direct the Veterans Health Administration Assistant Deputy Under Secretary for Health for Quality, Safety, and Value to review existing Veterans Crisis Line policies and determine whether the policies incorporate the appropriate Veterans Health Administration policies for veteran safety and risk management, and if not, establish appropriate action plans. 12. We recommended that the Under Secretary for Health ensure that Veterans Crisis Line quality management staff incorporate call audio recording into quality management data analysis. 13. We recommended that the Under Secretary for Health ensure that processes are in place to analyze performance and quality data from the Atlanta Call Center separately from the Canandaigua Call Center data. 14. We recommended that the Under Secretary for Health ensure that quality assurance monitoring policies and procedures are in place and consistent for both Social Service Assistants and responders. 15. We recommended that the Under Secretary for Health ensure that supervisors certify Social Service Assistant training prior to engaging in independent assistance with rescues. 16. We recommended that the Under Secretary for Health ensure a process is in place to establish, maintain, distribute, and educate staff on all Veterans Crisis Line policies and directives that includes verifying the use of current versions when policies and directives are modified. VA Office of Inspector General x Evaluation of the Veterans Health Administration Veterans Crisis Line Comments The Acting Under Secretary for Health concurred with our recommendations and provided acceptable action plans. (See Appendix F, pages 57-67 for the Acting Under Secretary comments.) We will follow up on the planned actions until they are completed. JOHN D. DAIGH, JR., M.D. Assistant Inspector General for Healthcare Inspections VA Office of Inspector General xi Evaluation of the Veterans Health Administration Veterans Crisis Line Purpose The VA Office of Inspector General (OIG) conducted a healthcare inspection of the Veterans Health Administration (VHA) Veterans Crisis Line (VCL) in Canandaigua, NY. The inspection had four objectives: 1. To respond to a complaint received by OIG in 2016 regarding a veteran’s contact with the VCL. 2. To perform a detailed review of the VCL that included VCL governance structure, operations, and quality assurance functions in order to assess whether the VCL was effectively serving the needs of veterans. 3. To evaluate whether VHA satisfactorily completed planned actions in response to OIG recommendations published in its February 11, 2016 report titled Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (Report No. 14-03540-123). 4. To address complaints referred to Department of Veterans Affairs (VA) from the U.S. Office of Special Counsel (OSC) alleging inadequate training of VCL Social Service Assistants (SSAs) resulting in deficiencies in coordinating emergency rescue services for veterans in jeopardy. The OSC allegations noted deficiencies in SSA education and oversight. Prior to the initiation of this review, we received a request from Senator Bill Nelson (FL) to evaluate aspects of suicide prevention including the VCL. We address his concerns related to the VCL in this report. Our review of other aspects of suicide prevention will be addressed in future oversight. Background Suicide Suicide is the act of taking one's own life.17 Between 2006 and 2014, yearly suicides in the United States increased from 33,300 to 42,773. On August 3, 2016, VA issued a report discussing its review of over 55 million veteran records from 1979 to 2014.18 A key finding of this report was that “in 2014, an average of 20 veterans a day died from suicide.” The report also compared veteran to civilian suicide rates. It found that since 2001, U.S. adult civilian suicides increased 23 percent, while veteran suicides increased 32 percent in the same timeframe. In a more global context, suicide is a vital American public health issue. Beyond the loss of life to the victim, suicide takes a profound toll on survivors, caregivers, and the 17 Lippincott Williams & Wilkins. Stedman's Medical Dictionary, Twenty-Eighth edition. December 6, 2005. VA Releases Report on Nation’s Largest Analysis of Veteran Suicide, http://www.va.gov/opa/pressrel/pressrelease.cfm?id=2807. Accessed November 2, 2016. 18 VA Office of Inspector General 1 Evaluation of the Veterans Health Administration Veterans Crisis Line community. Likewise, incomplete suicides, taking the form of suicide attempts, gestures, and other acute self-destructive behaviors, are associated with injury, an emotional toll, and personal and societal financial burdens.19,20 Therefore, interventions that might reduce suicidal behaviors are of enormous importance. One such intervention is the "crisis line." The concept of a crisis line is predicated on several principles. As discussed by Robert E. Litman, et. al. over 50 years ago in a seminal article in the Journal of the American Medical Association, these principles include the concepts that:  Suicidal behavior tends to be associated with crises. These are life situations of special stress and limited temporal duration,  People consider suicide with great psychological ambivalence. Wishes to die or be killed exist simultaneously with and in conflict with wishes to be rescued and survive, and  Even in crises, people retain the human need to express themselves and communicate with others.21, A Crisis Line Brief Overview and Key Industry Leaders. The study of suicide and interventions to prevent suicide in the United States took an essential step forward in 1958, when the Suicide Prevention Center (currently known as the Didi Hirsch Suicide Prevention Center22) received funding from the United States Public Health Service and began operations in Los Angeles, California. In 1963, the Suicide Prevention Center pioneered the first suicide crisis hotline that became a prototype for other centers in the U.S. and abroad. In 1968, one of the Suicide Prevention Center’s co-directors founded the American Association of Suicidology (AAS)23, a national organization devoted to research, education, and clinical practice as it relates to preventing suicide. In 1976, the AAS established the first U.S. crisis center certification program. The AAS is the accrediting organization for a majority of U.S. suicide crisis call centers24 and currently accredits 90 of the 155 (58 percent) Lifeline centers. The AAS guidelines do not specify a minimum education or licensure requirement, standard, or recommendation for crisis line responders, supervisory staff, or management. AAS guidelines also make it clear that previous mental health experience or professional 19 According to the CDC, over 494,169 people with self-inflicted injuries were treated in U.S. Emergency Departments in 2013. https://www.cdc.gov/violenceprevention/suicide/. Accessed December 2, 2016. 20 According to the CDC, suicide resulted in an estimated $44.6 billion in combined medical and work loss costs in 2013. https://www.cdc.gov/violenceprevention/suicide/. Accessed December 2, 2016. 21 Litman, Farberow, Shneidman, Heilig, and Kramer. Suicide-prevention telephone service. JAMA, 1965, 192 (1): 21–25. Please note this article is summarizing the work of the Suicide Prevention Center (SPC) now known as Didi Hirsch Suicide Prevention Center. 22 Didi Hirsch Mental Health Services, 4760 S. Sepulveda, Culver City, CA 90230. 23 The AAS is an accrediting organization for crisis lines. The accreditation standards are outlined in the manual titled, American Association of Suicidology, Organization Accreditation Standards Manual, Twelfth Edition, January 2012. 24 NSPL – 2010 Lifeline Policy for Helping Callers at Imminent Risk of Suicide, Background Paper, reported that in 2010, 97 of the 147 (66 percent) Lifeline centers were AAS accredited. The second most predominant accreditation was CUSA (10 percent). In October, 2016, AAS remained the most predominant accreditation with 90 of the 155 (58 percent) Lifeline centers, and CUSA remained the second at 20 of 155 (13 percent). VA Office of Inspector General 2 Evaluation of the Veterans Health Administration Veterans Crisis Line training does “not necessarily qualify a person to work with people who are suicidal or 25 otherwise in crisis.” The VCL is accredited by the AAS. History of the VCL VHA established the VCL in 2007 as a telephone suicide crisis hotline. The VCL was initially located at the Canandaigua, New York VA Medical Center (VAMC). The primary mission of the VCL is “to provide 24/7, world class, suicide prevention and crisis intervention services to veterans, service members, and their family members.”26 However, any person concerned for a veteran’s or military service member’s safety or crisis status may call the VCL.27 VHA established the VCL through an agreement with the U.S. Department of Health and Human Services’ Substance Abuse and Mental Health Services Administration (SAMHSA). This agreement provided for VHA’s use of the already existing National Suicide Prevention Line (NSPL) toll-free number for crisis calls.28 A more detailed timeline of key milestones in the development of the VCL is presented in Appendix A. Since its launch in 2007, through September 2016, VCL staff have answered over 2.5 million calls and initiated the dispatch of emergency services to callers in crisis over 66,000 times. In order to establish greater capacity to respond to calls, the VCL may contract with up to six backup centers; although it currently utilizes only four of these six.29 When the VCL is being utilized to capacity, new, incoming calls will roll over to a backup center after 30 seconds. The responders at the backup centers are not VCL employees. In 2016, the VCL expanded to a second site in Atlanta, GA, in order to ensure geographic coverage and redundancy as well as to add capacity to be able to respond to its burgeoning number of calls. Planning for the Atlanta Call Center started on July 21, 2016, with a goal for full implementation no later than December 31, 2016. As of mid-December, the Atlanta site was operational, but not running at full capacity. Recently, for quality assurance purposes, the VCL implemented audio call recording capability for incoming and outgoing calls. As of December 15, 2016, VCL staff reported technological capability to listen to call recordings at the Canandaigua location. However, VCL staff reported that the Atlanta location did not have call recording capability, but planned to have call recording by the end of December 2016. Neither location had procedures, protocols, or policies that provided guidance for listening to or using call recording information. 25 AAS Standards for Accreditation, pp. 26–33. VCL Mission Statement. 27 VA Suicide Prevention, http://www.mentalhealth.va.gov/suicide_prevention/. Accessed November 8, 2016. 28 The toll-free number is (800) 273-8255. 29 VCL’s current back-up call center contract was established in April 2016. 26 VA Office of Inspector General 3 Evaluation of the Veterans Health Administration Veterans Crisis Line VCL Call Response. To reach the VCL through its toll-free number, a caller is instructed to press 1 (for veterans) on the telephone keypad. If the caller does not press 1, the caller is routed to a NSPL center. The caller still speaks with a responder. However, this route will take the caller to a non-VCL and non-VA contracted backup call center. If the caller presses 1, and the call cannot be answered within 30 seconds, it rolls over to a VA contracted backup center. When a call is answered by VCL staff, a trained crisis responder answers the call, and after engaging with the caller and building rapport, the responder asks about suicidal ideation.30 Depending upon the caller’s answer, the responder may conduct a more detailed assessment of lethality, which addresses a range of both suicide risk factors as well as protective factors. The responder assesses the caller’s risk for suicide, and if present, a safety plan is developed. A safety plan is an individualized list of coping strategies and sources of support to help the caller lower the imminent risk of suicidal behavior. If the responder and caller are unable to create a safety plan, the responder contacts VCL SSAs to initiate a rescue.31 A rescue is an in-person welfare check that utilizes emergency services located near the caller. Callers may choose to remain anonymous during their contact with the VCL. In this scenario, when the need to initiate a rescue arises, the VCL can identify the caller only by telephone number. If the caller has not provided more personal contact information, the SSA utilizes a set of VCL-approved resources to locate the caller, starting with information about the caller’s phone number such as the caller’s area code. A search of this nature is often complex and relies on cooperation and assistance from the caller’s local law enforcement authorities. Rescue Services. VCL responders initiate contact with SSAs via instant messaging. They work closely with SSAs when initiating rescue services. It is the responder’s responsibility to maintain the connection with the caller until emergency services have arrived at the caller’s location. Finally, the responder is responsible for initiating and maintaining documentation of the rescue attempt and its details. The SSAs’ responsibilities include attempting to locate the caller based on information obtained by the responder, communicating with the responder and VCL supervisors, and obtaining assistance from emergency rescue services. SSAs have access to and utilize a database of local emergency resources pertinent to the caller’s location. Follow-up. For callers who are veterans, if a rescue was initiated, an SSA notifies the Suicide Prevention Coordinator (SPC) at the caller’s local VAMC in order to ensure follow-up. Even when a rescue was not initiated, for example, because the responder assessed the veteran as a low risk for suicide, the responder offers the veteran a referral to the local SPC. SPCs follow up with veteran callers to provide assistance and 30 Suicidal ideation is thinking about, considering, or planning suicide. Centers for Disease Control and Prevention, http://www.cdc.gov/violenceprevention/suicide/definitions.html. Accessed December 2, 2016. 31 VCL staff consider rescues, welfare checks, and dispatch of emergency services to be equivalent terms. VA Office of Inspector General 4 Evaluation of the Veterans Health Administration Veterans Crisis Line document the interactions in the veteran’s electronic health record (EHR). SSAs review the SPC’s documentation to ensure that the veteran received follow-up. Veterans are also referred to the closest VA facility with mental health services. Community Resources. VCL staff works closely with community resources to provide 24-hour-a-day access to emergency services, follow-up, and outreach services. In addition to referrals to SPCs at VAMCs, callers are referred for treatment and support by local clinics, Emergency Departments, and agencies. Call Documentation. VCL responders document interactions with the caller using an electronic record system that is distinct and separate from the VHA EHR. This documentation includes a call synopsis that notes the caller’s telephone number (if provided and/or otherwise obtained), other available caller demographic information (for example age, sex, period of military service), the reason for the call, and the caller’s responses to questions about suicide risk. Although some documentation is entered into a pre-established template, the call’s synopsis is written in the responder’s own words, and the responder must enter the caller’s telephone number manually by transcribing it from the caller ID provided by the VCL’s telephone system. Responders and SSAs document SPC referrals and follow-up in the VCL electronic record system. VCL Online Chat and Text Contacts. In 2009, the Canandaigua Call Center added an anonymous online chat service.32 As of September 2016, the Canandaigua Call Center had engaged in more than 308,000 chats. In November 2011, the Canandaigua Call Center introduced a text-messaging33 service to enable callers to connect with VCL staff confidentially. The text messaging service is available around-the-clock for support. As of September 2016, the Canandaigua Call Center has responded to more than 60,000 texts. For online chats, VCL responders interact using keyboarded messages with chat visitors and record basic information, including lethality assessments. Chats cannot be documented with VCL call information because the visitor’s contact does not originate from a phone number. If and when a chat visitor transfers to a telephone line or text messaging for referrals to an SPC, responders may then document the referral as they would for a call. For text messages, VCL responders interact with text visitors using keyboarded messages that originate from a cell phone or computer texting application. These messages are documented with call documentation because the visitor’s phone number is associated with the contact. 32 Online Chat may refer to any kind of communication over the internet that offers real-time transmission of text messages from sender to receiver. 33 A text message is a short keyboarded message sent or received on a mobile device such as a cell phone. VA Office of Inspector General 5 Evaluation of the Veterans Health Administration Veterans Crisis Line Funding. VCL funding comes from the VA budget. The funding for VCL contracts was provided through the appropriations for the Office of Mental Health Operations (OMHO). This program office funding includes the cost of the Chief Mental Health Operations Officer, personnel, and other operating costs. Funding for the contracts was identified specifically for IT support services, which means it can be used for information technology support (both Federal and non-Federal executive branch agency suppliers), including labor, consulting services and programming support for information technology, such as system design, analysis, and performance. It also may include services to assist and advise management on the efficient and effective operation of IT systems. The number of calls to the VCL has increased markedly since the VCL’s first full year of operation in 2007, with a corresponding increase in annual funding obligations for the VCL (see Figures 1 and 2 below.)34 Figure 1. VCL Call Volume and Funding Expenditures, FY 2007–2016 35 500000 30 25 400000 20 300000 15 200000 10 100000 5 0 0 2007 2008 2009 2010 2011 2012 2013 2014 Fiscal Year Total Number of Calls Answered 2015 Annual Expenditures (dollars in millions) Total Number of Calls Answered Call Volume and Expenditures, FY 2007‐2016 600000 2016 Expenditures Source: VHA 34 When a VA program or service initiates a transaction, the funds are considered committed, or obligated. VA Office of Inspector General 6 Evaluation of the Veterans Health Administration Veterans Crisis Line Figure 2. VCL Obligations and Expenditures 2010–2016 (Dollars in Millions) FY 2010 FY 2011 FY 2012 FY 2013 FY 2014 FY 2015 FY 2016 Obligations 9.406 9.812 12.507 20.539 24.664 27.189 31.117 Expenditures 9.401 9.810 12.426 20.382 24.650 26.914 29.255 Source: Chief Financial Officer, VHA Prior Relevant OIG Reports. We previously conducted a review of the VCL related to allegations of calls going to voice mail, inadequate SSA training, quality concerns, and issues related to backup centers. In our report of that review, Healthcare Inspection– Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (Report No. 14-03540-123, February 11, 2016), we made seven recommendations. We recommended that the OMHO (now Member Services) Executive Director:  Ensure that issues regarding response hold times when callers are routed to backup crisis centers are addressed and that data is collected, analyzed, tracked, and trended on an ongoing basis to identify system issues.  Ensure that orientation and ongoing training for all VCL staff is completed and documented.  Ensure that silent monitoring frequency meets the VCL and AAS requirement35 and that compliance is monitored.  Establish a formal quality assurance process, as required by VHA, to identify system issues by collecting, analyzing, tracking, and trending data from the VCL routing system and backup centers and that subsequent actions are implemented and tracked to resolution.  Consider the development of a VHA directive or handbook for the VCL.  Ensure that contractual arrangements concerning the VCL include specific language regarding training compliance, supervision, comprehensiveness of information provided in contact and disposition emails, and quality assurance tasks.  Consider the development of algorithms or progressive situation-specific stepwise processes to provide guidance in the rescue process. In response to these recommendations for improvement, VHA agreed to a completion deadline of September 30, 2016. Nevertheless, all seven recommendations remain open as of the publication of this report. 35 The AAS accreditation manual does not specify a specific method for crisis worker monitoring and states: “It is the responsibility of the organization to define its procedures for measuring the quality and show proof that it is using those procedures.” In the absence of an AAS requirement for silent monitoring, we referred to the VCL’s QM manual. VA Office of Inspector General 7 Evaluation of the Veterans Health Administration Veterans Crisis Line Recent Legislation. On November 28, 2016, President Obama signed into law the No Veterans Crisis Line Should Go Unanswered Act.36 This law requires VA to develop a quality assurance document to improve VCL functions. The document will outline clearly defined and measurable performance indicators and quantifiable timeframes. It is to be submitted to the Committees on Veterans’ Affairs of the House of Representatives and the Senate no later than 180 days after the date of enactment of the Act. The Act is also intended to ensure that all incoming communications received by the VCL and backup centers be answered in a timely manner by a person. Allegations. We received a complaint in 2016 regarding a veteran’s contact with the VCL. We immediately initiated a review of this case. Two months after we initiated our review we received complaints referred from the OSC alleging inadequate training of VCL SSAs that resulted in deficiencies in coordinating immediate emergency rescue services needed to prevent harm to callers in jeopardy. The OSC allegations noted deficiencies in SSA education and oversight. Allegations related to education deficiencies specified that SSAs:  Did not receive appropriate instruction on available systems, procedures, and policies used to facilitate emergency rescue responses.  Did not receive thorough training on available VCL resources to locate veterans for rescues.  Did not receive instruction on how to properly document call information, which resulted in significant factual discrepancies in VCL call records.  Did not receive training on how to use VA’s EHR to track suicide prevention referrals.  Did not receive proper instruction on coordinating suicide prevention follow-up with veterans through local VAMCs.  Had incorrect and outdated information and standard operating procedures in their training program undergoing development. The oversight concerns noted that SSAs:  Who, without supervision and regardless of performance or evaluation, were allowed to coordinate emergency rescue responses independently after the end of a 2-week training period.  Who had not completed training, were allowed to train newly hired SSAs. 36 H.R. 5392-No Veterans Crisis Line Call Should Go Unanswered Act. https://www.congress.gov/bill/114thcongress/house-bill/5392/text?format=txt. Accessed on December 2, 2016. VA Office of Inspector General 8 Evaluation of the Veterans Health Administration Veterans Crisis Line  Who worked different shifts, handled and recorded emergency rescue responses differently, which made quality assurance efforts difficult. Prior to the initiation of this review, we received a request from Senator Bill Nelson (FL) to evaluate aspects of suicide prevention including the VCL. We address his concerns related to the VCL in this report. Our review of other aspects of suicide prevention will be addressed in future oversight. Scope and Methodology We initiated our review in June 6, 2016 and completed work on December 15, 2016. Unless noted otherwise, information in this report was current as of December 15, 2016. We conducted site visits to Canandaigua VAMC on June 8–9, August 9–11, and September 6–9, 2016, and to the Atlanta Call Center on September 20–21, 2016. We requested, received, and reviewed hundreds of documents from various sources and interviewed 64 individuals who had experience with the VCL. We conducted interviews by telephone and in person. Select individuals were interviewed multiple times, for a total of 94 interviews. We interviewed:  The complainant.  The Deputy Secretary, U.S. Department of Veterans Affairs.  Current and former VCL leadership and staff.  VHA staff, including staff from VHA Member Services (Member Services); Office of Suicide Prevention (OSP); Mental Illness Research, Education and Clinical Centers (MIRECC); Health Eligibility Center (HEC); Office of Finance; and Office of Acquisition Operations.  The vendor for the contracted backup centers.  GAO report authors of Veterans Crisis Line: Additional Testing, Monitoring, and Information Needed to Ensure Better Quality Service, GAO-16-373, May 26, 2016.  Staff involved in care of the subject veteran at his/her local VAMC.  Staff members of Didi Hirsch Mental Health Services, Los Angeles, CA; Boys Town and the Boys Town National Hotline®, Omaha, NE; the National Domestic Violence Hotline, Austin, TX; and National 911 Emergency Response. For our case review, the veteran declined a formal interview, but provided some relevant information. More specifically, we interviewed VCL responders who had direct involvement in the veteran’s contact with the VCL, as well as other VCL staff and leaders. We reviewed internal VCL call documentation and call records from relevant backup centers. We listened to the only recorded call of the veteran, which was from a backup center, and facilitated a review of the recorded call by an OIG healthcare VA Office of Inspector General 9 Evaluation of the Veterans Health Administration Veterans Crisis Line inspector with a suicide prevention background. We reviewed the veteran’s EHR and VCL complaint documentation. Additionally, we reviewed internal VCL documents related to the VCL’s case analysis of the veteran’s contacts and VCL reporting to VHA leaders. We acquired and reviewed a relevant police report. To evaluate the VCL program, we reviewed the AAS Accreditation Standards Manual.37 We reviewed training records of VCL staff and observed responders and SSAs functioning in their roles. We reviewed relevant VHA and VCL policies, procedures, handbooks, and manuals, including the VCL Strategic Plan and a draft version of a VHA Directive Operations of the Veteran Crisis Line.38,39 We reviewed VCL quality management data. We reviewed documents relevant to VCL committee oversight, including the VCL Clinical Advisory Board Charter and minutes from Clinical Advisory Board and Executive Leadership Council meetings. We reviewed previously published reports including those from GAO40 and OIG;41 and we reviewed documents and data submitted by VHA regarding recommendations published on February 11, 2016, in our report titled Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (Report No. 14-03540-123). We reviewed VCL funding and expenditures from 2009–2016.42 We reviewed the contract with a vendor regarding the services provided by VCL’s backup crisis call centers. We reviewed recommended practices published by industry leaders including the AAS, Contact USA (CUSA),43 and the NSPL.44 Two policies cited in this report were expired or beyond the recertification date: 1. VHA Directive 2010-025, Peer Review for Quality Management, June 3, 2010 (expired June 30, 2015). 2. VHA Handbook 1050.01, VHA National Patient Safety Improvement Handbook, March 4, 2011 (recertification due date March 31, 2016). 37 American Association of Suicidology, Organization Accreditation Standards Manual, Twelfth Edition, January 2012. 38 VHA Draft Directive, Operations of the Veterans Crisis Line, April 4, 2016. 39 Veterans Crisis Line Strategic Plan FY 2017–2022. 40 United States Government Accountability Office, report to Congressional Requesters, Veterans Crisis Line Additional Testing Monitoring and Information Needed to Ensure Better Quality Service, GAO-16-373, May 26, 2016. 41 VA OIG, Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York, (Report No. 14-03540-123, February 11, 2016). 42 OIG requested obligations and expenditures for 2007-2016. We received funding information beginning in 2009. 43 Contact USA is an accrediting organization with established standards for crisis call centers and online emotional support. 44 The National Suicide Prevention Lifeline is a national network of local crisis centers that provides free and confidential emotional support to people in suicidal crisis or emotional distress 24 hours a day, 7 days a week. VA Office of Inspector General 10 Evaluation of the Veterans Health Administration Veterans Crisis Line We considered these policies to be in effect as they had not been superseded by more recent policy or guidance. In a June 29, 2016 memorandum to supplement policy provided by VHA Directive 6330(1),45 the VA Under Secretary for Health (USH) mandated the “…continued use of and adherence to VHA policy documents beyond their recertification date until the policy is rescinded, recertified, or superseded by a more recent policy or guidance.”46 The USH also tasked the Principal Deputy Under Secretary for Health and Deputy Under Secretaries for Health with ensuring “…the timely rescission or recertification of policy documents over which their program offices have primary responsibility.”47 We substantiate allegations when the facts and findings support that the alleged events or actions took place. We do not substantiate allegations when the facts show the allegations are unfounded. We cannot substantiate allegations when there is no conclusive evidence to either sustain or refute the allegation. We conducted the inspection in accordance with Quality Standards for Inspection and Evaluation published by the Council of the Inspectors General on Integrity and Efficiency. Inspection Results Objective 1. VCL Failure to Adequately Respond to a Veteran Caller We substantiated that VCL staff did not respond adequately to a veteran caller’s urgent needs. The veteran’s encounters with the VCL and its backup call centers exposed procedural and clinical concerns with VCL operations. Due to privacy concerns and to avoid discussion of protected information, we are unable to discuss further the veteran’s interaction with the VCL. However, in the course of our inspection of the care of a veteran, we identified systems issues requiring further review. Issues identified included:  Manual writing of VCL caller telephone numbers that allowed for error and led to gaps in caller service.  A lack of formal processes to review adverse outcomes.  An inability of the VCL staff to provide evidence that the backup centers provided the high level of service delineated in the contractual agreement. 45 VHA Directive 6330 (1), Controlled National Policy/Directives Management System, June 24, 2016, amended January 11, 2017. The amended Directive contains the same or similar language outlining controlled national policy. 46 VA Under Secretary for Health Memorandum. Validity of VHA Policy Document, June 29, 2016. 47 Ibid. VA Office of Inspector General 11 Evaluation of the Veterans Health Administration Veterans Crisis Line  An inability to record calls (at the VCL, unlike some backup centers) hindering real time quality assurance efforts. Objective 2: VCL Governance Structure, Operations, and Quality Assurance Functions We conducted a detailed review of the VCL governance structure, operations, and quality assurance functions to assess whether the VCL was effectively serving the needs of veterans. We reviewed the VCL mission statement and the leadership and organizational structure in place to meet the mission. We reviewed current performance measurement, the structure of the clinical and administrative functions, and the performance of contracted backup centers as extensions of VCL services. We reviewed the expansion of the VCL to a new location (Atlanta, GA). Finally, we reviewed VHA leadership’s approach to crisis intervention services for veterans, including policies, procedures, staffing, training, and supervision. A.  VCL Governance Structure Governance, Leadership, and Change of VCL Oversight. Governance is defined as the establishment of policies, and the continuous monitoring of their proper implementation, by members of the governing body of an organization.48 The VCL’s governing body and leadership changed over time due to realignments within VHA. From 2007 through early 2016, VCL underwent a variety of organizational alignments under the auspices of MHS, and then OMHO, with significant leadership oversight provided from the VA Suicide Prevention Program (now designated as the Office of Suicide Prevention (OSP). The OSP leads suicide prevention efforts for VHA and coordinates and disseminates evidence-based findings related to suicide prevention.49 The OSP leadership currently interfaces between facility suicide prevention coordinators and the VCL.50 During 2015, the VCL was without a permanent director for 10 months. At the end of 2015, a permanent director was selected. In February 2016, the VCL was realigned under VHA Member Services (Member Services), which provided oversight for other VA call centers. The new permanent director of the VCL resigned his position in June 2016. 48 Business Dictionary’s Definition of Governance. Accessed November 10, 2016. Email communication provided through the Director of the OSP: “The VA Office of Suicide Prevention uses a comprehensive, integrated public health approach to serve all 22 million of our Nation's Veterans. Through education, research, enhanced care, and a system of national and community-based partnerships, the office ensures that all Veterans at risk for suicide have access to high-quality resources and support throughout their lives.” 50 Each VA Medical Center has a suicide prevention coordinator to make sure veterans receive needed counseling and services. Calls from the NSPL are referred to these coordinators. http://vaww.mentalhealth.va.gov/rcsuicideprevention.asp. Accessed on December 23, 2016. 49 VA Office of Inspector General 12 Evaluation of the Veterans Health Administration Veterans Crisis Line By email on February 19, 2016, then-VA Deputy Secretary Sloan Gibson announced the VCL realignment under Member Services and stated that it was effective immediately. The Deputy Secretary stated that the Acting Director of Member Services: …has a track record in providing solutions-oriented approaches to resolve complex organizational issues. Additionally, VCL will benefit from the experience within Member Services providing contact center support and operations. Similar to the transition of the National Call Center for Homeless Veterans and the relationship with the Homeless Program Office, VCL will remain closely tethered to the Office of Mental Health Operations and VHA Clinical Operations. The organizational realignment will support the Veterans Crisis Line and expedite enhancements to create a reliable, robust, and resilient service for veterans and family members. A former VCL leader told us that several factors influenced the decision to realign the VCL structurally under Member Services instead of OMHO. These factors included media reports of caller dissatisfaction with the VCL’s services, a critical OIG report, and the VA Deputy Secretary’s dissatisfaction with OMHO’s progress regarding service improvements. The VCL leadership’s stated objective for the reorganization was to establish a “world class crisis line” to serve our nation’s veterans and active duty military. The new organizational alignment emphasized the role of Member Services providing the VCL with business leadership. Member Services staff had substantial expertise in the areas of call center infrastructure and personnel management. The latest realignment represented a move away from a series of alignments directed towards OMHO providing clinical oversight and programming. Under the current system, the VCL provides services using business practices, but the services are still defined by clinical endpoints such as suicides, either attempted or completed. Following realignment, Member Services leadership focused on the Deputy Secretary’s priority of achieving zero rollover calls, by hiring staff and opening a second VCL call center.51 Member Services leadership also focused on remediation of deficiencies identified in our report, Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (Report No. 14-03540-123, February 11, 2016) and a GAO report, Veterans Crisis Line: Additional Testing, Monitoring, and Information Needed to Ensure Better Quality Service (GAO-16-373, May 26, 2016).52 A number of VCL staff expressed concern to us that under Member Services leadership the emphasis is on business and efficiency metrics with less emphasis on clinical quality measurement. Examples of data currently analyzed include veteran complaint data, 51 Zero rollover refers to the capacity of the VCL to handle all veteran calls without having calls roll over to a backup (non-VA) crisis line. 52 Government Accountability Office. Veterans Crisis Line: Additional Testing, Monitoring, and Information Needed to Ensure Better Quality Service (GAO-16-373, May 26, 2016). VA Office of Inspector General 13 Evaluation of the Veterans Health Administration Veterans Crisis Line veteran satisfaction data, numbers of calls, and time spent in calls. Silent monitoring53 of VCL call responders is the only real-time method of measuring the performance of VCL staff against critical factors deemed to constitute a successful call. Silent monitoring is performed on approximately one percent of the calls received. While the stated objective in the VCL’s mission statement is to establish a world-class crisis line, a method of measuring outcomes specific to the actions of VCL staff or leadership has not been established., During an interview, the leadership of OSP expressed concern about the lack of utilization of clinical advice and guidance. The Director of OSP stated that the Acting Director of Member Services informed her that she had no authority over the VCL. The OSP Director further stated that Member Services leadership is making clinical decisions without understanding the impact on veterans. One interviewee stated that the Acting Director of Member Services unilaterally disallowed a legitimate intervention to deescalate behavior that allowed for delaying calls from high frequency abusive callers,54 despite a clear expectation outlined in the Deputy Secretary memorandum of February 2016 that the new alignment would create a close partnership between clinical and administrative leaders. The original charter for the Clinical Advisory Board outlined a prominent role for the OSP, but that was later removed when the charter for the CAB was signed and published. Current and former clinical VCL managers shared the perspective of clinical decisions being made in a business model rather than a clinical model; however, Member Services leadership dismissed their concerns. The VCL’s Acting Director told us that he is using the Baldridge55 framework for governance. This includes five bodies: partnership, employee experience, quality, veteran experience, and business operations. For the VCL, the central leadership group in this model would be the Executive Leadership Council (ELC).56 The ELC integrates the business and clinical aspects of operating the VCL. The ELC had its first meeting on September 14, 2016, and outlined its purpose and goals as follows:  Provide a strong foundation for accountability 53 Silent monitoring is a process for listening to and evaluating a VCL call responder’s interactions with veterans and coaching them on areas of strength and areas for improvement. Monitoring is “silent” in that the responder is unaware that the call is being monitored. Calls are monitored in real time and feedback is provided immediately following the responder’s call documentation. VCL managers have identified critical and non-critical elements of call quality for evaluation; for example, Greeting, Rapport Building, Safety Planning, and Suicide Risk Assessment. 54 The Deputy Secretary later reversed the decision by the Acting Director of Member Services. 55 The Malcolm Baldrige National Quality Award is the highest level of national recognition for performance excellence that a U.S. organization can receive. The award focuses on performance in five key areas: product and process outcomes, customer outcomes, workforce outcomes, leadership and governance outcomes, financial and market outcomes. https://www.nist.gov/baldrige/baldrige-award. Accessed December 23, 2016. 56 ELC membership includes VCL Director, Chairperson, VCL Deputy Director, Business Operations Lead, Veteran Experience Lead, Employee Experience Lead, Partnerships Lead, Clinical Quality Lead, AFGE Leadership Member, Union Leadership Member, Clinical Psychologist, and CAC. VA Office of Inspector General 14 Evaluation of the Veterans Health Administration Veterans Crisis Line  Eliminate working in crisis mode  Minimize the variation in the way information is reported  Provide evidence of data-driven decision-making  Ensure leadership awareness of performance  Track issues to completion  Transform data into useable information that is actionable  Progress to a culture of constant improvement. We requested all ELC draft policies to ensure that the ELC had a process for achieving its intended goals. We were informed that no current policies related to the ELC existed and that creation of such was in progress. VHA does not have a policy on the operations of the VCL. In April 2016, VHA leadership drafted a policy that has not yet been finalized and formally issued. The VCL and the services it provides have grown considerably since 2007, but VCL leadership did not develop a plan that defined the strategic approach for the VCL to provide consistent, timely, and high quality suicide prevention services until 2016. For its Baldridge Framework goals, VCL leadership was unable to provide policies, dashboards, or quality monitors for this governance initiative. The VCL Acting Director informed us that leadership is working on this initiative. Clinical Leadership Through the Clinical Advisory Board. In March 2016, the Member Services Acting Director requested the formation of a Clinical Advisory Board (CAB) for the VCL, comprised of key stakeholders, to foster collaboration among capable experts and to leverage collective expertise thereby facilitating an improved veteran experience, efficiencies, and increased access to the VCL. We were informed that a draft charter for a clinical advisory board (CAB) allowed for input from an outside clinical group to advise, recommend, authorize, and approve clinical changes to the program. We found that Member Services leadership changed the CAB charter to a consultative body. Call data is presented at CAB meetings, but the CAB does not have a plan that specifies how the data will be used to improve the services provided by the VCL. The CAB has a one-hour call once a month. We learned of concerns that the CAB is not effectively used to obtain clinical input for policy decisions. Our review of the CAB meeting minutes from May 2016–August 2016 found low attendance, minimal discussion of performance information, and a lack of clarity about follow-up of issues previously discussed. The Member Services Acting Director is a member of the CAB. However, we found no documented evidence that he attended CAB meetings. Conclusions. Leadership, governance, and policies are all necessary to run the VCL program in an orderly and effective manner. Both administrative and clinical leaders are vital to running a crisis line. However, the lack of clinical decision makers at the highest level of VCL governance may impair the ability of Member Services to optimally run the VCL program. If business metrics prove that answering VCL calls under the Baldrige framework of governance reduces the numbers of attempted and completed suicides, the business model may be considered successful. However, if the numbers of attempted and completed suicides do not improve under the business model, and VA Office of Inspector General 15 Evaluation of the Veterans Health Administration Veterans Crisis Line clinical aspects of the VCL are not effectively analyzed, VHA may miss opportunities to decrease suicide rates through systematic evaluation of the clinical effectiveness of VCL services. The Deputy Secretary of VA set an expectation that Member Services leadership and the subject matter experts on suicide prevention work closely together, but there is substantial disagreement about key decisions between these two groups at this time. Administrative staff made decisions that have clinical implications. Examples include disagreements about the scope of services associated with core versus non-core calls57 and the selection of training staff who do not have clinical backgrounds. The intention of the realignment of VCL under Member Services was to create a state-of-the-art VCL, by coupling the business expertise of Member Services in call center technology, infrastructure, and customer service with the clinical expertise of OSP. The conflicted and undefined relationship between the OSP (subject matter experts on suicide prevention) and the Member Services (subject matter experts on management of traditional call service centers) hindered achieving that goal. We found substantial disagreement between the view of Member Services and the view of OSP and VCL clinical leaders regarding the partnership. Specifically, OSP and VCL clinical staff felt marginalized concerning decision-making with clinical implications. While clinical leadership and staff stated they felt excluded from decisions, administrative leadership in Member Services believed that the clinical leadership was represented and that the CAB was the appropriate structure for integrating clinical input. We found that the CAB was established in March 2016 and was limited to advisory functions. VHA should develop guidelines that delineate clinical and administrative decision-making, assuring that clinical staff, in accordance with sound clinical practice, make decisions directly affecting clinical care of veterans. B.  VCL Operations Organizational Structure. The VCL experienced organizational changes during the course of our review. We reviewed three versions of the VCL organizational chart dated between June and September 2016. Staff. As of September 2016, VCL leaders reported their full-time equivalent employee (FTE) total was 687. This included administrative staff, responders, trainers, and quality management personnel. The organizational chart indicated that 279 responders and an additional 28 Crisis Intervention Specialists (responders hired under a different authority with a lesser requirement for education) were designated for Canandaigua and 200 responders were designated for the Atlanta Call Center. 57 Core calls are calls defined as calls resulting in referral to SPC and/or calls requiring the application of crisis management skills (example: a suicidal caller). Non-core calls are defined as those that do not require specific crisis intervention skills (example: a caller inquiring about benefits). VA Office of Inspector General 16 Evaluation of the Veterans Health Administration Veterans Crisis Line Full-Time Equivalent Employees. VCL leaders contracted with PipkinsSM 58 to create a staffing model to forecast VCL staffing levels. PipkinsSM utilized the VCL hourly call volume rate to estimate the number of staff needed, and considered duration of training as a factor in its staffing model formula. The staffing model for the VCL was based on assumptions of zero percent rollover,59, zero percent abandoned calls,60 historical call volume rate,61 average call handling times (which cannot be duplicated),62 and a 36 percent staff shrinkage rate.63,64 In order to maintain dynamic staffing forecasts, the model was updated if changes were noted in call volumes. The original VCL staffing model was based on a service level of zero percent rollover, answering all calls within 5 seconds, and forecasting call volume based on historical interval data.65 The result of this process established the staffing requirement for each 15-minute time interval, thereby ensuring that responders were ready and waiting for a call. This model utilized the highest peak volume for each interval66 for staff forecasting, rather than the average volume. The employee turnover rate for the VCL from January 2016 through October 2016 was eight percent. The turnover rate was based on resignations, retirement, loss to another station, and/or termination. The staffing model established staffing requirements for both call centers, Canandaigua and Atlanta. Backup Center Call Handling and Contract Oversight The process that directs callers to the VCL through the toll-free NSPL telephone number requires a mechanism for routing the call. VHA contracted with L2HS, an incorporated subsidiary of the non-profit Mental Health Association of New York City, to provide a platform for routing telephone calls from the NSPL toll-free number to the VCL. To assure continuity of service when VCL call demand exceeded capacity, VHA authorized up to six call centers to provide backup services for telephone contacts; 67 however, VCL currently only uses four. None of the backup centers provides chat or 58 PipkinsSM is a supplier of workforce management software and services to the contact center industry, providing forecasting and scheduling technology. http://www.pipkins.com/. Accessed November 16, 2016. 59 Telephone line rollover is the method used to allow more than one caller to get through by routing each caller through the next available phone line. 60 An abandoned call is a call or other type of contact initiated to a call center that ends before a conversation occurs. 61 The call volume rate is the number of calls with response times. It is used for forecasting and adjusting staffing needs when call volumes are higher or lower on any certain day or time. 62 Average handle time (AHT) is a call center metric for the average duration of one transaction, typically measured from the customer's initiation of the call, and includes any hold time, talk time, and related tasks that follow the transaction. 63 Shrinkage is a measure of how much staff time is lost in the call center to things such as vacation, breaks, lunch, holidays, sick time, and training. 64 VHA Draft Directive, Operations of the Veterans Crisis Line, April 4, 2016. 65 The staffing model has lower tolerances than the actual call response parameters. 66 Peak volume is a spike in calls during various hours of the day. 67 The L2HS contract covers all backup centers. VA Office of Inspector General 17 Evaluation of the Veterans Health Administration Veterans Crisis Line text services, as this is not a requirement in the L2HS contract. L2HS, in addition to routing calls to the VCL, also provides services to forward unanswered VCL calls to the backup centers. Incoming VCL calls are routed to the backup centers if calls are not answered after 30 seconds. Callers are not aware whether their call is answered at the VCL or one of the contracted backup centers. To track and manage incoming calls, the VCL uses an internal software system, Avaya Call Management System.™68 L2HS manages the collection of data regarding VCL-originated calls subsequently rolled over to backup call centers, in accordance with the terms of a contract executed between L2HS and the VHA. VCL and Backup Center Call Volume. When the VCL program started in 2007, management initially projected that approximately 10 percent of calls would be rolled over to a backup center.69 In fact, call rollover to backup centers has increased above initial projections (see Table 1). Table 1. VCL and Backup Center Monthly Call Totals, Quarters 3 and 4, FY 2016 Month Calls Inbound VCL Answered VCL Rollover VCL Rollover Rate April 2016 43,327 27,723 14,150 32.7% May 2016 47,223 31,643 13,695 29.0% June 2016 48,175 32,990 13,204 27.4% July 2016 46,442 33,754 10,995 23.7% August 2016 48,725 36,385 10,364 21.3% September 2016 48,601 32,777 13,840 28.5% October 2016 51,393 28,800 (Canandaigua) 2,203 (Atlanta) 17,955 34.9% November 2016 51,067 26,180 (Canandaigua) 8,145 (Atlanta) 14,638 29.9% Total 384,953 260,600 108,841 28.4%* Source: VCL staff retrieved the data from the Suicide Prevention Lifeline website and provided the data to us on 12/21/2016. VHA relies on this data to determine VCL rollover rate. *This value is the average rollover rate. The values in the table do not include disposition of all inbound calls. Some inbound calls result in abandonment. 68 The Avaya Call Management System provides a platform for monitoring and collecting historical data for call centers. https://www.avaya.com/en/documents/avaya-call-management-system-cc7349.pdf. Accessed November 16, 2016. 69 VAOIG. Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, NY (Report No. 14-03540-123, February 11, 2016). VA Office of Inspector General 18 Evaluation of the Veterans Health Administration Veterans Crisis Line Backup Center Performance Requirements. VHA contracted with L2HS to oversee backup center performance and report to the VCL, with administrative and clinical oversight of this process to be performed by VCL managers. Quality Performance Requirements. The L2HS contract prohibited backup centers from using voice-answering systems or placing VCL overflow calls on hold after responders answered the call. However, backup call centers historically have placed VCL rollover calls into a queue without immediately providing service or risk assessment. A queued call is one that has been routed to a call center, but not yet answered. During this time, the caller waits for a responder to answer. The caller’s only option is to abandon the call (hang up) and call back, or continue to wait for the responder to pick up. Once a call is routed to a backup center and the call is answered, the responder is required to engage the VCL caller, and conduct and document a suicide risk assessment.70 Two of the four backup centers used a call answering system that placed VCL rollover calls in a queue. The queue retained the incoming call indefinitely until a responder answered it; however, the call answering system did not re-route the call to another backup center if no responder was available. The backup centers had processes to record caller wait times and call abandonment rates. The contract required that each backup center using an automated system that routed incoming calls provide to the VCL (through L2HS) weekly reports regarding the backup center’s performance. Of the four backup centers supporting the VCL, we found that three had such a routing system, thus requiring the weekly reports. L2HS created a secure website for providing the required call data to the VCL. Weekly reports were to include the number of calls presented, calls answered, average wait time in minutes, average talk time in minutes, percentage of calls answered, the number of calls abandoned after more than 120 seconds, the percentage of calls abandoned, the average time elapsed in minutes for abandoned calls, the longest wait time for an answered call, and the longest wait time for an abandoned call. VCL staff described that queuing a call before a responder answered was not the same process as placing a call on hold after a responder answered. We found that VCL leadership had not established expectations or targets for queued call times or thresholds for taking action on queue times. A veteran could be queued for 30 minutes, for example, and that wait time might not be reflected in hold time data; however, the result of the delay is the same, whether the veteran was in a queue or on hold. In addition to weekly performance reports, backup center responders are required to use reasonable efforts to send call documentation to the VCL via encrypted email within 70 NSPL staff developed Suicide Risk Standards at Link2Health Solutions, Inc. in collaboration with the NSPL Certification and Training Subcommittee. Suicide Risk Standards are recommendations for an approach to asking Lifeline callers about suicidality. VA Office of Inspector General 19 Evaluation of the Veterans Health Administration Veterans Crisis Line 30 minutes of a call’s end. We were told that VCL managers designated staff on each shift to review call documentation from the backup centers. The contract required L2HS to ensure that backup centers provided sound clinical support to VCL callers by facilitating monthly calls with VCL staff to discuss quality assurance (QA), clinical issues, and questions from backup centers. L2HS was responsible for submitting a meeting agenda and meeting minutes to the VCL Director. The VCL and L2HS staff relied on the monthly calls to communicate about VCL specific issues and for the VCL staff to provide training to backup centers. However, we found that a QA process was not in place or underway to measure the effectiveness of the calls. For example, no tracking and trending of data or designation of staff responsible for follow up was available in the official contract files. We found that Member Services and VCL leaders, L2HS officials, and others responsible for the L2HS contract did not ensure the fulfillment of all QA requirements. On November 16, 2016, the Senate passed the No Veteran Crisis Line Call Should Go Unanswered Act, (PL-114-247), which was signed by President Obama on November, 28, 2016. The Act directs the development of a quality assurance document for “carrying out” the VCL, which shall: (1) outline clearly defined and measurable performance indicators and objectives to improve the performance of the [VCL], including at backup call centers; (2) include quantifiable timeframes to meet designated objectives to assist the [VA] Secretary in tracking the progress of the [VCL] and such backup call centers in meeting the performance indicators and objectives; and (3) with respect to such timeframes and objectives, be consistent with guidance issued by the Office of Management and Budget. The Act requires that VA develop a plan to ensure that all telephone calls, text messages, and other communications received by the VCL, or designated backup call centers, be answered in a timely manner by a person, consistent with the guidance established by the American Association of Suicidology. Any new contract solicitations or contract options for backup call centers will need to reflect the requirements of this new legislation. VCL Oversight of Backup Center Performance Backup Center QA. We found that VHA included a detailed Quality Assurance Surveillance Plan (QASP) in the contract requirements for L2HS. The purpose of the QASP is to describe the methods to be used to monitor and document contractor performance and to identify the resources to be allocated to these tasks. This QASP defines the roles and responsibilities of all members of the integrated project team, identifies the performance objectives, defines the methodologies used to monitor and evaluate the contractor’s performance, describes quality assurance documentation requirements, and describes the analysis of quality assurance monitoring results. The QASP provides a means for VHA to ensure that the required performance standards or service levels are achieved by the contractor. The QASP is an important tool in evaluating whether the contractor is meeting the performance standards/quality VA Office of Inspector General 20 Evaluation of the Veterans Health Administration Veterans Crisis Line levels identified in the performance work statement and in the contractor’s quality control plan. VCL managers told us that the QASP was excluded from the current contract due to cost. We obtained from the Electronic Contract Management System (eCMS),71 a copy of the current L2HS contract that included a QASP, and we provided it to VCL managers, who were unaware that the current contract contained a QASP. After we identified the existence of a current QASP and confirmed it with VHA leadership, VHA managers stated that the QASP in the current contract was not actionable. The VCL staff was unaware that the QASP existed and therefore did not monitor standards delineated in the QASP. Upon a review of the QASP, we found, and Member Services staff agreed, that the QASP lacked appropriate enforcement provisions (for example, incremental penalties for non-performance). We determined that the responses by VHA, Member Services, and VCL staff indicated a fundamental misunderstanding of contract administration. The contractor is responsible for adhering to the quality control program, which ensures that the contractor’s work complies with the requirements of the contract. The VCL Contracting Officer’s Representative (COR) is the primary person responsible for monitoring the contractor’s quality control program, and the QASP is the method used by the COR to monitor it. While reviewing the L2HS File Transfer Protocol site,72 we also found that the required weekly backup center performance data was not routinely provided by L2HS or reviewed by VCL staff. L2HS started providing VCL staff with weekly reports on August 4, 2016. The L2HS staff told us that they had been collecting the data, but the delay in providing the reports occurred because they were waiting for VCL staff to tell them where they wanted the reports stored. VCL staff told us that the only performance metric with an established goal was the rate of calls abandoned after 120 seconds, with a goal of 15 percent or less. However, VCL staff had not established a formal process for reviewing data pertaining to this metric.. One backup center had an average call abandonment rate of approximately 30 percent from April 1 through September 1, 2016. Performance data is vital to the VCL ensuring the contracted services are provided in accordance with contract requirements. Review of the data is necessary to validate the quantity and the services provided on the monthly invoices, prior to the VCL authorizing payment of the invoices. 71 VA's Electronic Contract Management System (eCMS) Vendor Portal application allows industry partners to access VA procurement actions, provide responses and receive contract awards/orders, provide acknowledgement/shipping information, and submit invoices via the World Wide Web (www). 72 The File Transfer Protocol site is a secure electronic holding space where the L2HS data is uploaded for VCL staff review. VA Office of Inspector General 21 Evaluation of the Veterans Health Administration Veterans Crisis Line Administrative Oversight of Performance Requirements. The VCL COR is responsible for monitoring L2HS compliance with the contract requirements, including management of quality control for all work accomplished during the performance of the contract, with work and documentation produced by L2HS subject to regular review by the COR. The VCL’s current contract with L2HS defines the COR’s responsibilities, which include the COR’s preparing a quarterly written report summarizing the results of the quality assurance surveillance of the contractor’s performance. Such a report would enable VHA and VCL staff to confirm whether the contractor met performance requirements. The quarterly reports were not being completed as required. Quality control data provided by L2HS were not fully reviewed, and there was no validation that L2HS provided the required services, before payment of contract invoices was authorized. The contract requires L2HS facilitation to include meeting agendas and minutes of monthly supervision calls with backup centers and VCL staff to follow-up on activities and to discuss clinical questions. This requirement ensures that backup centers are providing clinically sound support to VCL callers. We learned that key VCL staff were not included in the monthly L2HS-facilitated meeting. We reviewed the monthly L2HS meeting minutes from February to August 2016 and found that meeting minutes were not recorded between March and June 2016. VCL staff provided agendas for these months in lieu of minutes, stating they met but did not keep documentation of their meetings. We learned that the fact that monthly meeting minutes were missing was not discussed by VCL staff with the VA’s Contracting Officer (CO). Although the contract included requirements such as mandatory training, a QASP, and a requirement for quality control, we found no documented evidence of oversight or follow through by appropriate staff or other VCL program management personnel. In July 2016, Member Services managers tasked an experienced program analyst to assist the COR. The program analyst told us that he/she realized after several months of working with the COR, the CO, and the L2HS program manager, that the next contract required revision. The program analyst felt the revision should include detailed performance data for each backup call center and a more robust QASP, because the current QASP included in the contract is not specific enough and does not include items such as evaluation criteria, percentages of accuracy required, or incentives (negative or positive) that would be implemented if performance measures were not met. The program analyst planned to volunteer to become the COR of the L2HS contract because he/she believed that it needed additional detail. Contract Costs from 2007–2016. We requested the itemized invoices and/or the purchase order documents associated with the backup center contracts. However, VCL staff was unable to provide the requested documents for the period before 2013. The table below represents money spent on backup contracts. The total value of these contracts is approximately 15 million dollars. VA Office of Inspector General 22 Evaluation of the Veterans Health Administration Veterans Crisis Line Table 2. Backup Center Contract Costs from FPDS73 2007–2016 Contract Start Date End Date Dollar Value VA101049AP2035 08/27/2007 09/30/2012 $3,604,885.75 VA798512C0011 10/01/2012 03/31/2016 $8,529,643.23 VA11916C0025 04/01/2016 12/31/2016 $2,823,752.00 Total Spent $14,958,280.98 Source: Federal Procurement Data System Atlanta Call Center A component of the VCL’s long-term continuing operations plan was to expand beyond the Canandaigua Call Center to a second site, to ensure geographic redundancy and meet increasing VCL demands. In May 2016, GAO reported that the VCL was unable to meet its targeted response times, coinciding with a 700 percent increase in call volume between FY2008 and FY2015. In July 2016, Member Services leadership determined that the implementation of various communication enhancements (in place or underway) that increased VCL access, including Press 7, voice recognition technology, vets.gov, and MyVA311,74 created increasing demand for services. Member Services leadership projected that this demand would continue to rise, and that the then-current pool of approximately 300 Canandaigua Call Center responders would be insufficient to meet this demand, particularly in the face of a goal of no call rollovers to backup centers. VCL and Member Services leadership determined that the Canandaigua Call Center location did not have the necessary space or applicant pool to allow for the needed future growth. Atlanta, GA, was chosen as the expansion location because Member Services had a preexisting call center infrastructure at its Atlanta-based Health Eligibility Center (HEC)75 Resources Center. The metropolitan Atlanta area also had a large pool of potential employee applicants. VCL leadership envisioned one VCL entity with two separate sites (the Canandaigua and Atlanta Call Centers) under the leadership of one director with the same processes, procedures, technology, and training content. 73 Government agencies are responsible for collecting and reporting data on federal procurements through the Federal Procurement Data System–Next Generation (FPDS-NG). The federal government uses the reported data to measure and assess the impact of federal procurement on the nation’s economy, learn how awards are made to businesses in various socioeconomic categories, understand the impact of full and open competition on the acquisition process, and address changes to procurement policy. Contracting Officers must submit complete reports on all contract actions, as required by the Federal Acquisition Regulation (FAR). 74 VA is introducing 1-844-MyVA311 (1-844-698-2311) as a go-to source for veterans and their families who do not know what number to call. Per the website: “This new national toll-free number will help eliminate the feeling of frustration and confusion that veterans and their families have expressed when navigating the 1000-plus phone numbers that currently exist.” 75 The VHA HEC provides information and customer service on key veteran issues such as benefits, eligibility, billing, and pharmacy. https://www.va.gov/CBO/memberservices.asp. Accessed December 1, 2016. VA Office of Inspector General 23 Evaluation of the Veterans Health Administration Veterans Crisis Line On July 21, 2016, planning for the new Atlanta-based call center started. An initial allocation of $620,000 from the VCL budget was made to Member Services to fund VCL’s expansion to Atlanta. Beginning October 10, 2016, the goal was for the Atlanta Call Center to serve as the initial site for rollover service.76 By November 21, 2016, Member Services anticipated that staffing at the Atlanta Call Center would be sufficient to allow for zero rollover calls to backup call centers.77 Member Services leaders planned to have the Atlanta facility fully staffed and telephonically operational by December 31, 2016. Text and chat services would begin in June 2017.78 As explained to OIG by Member Services leadership, to achieve these milestones, the Atlanta Call Center would need to accomplish the following tasks in the July 21, 2016 to December 31, 2016 timeframe.  Recruit, hire and train more than 200 staff including responders, SSAs and supervisors (66 percent increase in VCL staffing).  Relocate existing HEC Eligibility Call Center staff, who were utilizing the proposed Atlanta VCL Call Center space.  Acquire and install technology equipment.79  Modify the existing space to enhance the ability of Atlanta Call Center workers to communicate with each other as well as with callers in crisis.80 Overall, we found that profoundly different assessments of current and future readiness of the Atlanta Call Center existed between many of senior Member Services leaders and VCL’s clinical staff. For example, when we interviewed Member Services leadership, they informed us that Member Services and VCL clinical leadership are a “solid team working together,” the concept of zero rollover by November was “impressive” though feasible, and that the Atlanta Call Center project was “amazing.” We were also told that a strategy or backup plan was not in place in case they did not reach zero rollover, as they did not anticipate that this would happen. By contrast, VCL clinical staff opined that the timeline for opening the Atlanta facility was too ambitious based on current staffing levels and infrastructure and that VCL 76 The current practice is to direct incoming calls to the Canandaigua Call Center. If no VCL responders are available, the call rolls over to one of the four contracted backup centers. 77 Backup centers will be used on a contingent basis. 78 Responders are required to have 6 months of VCL telephone experience, prior to engaging in training for text and chat services. 79 The Atlanta Call Center will use the same Avaya platform as the Canandaigua Call Center, though servers required for call recording will not be installed until 2017. In order to meet the operational timeline, computer systems and monitors were redirected from shipments intended for VAMCs to the Atlanta Call Center via the lifecycle management process through the Technology Acquisition Center. 80 The existing HEC call center had 66 cubicles and required modification to accommodate 137 workspaces with white noise cancellation for audio privacy. In addition, workspaces were to be arranged so that supervisors would be in close proximity to teams and SSAs in close proximity to each other to allow for assistance and support during rescues. The modifications would also include a training room, break room, and relaxation and wellness area. VA Office of Inspector General 24 Evaluation of the Veterans Health Administration Veterans Crisis Line would not achieve its goal of zero rollover calls and program and policy implementation by the target date. VCL clinical staff did not believe that Member Services leaders grasped the impact the Atlanta Call Center would have on overall VCL functioning. For example, as Member Services leadership was standing up the Atlanta Call Center, they rolled out other projects that diverted staff time and resources, such as the implementation of a labor-intensive timekeeping system. At the same time these adjunct functions were being added to staff responsibilities, Member Services leadership was removing existing supportive infrastructure. Since Member Services assumed the VCL in February 2016, knowledgeable managers, trainers, analytics personnel, and timekeepers have been reassigned from the VCL to Member Services. Member Services leadership planned for the Atlanta Call Center to begin accepting calls by October 10, 2016. Bringing the Atlanta Call Center online in a three-month period entailed the rapid hiring and training of new staff. The training content is the same for responders at both the Atlanta and Canandaigua sites, but with notable differences in trainer-to-learner ratios. For instance, in order to accommodate the sizable number of trainees, class sizes are larger at the Atlanta Call Center, ranging from 44 to 62 trainees, versus 20 trainees per class at the Canandaigua Call Center. Once the responders have completed classroom training and passed a proficiency test, they are assigned to work with a preceptor for one to three weeks. The preceptor-to-responder ratio at the Canandaigua Call Center is 1:1. The original plan for the Atlanta Call Center called for a 1:2 or 1:3 preceptor to responder ratio. However, due to limited preceptor availability and large class sizes, the ratios have been as high as 1:16. The supervisors hired to work at the Atlanta Call Center did not have the same skill set as those at the Canandaigua Call Center. Canandaigua Call Center supervisors first served in a responder role, while most Atlanta Call Center supervisors had not. Because of this, we were told that Atlanta Call Center supervisors would be required to complete responder training prior to supervisor training. Additionally, after new supervisors at the Canandaigua Call Center completed training, they were scheduled on shifts with more experienced supervisors prior to assignment to solo tours. At the Atlanta Call Center, new supervisors will receive on-the-job training for one month before they are assigned solo tours. One VCL supervisor told OIG inspectors that inexperience might detrimentally affect practice at the Atlanta Call Center because new responders, particularly linked with new supervisors, may be too quick to call rescues whereas more experienced responders may be able to de-escalate the situation. Despite the experiential and training differences between sites and the potential for variances in practice, with the exception of silent monitoring, we found no documentation of plans to compare metrics between sites, including rescue rates. Member Services leadership anticipated some increase in rollover demand through October 2016 due to the need to send Canandaigua Call Center staff to the Atlanta Call Center for training purposes. However, Member Services leadership believed the “disruption would be minimal” because they estimated the number of staff being detailed to Atlanta would equal the number of new staff recently hired at the Canandaigua Call Center. With no significant loss of responders, Member Services leadership believed the Canandaigua Call Center would be able to maintain its rate of answering 75–80 VA Office of Inspector General 25 Evaluation of the Veterans Health Administration Veterans Crisis Line percent of VCL calls. Member Services leadership stated that they alerted the backup centers in anticipation of some increase in rollover calls. However, the rapid establishment of the Atlanta Call Center required that a substantial number of staff from the Canandaigua Call Center be detailed to the Atlanta Call Center to train staff as well as assist with workload. As of November 7, 2016, 114 staff had been detailed from the Canandaigua Call Center to the Atlanta Call Center. From September 18, 2016 to December 18, 2016, 15,940 FTE employee hours and 600 hours of travel compensation time were allocated for Atlanta Call Center training. The effective loss of FTE coincided with a 66% increase in the rollover of calls to backup centers. The rollover rate for the VCL increased from 21 percent to 28 percent from August to September 2016. Starting in October 2016 (when the Atlanta Call Center went live), the rollover numbers included calls that could not be taken by either the Canandaigua or Atlanta Call Centers. Despite the addition of the Atlanta Call Center, the rollover rate for October 2016 increased to 34.9 percent and exceeded 40 percent seven days of the month. Canandaigua Call Center managers opined that this will likely continue until the Atlanta Call Center is fully operational. The diversion of Canandaigua Call Center staff to Atlanta in order to achieve VCL programmatic milestones also contributed to a delay in the development and implementation of policies, programs, and procedures for the VCL. Examples of delays cited by staff include the deferral of annual lethality assessment training for responders, the delayed rollout of chat and text monitoring at the Canandaigua Call Center, and delayed implementation of wellness programs such as 1:1 quarterly meetings between responders and supervisors for “mental health checks.” Member Services leaders informed us that they utilized forecasting methods for staffing projections, performed demand analytics (such as calculating the impact of various VA programs on VCL demand, including Press 7 and 311), and created a work breakdown structure81 for the Atlanta Call Center. However, they did not develop a formal business plan for the implementation of the Atlanta Call Center that laid out steps for achieving strategic priorities and communicating key project objectives and goals. For example, it was clear that they wanted to hire a large number of responders quickly to answer phones and reduce reliance on backup call centers. It was not clear that they determined, in a prospective fashion, all of the critical components necessary to make this happen. For instance, the decision to hire Atlanta Call Center responders and supervisors simultaneously created a built-in lag for supervisor readiness, given the need for supervisors to first complete responder training. On-site supervisors must be available to clear responders to answer phones. In order to compensate for the lack of Atlanta Call Center supervisors, they were required to divert 960 supervisor hours from the Canandaigua Call Center, affecting Canandaigua functions. 81 A work breakdown structure is a document that breaks down projects into manageable components to be carried out by the project team; it is not a project plan. VA Office of Inspector General 26 Evaluation of the Veterans Health Administration Veterans Crisis Line The lack of strategic forethought also impacted the accuracy of forecasting calculations. For instance, forecasting imprecisions regarding rollover coincided with the detailing of more staff from the Canandaigua Call Center to the Atlanta Call Center than anticipated. Specifically, a Member Services leader anticipated that the opening of the Atlanta Call Center would result in a rollover increase to 180 calls per day through mid-November. However, we found in examining rollover data from October 1–November 4, 2016, that daily rollover call numbers ranged from 404 to 889. In addition, Member Services leaders made the decision to roll out the Atlanta Call Center without first establishing on-site Atlanta Call Center leadership, a critical piece to ensuring proficient execution of call center function. The September 2016 VCL organizational chart called for Atlanta to have its own Deputy Director and Director for Team Operations. However, as of September 20, 2016, the inaugural responder training class was in session with plans to begin operations on October 10, 2016, and the leadership positions had not been posted. As of November 8, 2016, this iteration of the organizational chart had been rescinded. VCL leadership structure reverted to that outlined in the July 2016 organizational chart, which does not include either a Deputy Director, a Director of Team Operations for Atlanta, or other leadership positions specific to the Atlanta Call Center. Member Services did not meet targeted deadlines. As of November 3, 2016, the Atlanta Call Center had answered more than 500 calls. However, these calls were answered by Canandaigua Call Center responders detailed to Atlanta. The timeline proposed by Member Services leadership called for the September 19 training class of 56 responders to be answering the Atlanta Call Center phones by October 31, 2016. However, as of November 3, fewer than 11 responders were cleared to take calls independently and scheduled for future assignments. Member Services did not meet the zero rollover target of November 21, 2016. The deadline to meet the zero rollover target was amended to December 12, 2016, and again not met. VCL staff informed us that due to pressure to clear responders to meet the zero rollover goal, Member Services leadership has proposed a temporary revocation of training standards. Specifically, they would like to temporarily detail responders with as little as one month of independent phone work to supervisory roles without supervisory training, in order to clear additional responders to answer calls. VCL staff described these decisions to OIG inspectors as both compromising standards of training and “threats to patient safety.” VCL staff are currently identifying strategies that balance training efficiencies and safety. The last revised deadline for zero rollover was December 31, 2016. C.  VCL Quality Assurance Quality Management Quality management (QM) is an approach focused on making and measuring improvements to a program with the prevention of problems being the primary objective. As problems are prevented, achievement of the desired outcomes improves. Outcomes VA Office of Inspector General 27 Evaluation of the Veterans Health Administration Veterans Crisis Line are measured, changes are made based on analysis of the outcomes, and outcomes are measured again. Systematic collection of relevant and actionable data for analysis is crucial when making decisions that will prevent problems. As noted by a healthcare quality organization, healthcare quality professionals need to be well prepared to serve in an ever-changing environment. They should be well versed in health data analytics, population health and care transitions, regulatory requirements, safety, quality review and accountability, and performance and process improvement.82 To be effective, VCL’s QM data collection and analysis should be accurate and inform VHA and VCL leadership and staff whether their actions effectively serve veterans and others who use VCL services. In our February 2016 report,83 VHA leadership concurred with recommendations for establishing a formal quality assurance process and developing a VHA directive or VHA handbook for the VCL. We reviewed the VCL QM program structure and processes, the VCL QM program manual, and the draft VCL directive. We identified that systems deficiencies persisted in QM program processes, and that neither the VCL QM program manual nor the draft VCL directive provided a framework for a QM program structure. In the absence of established VCL guidance, we referred to existing VHA guidance for our current review. Quality Management Leadership. VHA Directive 1026, VHA Enterprise Framework for Quality, Safety, and Value, issued August 2, 2013, outlines responsibilities of leadership for program integration and communication, and the designation of individuals with appropriate background and skills to provide leadership to promote quality and safety of care.84,85 We found several challenges in VCL QM staff’s ability to collect, analyze, and effectively review relevant QM data in order to make changes to prevent problems and improve outcomes for callers. After reviewing the number and types of QM roles in the VCL as well as QM staff experience and background, we determined that the challenges likely stemmed from the QM staff’s lack of training in QM principles. Member Services leadership tasked QM staff with multiple responsibilities and competing priorities that included VCL QM program and policy development, data collection and analysis, data presentation for evaluation and action planning, and identification of outcomes measures. However, the QM staff had not been provided with training in the skills needed to provide leadership to promote quality and safety of care, leading to deficiencies in the QM program. 82 National Association for Healthcare Quality, Essential Competencies for the Healthcare Quality Professional, First Edition, 2015. 83 VAOIG, Healthcare Inspection: Veterans Crisis Line Caller Response and Quality Assurance Concerns, Canandaigua, New York, (Report No 14-03540-123, February 11, 2016). 84 VHA Directive 1026, VHA Enterprise for Framework for Quality, Safety And Value. August 2, 2013. 85 Ibid. VA Office of Inspector General 28 Evaluation of the Veterans Health Administration Veterans Crisis Line VCL and QM leadership were undergoing frequent restructuring during our review, and the organizational chart was revised three times from February through November 2016. Key QM positions remained unfilled during our review. As of November 30, 2016, a permanent VCL Director had not been named. The current acting VCL Director assumed the acting position in June 2016 after the VCL Director left after a 6-month tenure. In June 2016, the organizational chart included one position for QM, a Quality Assurance Officer, and this position was unfilled. In July 2016, the QM program no longer included the Quality Assurance Officer but listed four positions with the following titles: Quality Assurance Clinical Officer (QACO), Quality Management Officer (QMO), Quality Management Specialist, and Silent Monitor. We learned that some QM staff members did not have a background or training in quality management principles and processes. Six QM Silent Monitor staff documented call quality elements and provided immediate feedback to responders. The Silent Monitor staff did not conduct data analysis. QM managers collected and analyzed the call quality data. The silent monitors’ ability to conduct their job duties was not limited by a lack of QM training. In September 2016, VCL leadership added another QM position, Clinical Care Coordinator, to the organizational chart, and the Acting QACO assumed the responsibilities of this new position in addition to her already assigned duties. In an effort to improve quality, VCL QM staff collect data on three clinical quality performance measures: Silent Monitoring, End of Call Satisfaction Question, and Complaints/Compliments.86  Silent monitoring, as previously noted within this report, is a process for listening to and evaluating a responder’s interactions with veterans on incoming VCL calls and coaching responders on areas of strength and areas for improvement. Monitoring is conducted “silently” in that the responder is unaware that the call is being monitored; the call is monitored in real time, and feedback is provided immediately following the responder’s call documentation.87 VCL managers have identified critical and non-critical elements of call quality for evaluation; for example, Greeting, Rapport Building, Safety Planning, and Suicide Risk Assessment.  The End of Call Satisfaction question is a measure used for telephone contacts only, not for chat or text contacts to VCL. Responders are instructed to conclude each call by asking the caller, “If you were in crisis, would you call the Veterans Crisis Line again?” 86 Veterans Crisis Line, Quality Management Program Manual. Updated July 12, 2016. Although the responder does not know which call is monitored, responders have been informed that routine silent monitoring will occur. Callers were not notified that monitoring for quality management purposes may be taking place during the call. 87 VA Office of Inspector General 29 Evaluation of the Veterans Health Administration Veterans Crisis Line  Complaint Tracking is a mechanism used to evaluate complaints received through any type of contact with VCL staff (telephone, chat, or text). VCL staff used a templated complaint form, which responders complete and send by email to the Complaints Team. A member of the Complaints Team enters complaints into a log for follow-up.88 Although compliments are included as a clinical quality performance measure, we did not find a process outlined for addressing compliments. QM Data Analysis. We found that while VCL staff collect data on clinical quality performance measures, the QM program lacked defined processes for analyzing and presenting data and for developing a committee structure for reporting the analysis, making recommendations, and following up; examples follow.  In reviewing the VCL’s reporting of data analysis from Silent Monitor staff from April through June 2016, we found deficiencies in presentation of QM data for review and action planning, resulting in reports that were difficult to interpret. For example, in one report of percentages, results exceeded 100 percent (Appendix E, Tables 5–6).  We reviewed data analysis for the End of Call Satisfaction Question and found that VCL staff analyzed performance data as the percentage of veteran callers who answered the question, rather than of total callers. Although responders were instructed to ask each caller the End of Call Satisfaction Question, responders did not consistently ask the question. Using the number of callers who answered the question rather than the total number of callers may overestimate satisfaction rates. VCL staff reported the average satisfaction rate from November 2015 through August 2016 to be greater than 94 percent.89 This information was collected monthly and shared electronically among the Acting QMO, Silent Monitors, and Lead Analyst (not a QM position).  VCL staff maintained an electronic workbook for recording complaint follow up and resolution. The workbook was divided into multiple worksheets that tracked elements including: description of the complaint information, nature of complaint, descriptive data, action taken or processes, outcomes, and standards involved. Bar graphs embedded throughout the worksheet illustrated each month’s aggregate data. The QM Specialist, who was responsible for complaint followup, reviewed responders’ call documentation and contacted callers to resolve complaints. We noted comments in the complaint descriptive data that responders’ call documentation did not provide support for the complaint. Examples include: “Veteran complained that he was hung up on while trying to reach VCL. No proof in Medora [the VCL documentation record],” or “There was no specific information to investigate,” or “Veteran said responder discounted his problems. No proof in Medora.” We found that complaint resolution was a 88 VCL Standard Operating Procedure, Quality Assurance Complaint Tracking System, September 29, 2015. From November 2015 to August 2016, the percentage of veteran responses to the End of Call Satisfaction question ranged from 31.6 to 69.3 percent. 89 VA Office of Inspector General 30 Evaluation of the Veterans Health Administration Veterans Crisis Line manual process and that data analysis of substantiated complaints was based on subjective data. In summary, many variables affect the final analysis of a complaint. The predominant mechanism of having a complaint evaluated is through the responder's documentation and the QM specialist's interpretation of what was written.90 In August 2016, the VCL implemented audio call recording of incoming and outgoing calls for quality assurance purposes. As of November 3, 2016, calls were recorded at the Canandaigua Call Center and VCL managers planned to implement call recording at the Atlanta Call Center in the coming months. By November 3, 2016, the VCL recorded 64,666 total calls; however, the Acting QACO told us that the VCL lacked the ability to retrieve and listen to the recorded calls, due to information technology issues.    As of December 16, 2016, the information technology issues for the Canandaigua Call Center were resolved. The Atlanta Call Center site did not have access to the call recordings as of December 16, 2016, due to continued information technology issues. Although call recording began in August 2016, the VCL lacked a policy for managing recorded calls; however, VCL managers provided us a draft policy dated October 19, 2016. The Acting QACO told us that they planned to integrate call recording into the QM program data analysis processes as part of silent monitoring, with monitoring of live and recorded calls; and into complaint resolution processes, by reviewing a call in response to a complaint and providing feedback to responders as indicated. Because the VCL did not have the ability to retrieve and listen to the recorded calls, data analysis had not yet begun in November 2016. Additionally, the new call center in Atlanta (discussed above) could potentially have QM concerns that are no different from their Canandaigua partner, but the ability to recognize site-specific issues, especially in a new program may be facilitated by separating quality data elements by site. VCL QM Committees and Planning. VHA requires a standing committee to review data, information and risk intelligence, and to ensure that key quality, safety and value functions are discussed and integrated on a regular basis. This committee should be comprised of a multidisciplinary group, should meet quarterly, and should be chaired by the Director.91 We did not identify a VCL standing committee that met the intent of VHA requirements outlined in Directive 1026.92 90 Other general methods for evaluation of caller satisfaction could include a patient advocate system, post call survey, or call recording analysis. 91 VHA Directive 1026, VHA Enterprise Framework for Quality, Safety, and Value, August 2, 2013. 92 We found that neither the VCL QM program manual nor the draft VCL directive provided a framework for reviewing QM data analysis for action planning. In the absence of established VCL guidance, we referred to existing VHA guidance for QM programs. VA Office of Inspector General 31 Evaluation of the Veterans Health Administration Veterans Crisis Line Member Services leadership chartered the multidisciplinary CAB in April 2016 with the purpose of “sharing proven best practices with the strategic goal of clinical services and industry best business operational practices” (see discussion of CAB above). 93 We reviewed CAB meeting minutes from May to October 2016 and found that, over time, documented discussions at meetings became more comprehensive, although issues were not consistently tracked to resolution. Initially, from May to July 2016, minutes lacked documentation of review of the three clinical quality performance measures,94 and discussions focused on programmatic VCL issues; for example, strategies for high-frequency callers, readiness and resilience training for responders, and action plans for addressing GAO and OIG recommendations. August and September meeting minutes included attachments for silent monitor data analysis although we did not find discussion of the data. The meeting minutes for October included attachments for data analysis of the three clinical quality performance measures and documented discussions of the data. Planned actions were not consistently tracked to resolution through the October 2016 CAB meeting. For example, September meeting minutes documented plans to provide an update on information technology issues affecting both VCL and facility SPCs; however, October minutes did not include information technology issues as an agenda item. Also in September, the CAB planned to develop a message about VCL updates for dissemination to network directors, network mental health leaders, and facility directors, with a due date of September 29, 2016; however, the October meeting minutes did not contain an agenda item for follow-up on this plan. On September 14, 2016, the VCL’s governing board, the ELC (see above discussion of ELC) met for the first time. The ELC meeting minutes documented the VCL organizational governance structure, proposed future topics for discussion, and aggregated data for silent monitoring and complaints. Ongoing actions regarding silent monitoring data were presented to the ELC; however, no actions were presented to or recommended by the ELC regarding complaint data. The ELC did not review data for the third clinical quality performance measure, End of Call Satisfaction, during the meeting. Interviewees told us that data from the three clinical quality performance measures were obtained monthly, and the VCL QACO stated the ELC planned to review the data monthly after the ELC’s first meeting in September 2016. We reviewed October and November 2016 meeting minutes and found data analysis and planning for clinical quality performance measures in the October minutes; however, November ELC minutes contained no discussion of data analysis or review; instead the discussion included proposed programmatic actions that were to be developed., 93 The CAB membership included representation from the VA National Suicide Prevention Program, VHA Member Services, VISN 2 Behavioral Health, Substance Abuse and Mental Health Services Administration (SAMSHA), and leadership from the VCL, including the Director, Deputy Director, Quality Assurance Clinical Officer, and the Deputy for Business Operations. There was also some initial involvement from the Department of Defense Suicide Prevention Office. 94 Silent Monitor, End of Call Satisfaction, or Complaints/Compliments. VA Office of Inspector General 32 Evaluation of the Veterans Health Administration Veterans Crisis Line Long-term VCL planning was described in the VCL strategic plan for 2017–2022. Although the plan included goals for QM, the strategic plan lacked clearly defined action plans for achieving the goals. Policies, Procedures, and Handbooks. VHA Directive 6330 (1), Controlled National Policy/Directive Management System, established policy and responsibilities for managing, distributing, and communicating VHA directives.95 VCL policies have been created in response to external reviews and internal processes but a controlling directive has not yet been published. A draft directive was in development, dated April 4, 2016; however, it lacked defined roles and responsibilities for VCL leaders, such as the VCL Director.96,97 We found that VCL policies, procedures, or handbooks were not readily accessible for staff reference, but were embedded within training materials located on the VCL SharePoint site for training. VCL leaders developed a QM Program Manual which was updated in July 2016 (no initial publication date was available).98 The program manual did not outline a framework for the QM program that is consistent with relevant existing VHA directives providing guidance for QM programs, including:  VHA Directive 1026, VHA Enterprise Framework for Quality, Safety, and Value, August 2, 2013  VHA Handbook 1050.01, VHA National Patient Safety Improvement Handbook, March 4, 2011  VHA Directive 2008-077, Quality Management and Patient Safety Activities That Can Generate Confidential Documents, November 7, 2008 VHA leaders concurred with the findings of the February 2016 OIG report recommending the development of a VHA Directive or Handbook and agreed to a target completion date of June 2016. The OIG report outlined that the lack of a VCL-specific directive or handbook was a contributing factor for the absence of an organized QM process. As of November 30, 2016, this recommendation is open, but VHA has prepared a draft directive. We reviewed the VHA Directive Transmittal Sheet, Operation of the Veterans Crisis Line, which was an unpublished draft document dated April 2016. According to this draft directive, VHA planned for the Member Services Quality Assurance Team to be responsible for ensuring that the VCL provide high-quality suicide prevention and crisis intervention, follow procedures, provide individual coaching to frontline staff, and identify trends to inform policy and training development. In addition, the Member Services 95 VHA Directive 6330(1), Controlled National Policy/Directives Management System, June 24, 2016, amended January 11, 2017. 96 Veterans Crisis Line, Quality Management Program Manual. Updated July 12, 2016. 97 VCL Quality Assurance Complaint Tracking System, Standard Operating Procedure (SOP: September 29, 2015) Draft. 98 Veterans Crisis Line, Quality Management Program Manual. Updated July 12, 2016. VA Office of Inspector General 33 Evaluation of the Veterans Health Administration Veterans Crisis Line Analytics Team would be responsible for compiling and analyzing VCL call center data for tracking and trending purposes. The draft directive did not define roles or membership for either team, or define specific responsibilities for existing VCL roles. 99 Outcome Measures for Quality Improvement. We found that while the VCL measured internal performance of its staff (silent monitors, End of Call Satisfaction question, and complaints), its QM data analysis did not include measures of clinical outcomes for callers. During interviews, we inquired about outcome measures to evaluate the success of a veteran’s transition from the VCL to other dispositions. Specifically, we asked whether the VCL collected data that would answer the following questions:  Who follows up with the veteran to make sure he/she receives additional services as requested or planned?  Are the SPCs addressing consults from the VCL?  Do VCL staff provide enough caller information to the SPC?  What could be done to improve processes? We learned that responders asked the End of Call Satisfaction Question and that VCL staff collected data regarding facility SPC responses to VCL referrals, including timeliness of closure of SPC referrals as documented by the SSAs. VCL leaders provided data to us; however, the data represented individual instances of follow-up on SPC referrals.100 We found no evidence that QM staff conducted data analysis in order to identify opportunities for improvement in SPC referral follow-up. Without analyzing the SPC referral data, VCL managers were unable to identify trends or plan actions for improvement. Further, we did not find information or discussion of this data documented in VCL, CAB, or ELC meeting minutes. We identified deficiencies in the VCL QM program including data analysis and presentation of clinical quality performance measures, lack of development of a directive consistent with established VHA guidance, lack of a reporting structure for regular review of performance measures, and frequent changes in the organizational structure of the QM program. We found that deficiencies in the QM program were related to VHA leadership failing to provide a developmental plan, appointing staff into positions without formal QM training, and assigning staff multiple competing priorities.101 Measurement of Program Success with Adverse Outcomes Reviews. We found that the VCL had no process in place for routinely obtaining or reviewing data on serious adverse outcomes, such as attempted or completed suicides by veterans who made contact with the VCL prior to the event. By not reviewing serious adverse outcomes, VCL QM managers missed opportunities for quality improvement. 99 VHA Directive Transmittal Sheet, Operations of the Veterans Crisis Line, April 4, 2016. SPC data included the SPC Referral Identification number, Site Name, Referral Date, Call Synopsis, and SPC Details. 101 VHA Directive 1026, VHA Enterprise for Framework for Quality, Safety and Value, August 2, 2013. 100 VA Office of Inspector General 34 Evaluation of the Veterans Health Administration Veterans Crisis Line An example of a process improvement evaluation review that is a component of VHA’s quality assurance program is the use of root cause analysis to identify contributing factors, or precipitating factors, in effect prior to an adverse outcome.102 In addition to such reviews, VHA leadership sets requirements for reporting serious adverse outcomes in issue briefs. We learned that adverse outcomes were not aggregated for review by VCL leadership in order to measure performance improvement for achieving more successful outcomes. VCL staff completed a review to supply a requested issue brief to VHA related to the veteran referenced in this report. In this case, VCL staff submitted an issue brief to VHA due to media attention. The VCL issue brief was limited and did not document whether any of the responders were interviewed, did not contain any fact finding to evaluate the adequacy of the phone care rendered by the responders, and did not include a plan to evaluate the case further. The Acting Director and Acting QACO confirmed that debriefings or other reviews were not conducted after known suicide attempts or completions. Training and Supervision of Responders and SSAs New Employee Orientation (NEO) Training. New VCL employees undergo NEO training, which includes some training modules common to both responder and SSA duties, such as basic orientation to the physical space and military culture instruction. Responders also receive training on interacting with callers. SSAs do not have direct contact with callers, so they do not receive caller interactions training and are not subject to silent monitoring. Responder NEO training includes simulated calls during training and precepting of the new employee during live calls after completion of the training modules. The NEO training guides include links to VCL procedures located on the VCL’s internal training website. Although staff had access to the internal website, we found the VCL lacked a mechanism for ensuring that staff received updated information on revised procedures. For example, from June to July 2016, the VCL revised training regarding increased supervisor contact with callers and prohibitions against SSA contacts with callers. However, the VCL had no records that staff acknowledged the changes. Monitoring of Responders and SSAs. VCL QM staff monitored the quality of a responder’s interaction with callers through silent monitoring. Silent monitoring occurred in real time and the responder was unaware that the call was being monitored. QM staff assessed the quality of critical and noncritical elements of the call and provided feedback and coaching to the responder on areas of strength and areas for improvement immediately after the responder completed call documentation. The VCL attempted to monitor each responder once per pay period although this was not always possible due to other duties that took responders off the phones. The VCL did not have a process for monitoring quality of service from SSAs. 102 VHA Handbook 1050.01, VHA National Patient Safety Improvement Handbook, March 4, 2011. VA Office of Inspector General 35 Evaluation of the Veterans Health Administration Veterans Crisis Line Comparison of the VCL with Non-VA Crisis Call Centers. We compared the VCL with four well-established and recognized non-VA crisis call centers. We accomplished this by conducting a literature review of the characteristics of non-VA crisis call centers, accrediting bodies, and industry standards, and by conducting telephone interviews and/or site visits to the following crisis lines:  Didi Hirsch Suicide Prevention Center 103  Boys Town National Suicide Hotline104  National Domestic Violence Hotline (NDVH)105  9-1-1 National Emergency Number106 We found that both the VCL and all of the suicide-specific non-VA crisis call centers reviewed were AAS accredited and operated 24-hour crisis hotline services. The VCL offers text and chat on a 24/7/365 basis. Not all of the non-VA crisis call centers provide both online crisis chat and text services; those that do so have limited hours of operation. We found that there were no industry performance standards or quality metrics (clinical and/or administrative) specific to crisis call centers. Both VCL and non-VA crisis call centers develop their own measures. As previously mentioned, one example of how VCL measures performance is the End of Call Satisfaction question where the responder asks the caller, “If you were in crisis, would you call the Veterans Crisis Line again?” Staff at one of the non-VA crisis centers ask callers at the beginning and end of each call how they would rate themselves on a scale of 1 to 5 on their likelihood to harm themselves. The AAS does not specify a minimum education or licensure requirement, standard, or recommendation for crisis line responders, supervisory staff, or management. AAS strongly emphasizes the selection of individuals who are empathic, nonjudgmental, compassionate, and able to engage with others. For both the VCL and non-VA crisis call centers, we noted a wide variation of responder qualification requirements. Educational requirements ranged from none (education was not one of the eligibility criteria) to having a bachelor’s degree in a social science field. AAS also emphasizes the provision of crisis specific training, interactive role play, shadowing an experienced peer, trainer/supervisor feedback, and skill demonstration.107 Both VA and non-VA crisis call centers have some degree of training, shadowing or mentoring, trainer/supervisor feedback, and skill demonstration. 103 Didi Hirsch Mental Health Services, 4760 South Sepulveda, Culver City, CA 90230. Boys Town National Suicide Hotline, 14100 Crawford Street, Boys Town, NE 68010. 105 National Domestic Violence Hotline, P.O. Box 161810, Austin, TX 78716, Office Line: (512)-453-8117. 106 National Emergency Number Association, 1700 Diagonal Road, Suite 500, Alexandria, VA 22314. 107 AAS Standards for Accreditation p. 26-33. 104 VA Office of Inspector General 36 Evaluation of the Veterans Health Administration Veterans Crisis Line For both VCL and non-VA crisis call center responders, the stressful nature of working with individuals in crisis and the intense high-risk interactions with callers can prove challenging. Responders may experience secondary trauma and become burned out (depressed, emotionally distant, or loss of interest), resulting in decreased effectiveness and a high level of turnover. This underscores the importance of wellness programs for responders. The VCL wellness program includes an Acting Wellness Program Coordinator and has several initiatives in place. However, we were told that the wellness initiatives are currently lagging due to a staffing shortage with the diversion of VCL staff to Atlanta. The initiatives include Readiness and Resilience,108 Standard Work for Postvention,109 Speaker’s Bureau,110 Peer Groups,111 Wellness Coaches,112 and Employee Recognition.113 The non-VA crisis call centers’ supportive measures include maintaining a peaceful and supportive environment for responders. Their wellness program initiatives include some or all of the following: quiet reading and relaxation rooms with peaceful décor and soothing colors, a workout room and a full time wellness coach that works with responders to create a work/life balance. The wellness coach also offers individual support and teaches self-care tools. Debriefing includes both formal and informal procedures with staff after an adverse outcome (such as a suicide). One non-VA crisis center encourages staff to take a day-and-a-half paid leave immediately following an adverse outcome. Objective 3: Status of VHA’s Implementation of Action Plans in Response to February 2016 OIG Report In February 2016, OIG issued a report titled Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (Report No. 14-03540-123) that addressed allegations relating to VCL caller response times and other quality assurance issues. In this prior report, we substantiated allegations that:  Some calls routed to backup crisis centers were answered by voicemail.  Callers did not always receive immediate assistance from VCL and/or backup center staff. 108 The intent of the Readiness and Resilience Program is to allow responders and SSAs 15 minutes before and after a shift to prepare for the shift and to process their experience after the shift. 109 Standard Work for Postvention is a process where the supervisor debriefs with a responder following a completed suicide by a caller. 110 The Speaker’s Bureau is an informal group of VCL staff who participate in community resource fairs and events, specifically as a part of the VCL staff wellness program. 111 Peer Groups include a general processing and an article review group. 112 Wellness coaches implement and lead projects that include a lunch and learn, a book club, and an art show. 113 Employee Recognition includes awards for employee of the month, active service, and a team player. VA Office of Inspector General 37 Evaluation of the Veterans Health Administration Veterans Crisis Line  VCL management did not provide SSAs with adequate orientation and ongoing training. We also identified gaps in the VCL quality assurance process. We made seven recommendations to the Director of OMHO (as noted above, the VCL has since been re-aligned under Member Services) who submitted appropriate action plans for the seven recommendations that were to be implemented by September 30, 2016. We reviewed VHA documents submitted to OIG as evidence to support the completion of the planned actions and determined that VHA has not completed the planned actions. The seven recommendations remain open and VHA continues to work on actions to close them. Our specific findings are below: February 2016 Recommendation 1. We recommended that the Office of Mental Health Operations (now Member Services) Executive Director ensure that issues regarding response hold times when callers are routed to backup crisis centers are addressed and that data is collected, analyzed, tracked, and trended on an ongoing basis to identify system issues. Status Update as of December 15, 2016: We found that Member Services and VCL leaders did not demonstrate the use of the L2HS hold time data to improve performance (the data comes from computer software that routes calls from the main NSPL to backup centers). Although we found evidence that VCL staff reviewed data, we did not find that they used the data systematically to provide feedback to backup centers regarding performance parameters such as queue times, abandonment rate, and call answer rate. At the time of this report, VHA had not completed the necessary actions to close this recommendation. February 2016 Recommendation 2. We recommended that the OMHO (now Member Services) Executive Director ensure that orientation and ongoing training for all VCL staff is completed and documented. Status Update as of December 15, 2016: We found that VHA did not ensure that orientation training for all VCL staff was completed and documented. We reviewed documentation for 52 staff (45 responders and 7 SSAs) who started NEO training between February and May 2016 and found that the documentation was incomplete in regard to training module completion and/or silent monitoring114 that was conducted as part of NEO training. Seven of 45 responders’ NEO training checklists lacked dates documenting completion of some or all of the training modules, and documentation showed discrepancies or was 114 As noted above, silent monitoring is conducted by VCL QA staff who listen to responder/caller communications during an incoming call to evaluate the following: Greeting, Rapport Building, Safety Planning, and Suicide Risk Assessment. VA Office of Inspector General 38 Evaluation of the Veterans Health Administration Veterans Crisis Line incomplete.115 For example, the responder NEO training checklist has spaces for documentation of dates of supervisor approval for three silent monitors conducted during training. Of the 45 responder checklists we reviewed, 5 (11 percent) had documentation of 3 silent monitors although the documentation was inconsistent;116 25 (55 percent) documented 1 or 2 silent monitors; and 3 (7 percent) had no silent monitors documented. In the remaining 12 (27 percent) responder checklists, we could not determine the number of silent monitors documented; for example, 3 sets of the same supervisor initials listed consecutively without accompanying dates, or handwritten lines grouping the 3 silent monitor spaces together and pointing to a single approval. Of the 52 NEO training checklists we reviewed (45 responder and 7 SSA), 2 lacked a date documenting NEO training completion,117 and therefore we could not determine the length of time for training. We also found inconsistencies in the time documented from start to completion of NEO training. For responder training, the documented time for training ranged from 31 days to 185 days.118 Total documented time for SSA training ranged from 37 to 64 days. At the time of this report, VHA had not completed the necessary actions to close this recommendation. February 2016 Recommendation 3. We recommended that the OMHO (now Member Services) Executive Director ensure that silent monitoring frequency meets the VCL and AAS requirement119 and that compliance is monitored. Status Update as of December 15, 2016: We found that VHA did not ensure that the VCL established a requirement for silent monitoring frequency and therefore could not have monitored compliance with that requirement. The VCL QM manual assigned Silent Monitor staff to attempt to monitor each responder at least once per pay period. The QM manual acknowledged the aspirational nature of this objective by stating, “At times, this is not possible due to employee’s assignment to 115 For example, all training for one employee was documented as occurring on the same date. For another employee there was a 3 month gap in dates for completion of training modules, with some modules being completed on a date (June 18, 2016) after the supervisor gave final approval (June 13, 2016). The records of other employees included documentation of training modules that the training staff told us had been removed although the checklist was not revised to reflect the changes. 116 For example, one used the same date for all three silent monitors; one had only one date but three different supervisors signed their approvals. 117 Of the two, one checklist was for responder training and the other was for SSA training. 118 Some dates reflected an extension of times to complete NEO training that coincided with our notification to the VCL of our review; this may have reflected administrative activities taken to provide completed paperwork by the time of our review. 119 The AAS accreditation manual does not specify a specific method for crisis worker monitoring: “It is the responsibility of the organization to define its procedures for measuring the quality and show proof that it is using those procedures.” In the absence of an AAS requirement for silent monitoring, we referred to the VCL’s QM manual. VA Office of Inspector General 39 Evaluation of the Veterans Health Administration Veterans Crisis Line other duties which take them off the phones, such as precepting new hires or participating in other special projects.” VCL QM managers analyzed silent monitor data as the percentage of responders monitored monthly per total responder staff, rather than comparing the monitoring of individual responders to the aspirational goal of once per pay period per responder. We found that VCL managers did not demonstrate measuring compliance with a requirement for silent monitor frequency. We reviewed VCL data for calls that received silent monitoring from April through June 2016 and determined that silent monitors evaluated quality assurance for approximately one percent of VCL calls answered., Table 3. Percentage of Silent Monitors of Calls Answered by VCL Responders, April–June 2016 Number of Silent Monitors. Total Calls. Percentage April 371 41394 0.90 May 419 44822 0.93 June 475 45635 1.04 Month Source: VHA VCL Silent Monitoring Data VCL managers conducted silent monitoring of “all types of calls, including prank calls, non-veteran calls, misdirected calls, and abusive and/or high frequency calls.” The managers defined these types of calls as non-core business calls. We reviewed silent monitor data recorded from April through June 2016, for a total of 1,265 calls. We found that 632 of the 1,265 calls (50 percent) that received silent monitoring were prank calls, misdirected calls, abusive and/or high frequency calls, or other non-core calls (civilian calls, veteran general support, or Office of Information Technology (OIT) test calls). Table 4. Silent Monitors of VCL Non-Core Calls April–June 2016 Type of Call Total Silent Monitors April - June 2016 Prank calls 106 Misdirected calls (VBA, VHA, Pharmacy issues) 53 Abusive / High frequency callers 120 Other Non-Core Calls (Civilian, OIT Test Calls, Veteran General Support) 353 Total 632 Source: VHA VCL Silent Monitoring Data VA Office of Inspector General 40 Evaluation of the Veterans Health Administration Veterans Crisis Line Fifty percent of the calls monitored were not from veterans in crisis or other core VCL business calls; however, once a responder was monitored for a call of this type, that responder would not be monitored again until the next pay period cycle, and may not be monitored at that time, if assigned to “other duties which take them off the phones.” VCL managers conducted silent monitoring on one percent of calls answered, and half of that was for non-core VCL business. We identified that the VCL’s processes for silent monitoring, (attempting to monitor each responder at least once per pay period), did not sample the number of calls that would be representative of calls overall, and therefore VCL QM managers could not accurately evaluate the quality of calls answered by VCL responders. Without accurate evaluation of its core business, VCL leadership lacked a process to measure its ability to meet its stated mission of providing “world­ class suicide prevention and crisis intervention services to Veterans, Service Members and their family members.” VHA requested an extension until March 2017, and this recommendation remains open. February 2016 Recommendation 4. We recommended that the OMHO (now Member Services) Executive Director establish a formal quality assurance process, as required by VHA, to identify system issues by collecting, analyzing, tracking, and trending data from the VCL routing system and backup centers and to ensure that subsequent actions are implemented and tracked to resolution. Status Update as of December 15, 2016: We found that the VCL lacked key components of a formal quality assurance process necessary to comply with VHA requirements. Specifically, VCL managers had not designated individuals with appropriate backgrounds and skills to provide leadership to promote quality and safety of care. VCL policies did not incorporate relevant existing VHA directives that provide guidance for successful QM programs. The VCL lacked a committee that regularly reviewed data, information, and risk intelligence to ensure that key quality, safety, and value functions were discussed and integrated into VCL processes. VHA requested an extension until March 2017; thus this recommendation remains open. February 2016 Recommendation 5. We recommended that the OMHO (now Member Services) Executive Director consider the development of a VHA directive or handbook for the VCL. Status Update as of December 15, 2016: In the original response from VHA regarding this recommendation, we received concurrence and VHA leadership stated that they would establish a VHA directive and update the employee handbook. We found that VHA updated the employee handbook and developed but did not complete a VHA directive for operating the VCL (outlining the VCL purpose, roles and responsibilities) as stated in the original response. In December 2016, VHA requested an extension until March 2017 to complete the VHA directive for the VCL operations; thus, this recommendation remains open. February 2016 Recommendation 6. We recommended that the OMHO (now Member Services) Executive Director ensure that contractual arrangements concerning the VCL VA Office of Inspector General 41 Evaluation of the Veterans Health Administration Veterans Crisis Line include specific language regarding training compliance, supervision, comprehensiveness of information provided in contact and disposition emails, and quality assurance tasks. Status Update as of December 15, 2016: VHA established a new contract in April 2016 with the goal of improving QA monitoring of the VCL through a quality assurance surveillance plan. However, VHA did not ensure that the contract contained clear, specific, procedures for an effective quality assurance surveillance plan for VCL backup center performance. VHA staff expressed conflicting information regarding whether training compliance, supervision, and comprehensiveness of information provided in contact and disposition emails was included in contractual arrangements. Eventually VHA concluded that the quality assurance plan was not enforceable and stated plans to remediate this in the next contract, which was set to begin in January 2017. As of the publication of this report, VHA’s plan is to extend the current contract for 6 months and request an extension for implementation of this recommendation until September 2017. Thus, this recommendation remains open. February 2016 Recommendation 7. We recommended that the OMHO (now Member Services) Executive Director consider the development of algorithms or progressive situation-specific stepwise processes to provide guidance in the rescue process. Status Update as of December 15, 2016: In VHA’s original response to this recommendation they concurred and stated that they were developing standard work processes for all caller types, including SSAs’ responsibilities during emergency dispatch. We found that VCL managers have developed standard situation-specific, progressive stepwise processes to provide guidance in the rescue processes. VHA needs to demonstrate the training and competence of SSAs on the current protocol to achieve closure. VHA requested an extension until March 2017, and this recommendation remains open. Objective 4. Office of Special Counsel Complaint The U.S. Office of Special Counsel referred a complaint alleging inadequate training of VCL SSAs that resulted in deficiencies in coordinating immediate emergency services needed to prevent harm. The allegation related to education deficiencies specified that SSAs:  Did not receive appropriate instruction on available systems, procedures, and policies used to facilitate emergency rescue responses.120  Did not receive thorough training on available VCL resources to locate veterans for rescues. 120 VCL staff consider rescues, welfare checks, and dispatch of emergency services to be equivalent terms. VA Office of Inspector General 42 Evaluation of the Veterans Health Administration Veterans Crisis Line  Did not receive instruction on how to properly document call information, which resulted in significant factual discrepancies in VCL call records.  Were not taught how to use VA’s EHR to track suicide prevention referrals.  Did not receive proper instruction on coordinating suicide prevention follow-up with veterans through local VAMCs.  Had, within their developing training program, standard operating procedures that were incorrect and outdated. The OSC complainant also alleged that SSAs:  Who without supervision and regardless of performance or final evaluation. were allowed to coordinate emergency rescue responses independently after the end of a 2-week training period.  Who had not completed training, were allowed to train newly hired SSAs.  Who worked on different shifts handled and recorded emergency rescue responses differently, which made quality assurance efforts difficult. The OSC complainant reported three specific events: a. In mid-2016, a newly trained SSA contacted a caller in crisis to solicit his location by engaging him in conversation, violating VCL protocol, because SSAs did not have the appropriate clinical skills or training to speak directly with callers. b. In mid-2016, a call from a veteran in crisis resulted in conflicting documentation about whether a VCL staff member was able to contact the veteran the following day. The following day, an SSA documenting follow-up entered a code into the VCL's computerized record system indicating that the veteran was reached within 24 business hours. However, after examining the record further, the complainant discovered that the SSA manually entered a note stating VCL staff were unable to reach the veteran. The complainant was ultimately unable to determine which entry was accurate. c. In mid-2016, an SSA closed out a veteran’s case without entering required notes regarding mental health follow-ups from a local VAMC. VCL staff could not determine whether the veteran received appropriate follow-up care. The complainant attributed these issues to the lack of adequate training that SSAs receive when they are newly hired. SSA Training Curriculum. All VCL SSAs are required to complete standardized training provided by VCL instructors. We reviewed the VCL SSA training curriculum which included a list of mandatory orientation and education items to be completed over a 3 to 5 week period. The SSA curriculum includes orientation and training through lectures, computerized instructions, preceptorships, and simulation exercises. In addition, the SSA Trainee Workbook outlined exercises pertinent to the training. The workbook exercises were not mandatory. VCL SSAs were trained with multiple sets of VA Office of Inspector General 43 Evaluation of the Veterans Health Administration Veterans Crisis Line modules under a VCL Lesson Plan Outline presented by teaching staff. Specifically, these modules pertained to:  Standard Work – how consults to SPCs are followed, closed, and coded.  Welfare Checks (emergency rescue responses) and Facility Transportation Plans – proper procedures to facilitate and document Welfare Checks.  Documentation – standard documentation for Welfare checks and Facility Transportation Plans.  Consult Process – using the VCL electronic documentation system and email to process consults.  CAPRI (VA’s EHR) – how to access information.  Searching 101 and 102 – how to conduct a search to locate a veteran in crisis.  Mock Workstation – assemble electronic tools for conducting job duties.  Simulation of a welfare check with obstacles. Instructors, Trainees, and Documentation of Training. We reviewed all seven of the SSA training sheets for SSAs who started NEO training between February and May 2016. We chose this timeframe because SSA training deficiencies were noted in the previous OIG report on the VCL, Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (VA OIG Report No. 14-03540-123, February 11, 2016). We found two of seven SSA training sheets with dates missing for education modules. We found two of seven SSA training sheets without a date of when SSA preceptorships occurred. Both the trainees and instructors initialed and dated each course as they were completed. The supervisor signed off on the training sheets after he/she affirmed that the SSA could work independently. The SSA cannot engage in independent rescues without a supervisor signing off on the entirety of the training. The VCL trainer noted instances where more training was required prior to SSAs’ independent assists with responders. Per the VCL trainer, trainees who have not completed training do not train newer employees. We interviewed three SSAs121 while they were on duty to ascertain their knowledge of their role in a rescue, processes on interventions, accessing databases, SSA documentation requirements, and use of the VA’s EHR. The SSAs gave answers that were consistent with their training modules. We also observed the SSA who was assigned the tasks of assessing SPC follow-up consults, documenting the outcome of the communication with the SPC in VCL’s electronic documentation system, and coding for those consults. The SSA’s actions were consistent with the training module under Standard Work. 121 OIG staff randomly selected the interviewees. VA Office of Inspector General 44 Evaluation of the Veterans Health Administration Veterans Crisis Line We witnessed an SSA help a responder initiate a rescue on a caller with only a name. We observed as the SSA searched through approved databases, but did not find the caller’s current address. We then observed the SSA request help from local police to help with locating the caller. For this call, the caller’s local police would not assist with pinging a cell phone tower to further identify the caller’s location. Ultimately, the Canandaigua local police department provided assistance with the pinging. The Canandaigua pinging request required an approval by a supervisor who was available immediately and approved the request. We did not substantiate the OSC complainant’s allegations that SSAs did not receive appropriate instruction on available systems, procedures, and policies used to facilitate emergency rescue responses. We found procedures within the training manual for initiating Welfare Checks and an algorithm that delineated the Welfare Check process. The available systems for database searches were under the searching modules. All seven SSAs had documented completion of the Welfare Check and Searching modules. We did not substantiate the OSC complainant’s allegations that SSAs did not receive thorough training on available VCL resources to locate veterans for rescues. We found that SSAs were taught how to perform rescues by locating a caller through approved internet searches and received guidance on how to conduct searches when minimal caller information is available. All seven SSAs had documented completion of training in Searching 101 and 102. We did not substantiate the OSC complainant’s allegations that SSAs did not receive instruction on how to properly document call information, which allegedly resulted in significant factual discrepancies in VCL call records. We found that SSAs were trained how to properly document call information under multiple circumstances, including initiation of Welfare Checks, updates to Welfare Checks, and consults to SPCs. All seven SSAs completed the documentation training module. We did not substantiate the OSC complainant’s allegations that SSAs were not taught how to use VHA’s EHR to track suicide prevention referrals. We found that SSAs were instructed on how to use VHA’s EHR and Joint Legacy Viewer to help locate veterans’ contact information. This training module also teaches SSAs where to look for High Risk for Suicide Flags within a veteran’s EHR. Additionally, we found specific instructions on how to access veterans’ scheduled appointments. All seven SSAs had documented completion of EHR training. We did not substantiate the OSC complainant’s allegations that SSAs did not receive proper instruction on coordinating suicide prevention follow up with veterans through local VAMCs. We reviewed education that specifically referenced how local VAMC SPCs are to be contacted via both the VCL’s electronic documentation system and email. We found standardized education for initiating and following consults in the Standard Work and Consult Process modules. All seven SSAs had documented completion of training in coordinating suicide prevention follow up with local VAMCs. VA Office of Inspector General 45 Evaluation of the Veterans Health Administration Veterans Crisis Line We could not substantiate the OSC complainant’s allegations that SSAs had, within their developing training program, standard operating procedures that were incorrect and outdated. Training revisions were periodically updated with new information, although version numbers were not identified. As a result, a manual may be revised but the new information may not be readily identifiable by the reader. The VCL training instructor noted that VCL supervisors share new training via their SharePoint site. However, we noted that there is no mechanism available for an SSA to confirm they read or received distributed updated information. We substantiated the OSC complainant’s allegations that SSAs were allowed to coordinate emergency rescue responses independently after the end of a 2-week training period, without supervision, and regardless of performance or final evaluation. We requested three specific dates for all SSA documentation that involved rescues. The dates chosen were dates on which the seven new SSAs were still in training. We received 106 rescue documents for those three days. We reviewed all rescue documentation authored by SSAs. We found that 24 of the 106 encounters reviewed were documented by SSA trainees who did not have their supervisors sign off on their training sheets. Additionally, we did not find documentation of co-signatures or supervisor signatures that would represent oversight of the SSA trainees during those rescue events. However, we found that the documentation from the SSA trainees was adequate and consistent with established SSA training modules. We did not substantiate the OSC complainant’s allegations that SSAs who had not completed training were allowed to train newly hired SSAs. We reviewed all seven of the training sheets that contained the initials of SSA preceptors training new SSAs. None of the initials corresponded to SSAs that were in training. However, we did find SSA preceptors’ initials with missing dates in two of the seven training sheets. We did not substantiate the OSC complainant’s allegations that on different shifts SSAs handled and recorded emergency rescue responses differently. We reviewed documentation requirements as delineated in the Social Service Assistant Participant Guide. We reviewed those requirements and evaluated SSA documentation for all 106 rescues. We found that SSA documentation contained the required elements of time, date, notification to VAMCs or emergency responders, where the caller was transported, and the final disposition, if known. The documentation was consistent among different SSAs. We did not find that SSAs handled and recorded rescues differently across different shifts. However, SSAs do not have a continuous QA monitoring similar to responders. Without a continuous monitoring process, an evaluation of rescue efforts by SSAs and the ability to improve intervention pathways for callers is impaired. Concerning the three specific events reported by the OSC complainant:  We substantiated that in mid-2016, a newly trained SSA contacted a caller in crisis via telephone to solicit the veteran's location. Management reviewed this incident and on September 12, 2016, issued a written reprimand to the SSA for violating VA Privacy and Security Policy and misrepresenting himself/herself to a VA Office of Inspector General 46 Evaluation of the Veterans Health Administration Veterans Crisis Line veteran. The SSA used VCL information obtained while on duty, contacted the veteran off duty, and represented himself/herself as a member of the Wounded Warriors Project. Because of this case, supervisors reminded VCL staff of the VA Privacy and Security Policy via email and SharePoint site notification. In addition, VCL managers added specific language to the Social Service Assistant Participant Guide stating, “SSAs will never have direct contact with Caller. If further outreach is needed, an SSA will work with a Supervisor to enlist the help of a Responder.” The revised SSA manual was uploaded to the VCL’s SharePoint site in September 2016. The new specific language was in this revised manual. However, the manual cover page was dated May 2016 despite the revision. Further, there was no mechanism within the VCL to verify that all SSAs received this update.  We substantiated the lack of documentation by an SSA in mid-2016. The supervisor counseled the SSA on documentation requirements.    We verified that the documentation concern was not a training issue but a specific personnel concern.  We could not substantiate the allegations that in mid-2016, documentation entered by an SSA resulted in conflicting information on whether a veteran in crisis was contacted within 24 business hours. Because of the complainant’s anonymity, we could not interview the complainant to obtain information that identified the caller, and the VCL’s managers could not locate the record without any caller identifiers. VCL managers described the potential for an SSA to initially document that the veteran could not be contacted, and if that was the case, enter a referral to the SPC. Once the SPC reached the veteran, the coding could be changed to a “contacted” status, resulting in an apparent conflict. Conclusions We substantiated that VCL staff did not respond adequately to a veteran’s urgent needs during multiple calls to the VCL and its backup call centers. VCL supervisory staff did not identify the deficiencies in their internal review of the matter. We identified a deficiency in the VCL’s processes for managing incoming telephone calls. Callers may decide to remain anonymous, but in every case responders document the incoming telephone number. However, responders must manually enter the number into the electronic documentation system, which increases the risks of human error. While reviewing responders’ call documentation, we found that the documentation was often lacking in sufficient detail to facilitate retrospective assessment of the interaction between the caller and responder. We found deficiencies in governance and oversight of VCL operations. In February 2016, the VCL was realigned under Member Services, although VA leadership stated that the VCL would remain closely tethered to the OMHO and VHA Clinical Operations. While the expectation was that Member Services and subject matter VA Office of Inspector General 47 Evaluation of the Veterans Health Administration Veterans Crisis Line experts on suicide prevention would work closely together, we found substantial disagreement about key decisions and oversight between the two groups. The lack of effective utilization of clinical decision makers at the highest level of VCL governance resulted in the failure to fully include clinical perspectives impacting the operations of the VCL. Member Services leaders collected VCL call performance data but did not collect data regarding attempted or completed suicides following a veteran’s contact with the VCL. Because VCL leaders did not collect data regarding attempted or completed suicides following a veteran’s contact with the VCL, they could not demonstrate whether the performance data improved the ability to evaluate the efficacy of the clinical mission. VHA’s OSP leads suicide prevention efforts for VHA and coordinates and disseminates evidence-based findings related to suicide prevention. However, we found a disconnect between the VHA Suicide Prevention Office and Member Services in communicating about suicide prevention and the VCL. VCL call complaint data included callers’ complaints about being on hold. We found that some contracted backup call centers used a queuing (waiting) process that callers may perceive as being on hold. During the queue time, or wait time, the caller waits for a responder to answer. The caller’s only option is to abandon the call (hang up) and call back, or continue to wait for a responder to pick up. The backup centers had processes to record wait times and abandonment rates. We found that VCL leadership had not established expectations or targets for queued call times, or thresholds for taking action on queue times, resulting in a systems deficiency for addressing these types of complaints. VHA contracted with an external vendor (L2HS) to manage backup center performance and report back to the VCL, with administrative and clinical oversight of the contract terms by VCL managers. We found that the VHA contracting staff and Member Services and VCL leaders responsible for verifying and enforcing terms of the contract did not provide the necessary oversight and did not validate that L2HS provided the required services before authorizing payment. We found systems deficiencies in the Quality Management program. VHA provides a framework for QM program structure and leadership to ensure delivery of safe and effective care; however, VCL QM staff lacked background or training in quality management principles and processes, leading to deficiencies in quality management. VCL QM managers collected data for clinical performance measures but lacked a reporting structure for communication of the data analysis, development of action plans, or follow-up assessments for process improvement. VCL QM managers did not collect data for caller outcomes, such as follow-up on SPC referrals or serious adverse outcomes like suicide attempts or completions following contact with the VCL, in order to measure quality improvement. VCL policies were not consistent with existing VHA policies for veteran safety or risk management. The VCL could enhance performance improvement evaluations by using call recording to monitor the quality of interactions between responders and callers and VA Office of Inspector General 48 Evaluation of the Veterans Health Administration Veterans Crisis Line by collecting and analyzing performance data from the new Atlanta Call Center separately from the Canandaigua Call Center. The new call center in Atlanta could potentially have QM concerns that are no different from their Canandaigua partner, but the ability to recognize site-specific issues, especially in a new program, may be facilitated by separating quality data elements by site. During our review of the allegations referred by the OSC, we found that VCL managers developed a monitoring process for responders; however, they lacked a process for monitoring the quality of performance by SSAs. We could not determine how VCL policies are disseminated to staff when VCL policies were changed, and VCL managers lacked the ability to monitor whether staff received updated policies. We substantiated the OSC complainant’s allegations that SSAs were allowed to coordinate emergency rescue responses independently after the end of a 2-week training period, without supervision and regardless of performance or final evaluation. We requested three specific dates for all SSA documentation that involved rescues. The dates chosen were dates when the seven new SSAs were still in training. We noted that 24 of the 106 encounters we reviewed were documented by SSA trainees who did not have their supervisors sign off on their training sheets. Additionally, we did not find documentation of co-signatures or supervisor signatures that would represent oversight of the SSA trainees during those rescue events. We substantiated the OSC complainant’s allegations that in mid-2016, a newly trained SSA contacted a caller in crisis by telephone to solicit the veteran's location. We found that no harm resulted from the interaction. VCL management reviewed this event, confirmed a violation of VCL policy, and took administrative action against the SSA. Because of this case, supervisors reminded VCL staff of the VA Privacy and Security Policy via email and SharePoint website notification. We substantiated the OSC complainant’s allegations regarding a lack of documentation by an SSA when closing out a veteran’s case in mid-2016. A supervisor counseled the SSA on documentation requirements. We verified that the documentation concern was not a training issue but a specific personnel concern. We could not substantiate the OSC complainant’s allegations that in mid-2016, documentation entered by an SSA resulted in conflicting information on whether a veteran in crisis was contacted within 24 business hours. Because of the complainant’s anonymity, we could not interview the complainant to obtain information that identified the caller, and the VCL’s managers could not locate the record without any caller identifiers. VCL managers described the potential for an SSA to initially document that the veteran could not be contacted, and if that was the case, enter a referral to the SPC. Once the SPC reached the veteran, the coding could be changed to a “contacted” status, resulting in an apparent conflict. In the report we published previously, Healthcare Inspection–Veterans Crisis Line Caller Response and Quality Assurance Concerns Canandaigua, New York (Report No. 14­ 03540-123, February 11, 2016), we made seven recommendations. In response to OIG VA Office of Inspector General 49 Evaluation of the Veterans Health Administration Veterans Crisis Line recommendations for improvement, VHA agreed to a completion deadline of September 30, 2016. All recommendations remain open as of the publication of this report. Recommendations 1. We recommended that the Under Secretary for Health implement an automated transcription function for callers’ phone numbers in the Veterans Crisis Line call documentation recording system. 2. We recommended that the Under Secretary for Health ensure that Veterans Crisis Line policies and procedures, staff education, Information Technology support, and monitoring are in place for audio call recording. 3. We recommended that the Under Secretary for Health implement a Veterans Crisis Line governance structure that ensures cooperation and collaboration between VHA Member Services and the Office of Suicide Prevention. 4. We recommended that the Under Secretary for Health develop clear guidelines that delineate clinical and administrative decision-making, assuring that clinical staff make decisions directly affecting clinical care of veterans in accordance with sound clinical practice. 5. We recommended that the Under Secretary for Health ensure processes are in place for routinely reviewing backup center data, establish wait-time targets for call queuing and rollover, and ensure plans are in place for corrective action when wait-time targets are exceeded. 6. We recommended that the Under Secretary for Health ensure processes are in place to require contracted backup centers to have the same standards as the Veterans Crisis Line related to call queuing and wait-time targets. 7. We recommended that the Under Secretary for Health ensure that VHA Member Services leadership, Veterans Crisis Line leadership, VHA Contracting Officers, and Contracting Officer Representatives implement the quality control plan and conduct ongoing oversight to ensure contractor accountability in accordance with their roles as specified in the contract with backup call centers. 8. We recommended that the Under Secretary for Health ensure that training is provided to Veterans Crisis Line quality management staff in the skills needed to provide leadership to promote quality and safety of care. 9. We recommended that the Under Secretary for Health ensure the development of structured oversight processes for tracking, trending, and reporting of clinical quality performance measures. VA Office of Inspector General 50 Evaluation of the Veterans Health Administration Veterans Crisis Line 10. We recommended that the Under Secretary for Health ensure processes for Veterans Crisis Line quality management staff to collect and review adverse outcomes so that established cohorts of severe adverse outcomes are analyzed. 11. We recommended that the Under Secretary for Health direct the Veterans Health Administration Assistant Deputy Under Secretary for Health for Quality, Safety, and Value to review existing Veterans Crisis Line policies and determine whether the policies incorporate the appropriate Veterans Health Administration policies for veteran safety and risk management, and if not, establish appropriate action plans. 12. We recommended that the Under Secretary for Health ensure that Veterans Crisis Line quality management staff incorporate call audio recording into quality management data analysis. 13. We recommended that the Under Secretary for Health ensure that processes are in place to analyze performance and quality data from the Atlanta Call Center separately from the Canandaigua Call Center data. 14. We recommended that the Under Secretary for Health ensure that quality assurance monitoring policies and procedures are in place and consistent for both Social Service Assistants and responders. 15. We recommended that the Under Secretary for Health ensure that supervisors certify Social Service Assistant training prior to engaging in independent assistance with rescues. 16. We recommended that the Under Secretary for Health ensure a process is in place to establish, maintain, distribute, and educate staff on all Veterans Crisis Line policies and directives that includes verifying the use of current versions when policies and directives are modified. VA Office of Inspector General 51 Evaluation of the Veterans Health Administration Veterans Crisis Line Appendix A Timeline: History of VCL, Chronology of OIG Reviews, and Publication of GAO Report122 Timeframe Event 2007 VCL opened under Office of Suicide Prevention in VHA 2009 Chat feature added to VCL services 2010 Mental Health Services and OMHO separated. VCL was placed under OMHO. However, the Suicide Prevention Program continued to be heavily involved in oversight. 2011 Name changed to Veterans Crisis Line 2011 Text feature added to VCL services 2013 Mobile website launched 2014 VCL moved fully under OMHO in VHA. Suicide Prevention Program no longer involved with oversight. February 2016 VCL moved to Member Services in VHA February 2016 OIG report published with seven recommendations for VCL May 2016 GAO report published with recommendations for VCL June 2016 OIG receives Hotline allegation regarding a veteran who called the VCL June 8–9, 2016 OIG fact-finding visit to VCL in Canandaigua June 15, 2016 OIG hotline officially opened August 9–10, 2016 Announced OIG inspection/site visit in Canandaigua September 6–8, 2016 Announced OIG inspection/site visit in Canandaigua September 20–21, 2016 Announced OIG inspection/site visit in Atlanta Source: VAOIG 122 The VCL was originally called The Veterans Suicide Prevention Line.  VA Office of Inspector General 52 Evaluation of the Veterans Health Administration Veterans Crisis Line Appendix B Veterans Crisis Line Organizational Chart June 2016 VA Office of Inspector General 53 Evaluation of the Veterans Health Administration Veterans Crisis Line Appendix C Veterans Crisis Line Organizational Chart July 2016 VA Office of Inspector General 54 Evaluation of the Veterans Health Administration Veterans Crisis Line Appendix D Veterans Crisis Line Organizational Chart September 2016 VA Office of Inspector General 55 Evaluation of the Veterans Health Administration Veterans Crisis Line Appendix E Two Examples of Data Analysis Deficiencies for Silent Monitoring Table 5. Silent Monitors for Quality Assurance Monthly Report June 2016 Critical Items, Percent Unsuccessful  (not mutually exclusive) 200 180 160 140 120 100 80 60 40 20 0 Reduced Means Offered SPC Consult Safety Plan 3rd party outreach CAPRI/JLV Assessed Suicide Assessed Past Suicide Satisfaction Ques Source: VHA VCL Table 6. Silent Monitors for Quality Assurance Monthly Report June 2016 Non‐Critical Items Unsuccessful  (not mutually exclusive) 70 60 50 40 30 20 10 0 Source: VHA VCL VA Office of Inspector General 56 Evaluation of the Veterans Health Administration Veterans Crisis Line Appendix F Acting Under Secretary for Health Comments Department of Veterans Affairs Date: From: Subj: To: Memorandum March 14, 2017 Acting Under Secretary for Health (10) OIG Draft Report Healthcare Inspection – Evaluation of the Veterans Health Administration Veterans Crisis Line (7769885) Assistant Inspector General for Healthcare Inspections (54) Director, Management Review Service (VHA 10E1D MRS Action) 1. Thank you for the opportunity to review the Office of Inspector General (OIG) draft report, Healthcare Inspection Evaluation of the Veterans Health Administration Veterans Crisis Line (VCL). I concur with the draft report content and OIG’s 16 recommendations. I provide the attached action plan to address all recommendations. 2. The VCL mission is to provide 24/7 world-class suicide prevention and crisis intervention services to Veterans, Service members and their family. Since 2007, VCL has answered nearly 2.6 million calls, and dispatched emergency services to callers in crisis over 67,000 times. Since launching chat in 2009 and text services in November 2011, the VCL has answered nearly 314,000 requests for chat and nearly 62,000 requests for text services. Staff has forwarded over 416,000 referrals to local VA Suicide Prevention Coordinators on behalf of Veterans to ensure continuity of care with Veterans local VA providers. 3. The VCL is the strongest it’s ever been since its inception in 2007. VA is making notable advances to improve access and quality of service to mental health crisis care for Veterans which is why we’ve opened the new Atlanta satellite office. The office officially opened in October 2016 with more than 200 new employees and continues to grow. VCL has implemented a comprehensive workforce management system and optimized staffing patterns. This will provide callers with immediate service and achieve zero percent routine rollover to contracted back-up centers. VA Office of Inspector General 57 Evaluation of the Veterans Health Administration Veterans Crisis Line 4. If you have any questions, please email Karen Rasmussen, M.D., Director, Management Review Service at VHA10E1MRSAction@va.gov. VA Office of Inspector General 58 Evaluation of the Veterans Health Administration Veterans Crisis Line Comments to OIG’s Report The following comments are submitted in response to the recommendations in the OIG report: OIG Recommendations Recommendation 1. We recommended that the Under Secretary for Health implement an automated transcription function for callers’ phone numbers in the Veterans Crisis Line call documentation recording system. Concur Target date for completion: December 2017 Facility response: This recommendation is related to GAO High Risk Area 1 (ambiguous policies and inconsistent processes). The automated transcription function will decrease the risk of human error when recording a caller’s phone number and increase the responder’s ability to call them back if the call gets interrupted. The Veterans Health Administration (VHA) concurs with this recommendation and will work to incorporate our new Customer Relationship Management (CRM) system into our phone system so that call records are automatically populated with the phone number of the caller. In the short term, Veterans Crisis Line (VCL) leadership will remind staff to always ask for two phone numbers to reach the Veteran. VCL will direct Social Service Assistants (SSA) to resolve any discrepancies between the phone numbers collected by responders and the phone numbers in the Veteran’s electronic medical record, when the caller has a medical record within the VA system. At completion of this action, VCL will provide OIG with the following documentation: 1. Demonstration of the automated transcription function for callers’ phone numbers; 2. Evidence of communication to VCL staff of the two-phone-number requirement; and 3. Evidence of communication to SSAs requiring they resolve phone number discrepancies. Recommendation 2. We recommended that the Under Secretary for Health ensure that Veterans Crisis Line policies and procedures, staff education, Information Technology support, and monitoring are in place for audio call recording. Concur Target date for completion: October 2017 Facility response: This recommendation is related to GAO High Risk Areas 1 (ambiguous policies and inconsistent processes), 2 (inadequate oversight and VA Office of Inspector General 59 Evaluation of the Veterans Health Administration Veterans Crisis Line accountability), 3 (information technology challenges) and 4 (inadequate training for VA staff). To improve operations VCL will: 1. Evaluate policies and procedures related to VCL call recordings; 2. Work with Department of Veterans Affairs Office of Information and Technology to provide information technology (IT) support to both VCL-Atlanta and VCLCanandaigua; 3. Educate all staff on call recording policies to include roles and responsibilities; 4. Obtain input from relevant stakeholders on revisions to current policy; and 5. Develop and implement a training plan for educating staff on the use of call recordings and how to walk a caller through any concerns that the call is being recorded. At completion of this action, VCL will provide OIG with the following documentation: 1. Findings from the policy and procedure; 2. Evidence that all staff have been educated; 3. Report on the input from stakeholders regarding policy; 4. Approved training plan and evidence of implementation; and 5. Report on decisions regarding IT support for VCL-Atlanta and VCL-Canandaigua. Recommendation 3. We recommended that the Under Secretary for Health implement a Veterans Crisis Line governance structure that ensures cooperation and collaboration between VHA Member Services and the Office of Suicide Prevention. Concur Target date for completion: May 2017 Facility response: The Office of the Deputy Under Secretary for Health for Operations and Management and the Office of the Deputy Under Secretary for Health for Policy and Services will collaborate to establish a governance structure that ensures cooperation and collaboration between Members Services (MS) and Office of Suicide Prevention (OSP). The new governance structure will determine whether revisions are needed to the organizational chart and will ensure that leadership in OSP, MS, and VCL have appropriate oversight responsibility for clinical and administrative functions. At completion of this action, VCL will provide OIG with the approved governance structure. Recommendation 4. We recommended that the Under Secretary for Health develop clear guidelines that delineate clinical and administrative decision-making, assuring that clinical staff make decisions directly affecting clinical care of veterans in accordance with sound clinical practice. Concur VA Office of Inspector General 60 Evaluation of the Veterans Health Administration Veterans Crisis Line Target date for completion: June 2017 Facility response: OSP and VCL will collaboratively develop high-level guidelines that delineate which office is responsible for clinical and administrative decision-making. These guidelines will clearly define roles and responsibilities. These guidelines will focus on ensuring Veterans who call receive high quality care based on clinical judgement and calls are managed with sound business operations. At completion of this action, VCL will provide OIG with: 1. The clinical and administrative decision-making guidelines; and 2. Evidence that the guidelines have been appropriately distributed Recommendation 5. We recommended that the Under Secretary for Health ensure processes are in place for routinely reviewing backup center data, establish wait-time targets for call queuing and rollover, and ensure plans are in place for corrective action when wait-time targets are exceeded. Concur Target date for completion: October 2017 Facility response: VCL will develop a process to review to ensure backup centers are meeting predefined metrics for wait-time targets for call queuing and rollover. VCL will ensure plans are in place for corrective action when wait time targets are exceeded. At completion of this action, VCL will provide OIG with the following documentation: 1. Three months of reports of back-up center data which demonstrates compliance with wait-time targets for call queuing and rollover; and 2. Corrective action plans if wait-time targets are exceeded. Recommendation 6. We recommended that the Under Secretary for Health ensure processes are in place to require contracted backup centers to have the same standards as the Veterans Crisis Line related to call queuing and wait-time targets. Concur in principle Target date for completion: October 2017 Facility response: VHA concurs in principle with this recommendation because while it would be preferable, currently no suicide call center in the country is able to achieve the current VA standards for call queueing and wait time targets. VCL will measure contract call centers against VHA’s metrics, however, VHA will not require that the call centers achieve these metrics, nor will VHA penalize call centers for failing to meet these metrics. VCL will continue to strive for handling all calls within our own call centers and will make every effort to contract only with those call centers that demonstrate responsible call queuing and wait times. It is absolutely imperative for VCL to have VA Office of Inspector General 61 Evaluation of the Veterans Health Administration Veterans Crisis Line contracts with back up call centers, especially in the event of an unanticipated event that might impair VCL from providing its services – such as a technological failure or national catastrophe. As with all contingency plans, the ability for the back-up system to achieve the same standards as the fully staffed system, is not a reasonable expectation. VCL will award a new contract in 2017 that formalizes the existing contract with backup centers which ensures VCL calls are answered first and prioritizes Veterans in need. At completion of this action, VCL will provide OIG with a copy of the backup center contract. Recommendation 7. We recommended that the Under Secretary for Health ensure that VHA Member Services leadership, Veterans Crisis Line leadership, VHA Contracting Officers, and Contracting Officer Representatives implement the quality control plan and conduct ongoing oversight to ensure contractor accountability in accordance with their roles as specified in the contract with backup call centers. Concur Target date for completion: October 2017 Facility response: VCL leadership will implement a quality control plan ensuring contractors adhere to their roles specified in the contract. VCL leadership will improve and increase efforts to document contractor’s performance. VCL’s Executive Leadership Committee (ELC) will be responsible for overseeing and documenting contractor performance and corrective actions. In addition, VCL will: 1. Contact the contractor immediately with concerns, document those concerns and document corrective actions to include due dates for completion; 2. Hold monthly calls with contractors to review performance; 3. Complete quarterly reports of contractor’s performance; and 4. Document the concerns, and changes made to address the concerns in the ELC meetings. At completion of this action, VCL will provide OIG with the following documentation: 1. 2. 3. 4. A copy of the quality control plan; Minutes from three meeting minutes from ELC meetings; A copy of the quarterly report on contractor performance; and Two months of minutes from monthly meetings with contractor. Recommendation 8. We recommended that the Under Secretary for Health ensure that training is provided to Veterans Crisis Line quality management staff in the skills needed to provide leadership to promote quality and safety of care. Concur VA Office of Inspector General 62 Evaluation of the Veterans Health Administration Veterans Crisis Line Target date for completion: August 2017 Facility response: VHA’s Office of Quality Safety and Value (QSV) (including National Center for Patient Safety (NCPS) will partner with VCL to train VCL management staff in core competencies of safe and high quality leadership. Existing curricula (Just Culture and Clinical Team Training modules) will be modified for VCL which include a complement of both leadership and followership communication skills and an emphasis on a non-punitive reporting culture that enables staff to escalate concerns. At completion of this action, QSV and VCL will provide OIG with the following documentation: 1. Examples of the curriculum and evidence that this training was completed, along with evaluations of the training from participants; and 2. Evidence that the appropriate VCL staff underwent the modified training required by QSV. Recommendation 9. We recommended that the Under Secretary for Health ensure the development of structured oversight processes for tracking, trending, and reporting of clinical quality performance measures. Concur Target date for completion: October 2017 Facility response: This recommendation is related to GAO High Risk Area 2 (inadequate oversight and accountability). The VCL ELC is the governance structure responsible for documenting, tracking, and directing action on clinical quality performance measures. The following will be added to the VCL ELC’s responsibilities: 1. The ELC will require VCL Quality/Safety committee to report monthly to the ELC; 2. All concerns, actions and changes made to VCL actions are to be documented in the ELC meeting minutes; 3. Team Operations Managers will update the ELC on the progress of implementing these changes; and 4. All clinical quality data, concerns, corrective actions will be communicated via documentation to Office of Suicide Prevention which will provide feedback that will be incorporated into ELC meetings as needed. At completion of this action, VCL will provide OIG with the following documentation: 1. Three sets of ELC Meeting minutes demonstrating tracking, trending, and reporting of clinical quality performance measures; and 2. The Quality/Safety Committee charter. Recommendation 10. We recommended that the Under Secretary for Health ensure processes for Veterans Crisis Line quality management staff to collect and review VA Office of Inspector General 63 Evaluation of the Veterans Health Administration Veterans Crisis Line adverse outcomes so that established cohorts of severe adverse outcomes are analyzed. Concur Target date for completion: October 2017 Facility response: VCL will implement a root cause analysis process for adverse outcomes. The results of this process will be reviewed as part of the ELC agenda and by OSP as part of an existing root cause analysis process. This information will be brought to the appropriate operations division to remediate any issues discovered by the root cause analysis. VCL, in collaboration with OSP will develop new guidelines and processes as needed. VCL and OSP will consult with QSV as needed. At completion of this action, VCL will provide OIG with the following documentation: 1. The documented root cause analysis process; and 2. Results of a root cause analysis Recommendation 11. We recommended that the Under Secretary for Health direct the Veterans Health Administration Assistant Deputy Under Secretary for Health for Quality, Safety, and Value to review existing Veterans Crisis Line policies and determine whether the policies incorporate the appropriate Veterans Health Administration policies for veteran safety and risk management, and if not, establish appropriate action plans. Concur Target date for completion: October 2017 Facility response: This recommendation is related to Government Accountability Office (GAO) High Risk Areas 1 (ambiguous policies and inconsistent processes). QSV (including NCPS) will partner with VCL leadership to review their existing policies and determine methods by which Patient Safety and Risk Management processes may be integrated into existing VCL processes and procedures. At completion of this action, NCPS will provide OIG with documentation of how safety and risk approaches have been integrated into existing VCL policies. Recommendation 12. We recommended that the Under Secretary for Health ensure that Veterans Crisis Line quality management staff incorporate call audio recording into quality management data analysis. Concur Target date for completion: July 2017 VA Office of Inspector General 64 Evaluation of the Veterans Health Administration Veterans Crisis Line Facility response: VCL will incorporate the new function of recorded calls into its quality management and data analysis. VCL, in consultation with QSV, will ensure that best practices are utilized in this process of quality management and data analysis. Additionally, all leadership staff will be trained on the new AVAYA system and the training will be documented. The VCL Quality Management manual will explicitly discuss how and when recorded calls are to be used. In conjunction with recommendation 2, the VCL will also: 1. Identify roles and responsibilities for IT support of call recording; 2. Finalize the Standard Operating Procedure for Quality Management Silent Monitoring and their use of recorded calls; and 3. Edit the form that is submitted by VCL Quality Management staff to include a field for if a recorded call was used in the evaluation. At completion of this action, VCL will provide OIG with the following documentation: 1. A copy of the standard operating procedure (to include identification of roles and responsibilities of IT support); 2. A copy of standard operating procedure for Quality Management use of recorded calls for silent monitoring; and 3. A copy of two monthly data reports to show that call recording was used in quality management evaluation. Recommendation 13. We recommended that the Under Secretary for Health ensure that processes are in place to analyze performance and quality data from the Atlanta Call Center separately from the Canandaigua Call Center data. Concur Target date for completion: May 2017 Facility response: VCL will ensure that quality data for the Canandaigua and Atlanta call centers is documented both individually and combined. At completion of this action, VCL will provide OIG with the following documentation: 1. One quarter analysis of quality improvement data individually and combined for Atlanta and Canandaigua; and 2. Three sets of ELC minutes that will demonstrate leadership involvement and oversight. Recommendation 14. We recommended that the Under Secretary for Health ensure that quality assurance monitoring policies and procedures are in place and consistent for both Social Service Assistants and responders. Concur VA Office of Inspector General 65 Evaluation of the Veterans Health Administration Veterans Crisis Line Target date for completion: December 2017 Facility response: We appreciate OIG’s recommendation for improving consistency across SSAs and responders with respect to quality assurance monitoring. VCL currently monitors responders through a robust silent monitoring process. We feel this has been effective at identifying quality variances in our responders’ performance. VCL will apply the same or similar silent monitoring process to SSA calls with dispatch and emergency services. VCL will update the office’s process documents to ensure internal policy on quality assurance monitoring is consistent for both employee positions. To implement silent monitoring for SSAs that is consistent with silent monitoring for responders requires expanding technologies associated with phone recording. VCL will need to determine what additional resources will be needed to expand these technologies. The one year target completion date takes into account the potential for additional budget allocations, potential IT involvement, and a reasonable period of data collection. At completion of this action, VCL will provide OIG with the following documentation: 1. Updated VCL office process document (e.g., Standard Operating Procedure) 2. Evidence of silent monitoring for SSAs and if any deficiencies are noted, examples of corrective actions. Recommendation 15. We recommended that the Under Secretary for Health ensure that supervisors certify Social Service Assistant training prior to engaging in independent assistance with rescues. Concur Target date for completion: October 2017 Facility response: VCL will develop a standard form for supervisors to certify training has been completed by the SSA. VCL will develop a training tracking system for all areas of competency, including required training needed to engage in independent rescue work. At completion of this action, VCL will provide OIG with the following documentation: 1. The training completion certification form used by supervisors; and 2. Evidence that all SSAs are compliant with training requirements. Recommendation 16. We recommended that the Under Secretary for Health ensure a process is in place to establish, maintain, distribute, and educate staff on all Veterans Crisis Line policies and directives that includes verifying the use of current versions when policies and directives are modified. Concur VA Office of Inspector General 66 Evaluation of the Veterans Health Administration Veterans Crisis Line Target date for completion: October 2017 Facility response: VCL will develop a process which will outline policy establishment, updates, staff education, implementation, as well as leadership involvement and oversight. At completion of this action, VCL will provide OIG with a copy of the process that was developed outlining the use of policies, updates to policies and identification of policy modifications and evidence that the process is effective. VA Office of Inspector General 67 Evaluation of the Veterans Health Administration Veterans Crisis Line Appendix G OIG Contact and Staff Acknowledgments Contact Contributors For more information about this report, please contact the OIG at (202) 461-4720. Laura Tovar, MSW, LSCSW, Team Leader Lisa Barnes, MSW Gail Bozzelli, RN Cynthia Cavalieri Kimberly Choplin Myra Conway, RN, MS Stacy S. Depriest, LCSW David Engelen Katharine Foster, RN Lindsay Gold, LCSW Martha Kearns, FNP, MSN Robin Moyer, MD, MBA Marci Mylan, PhD, MHA Bruce Nielson, JD Laura Snow, LCSW, MHCL Jolynette Spearman Monika Spinks, RN Marilyn Stones, BS Jennifer Tinsley, LMSW-C Ann Ver Linden, RN, MBA Andrew Waghorn, JD Joanne Wasko, LCSW George Wesley, MD Michelle Wilt, BSN, MBA Thomas Wong, DO The VA Office of Inspector General would also like to acknowledge the generous assistance of the staffs of Didi Hirsch Mental Health Services, Los Angeles, California; Boys Town and the Boys Town National Hotline®, Boys Town, Nebraska; and the National Domestic Violence Hotline, Austin, Texas. VA Office of Inspector General 68 Evaluation of the Veterans Health Administration Veterans Crisis Line Appendix H Report Distribution VA Distribution Office of the Secretary Veterans Health Administration Assistant Secretaries General Counsel Director, VHA Member Services (10NF) Director, Veterans Crisis Line (10NF3) Non-VA Distribution House Committee on Veterans’ Affairs House Appropriations Subcommittee on Military Construction, Veterans Affairs, and Related Agencies House Committee on Oversight and Government Reform Senate Committee on Veterans’ Affairs Senate Appropriations Subcommittee on Military Construction, Veterans Affairs, and Related Agencies Senate Committee on Homeland Security and Governmental Affairs National Veterans Service Organizations Government Accountability Office Office of Management and Budget U.S. Senate: Bill Nelson This report is available on our web site at www.va.gov/oig. VA Office of Inspector General 69