United States Government Accountability Office Report to the Chairman, Committee on Veterans’ Affairs, House of Representatives October 2018 VA DISABILITY EXAMS Improved Performance Analysis and Training Oversight Needed for Contracted Exams GAO-19-13 October 2018 VA DISABILITY EXAMS Improved Performance Analysis and Training Oversight Needed for Contracted Exams Highlights of GAO-19-13, a report to the Chairman, Committee on Veterans' Affairs, House of Representatives Why GAO Did This Study What GAO Found In 2016, VBA awarded 12 contracts to five private firms for up to $6.8 billion lasting up to 5 years to conduct veterans’ disability medical exams. Both VBA contracted medical examiners and medical providers from the Veterans Health Administration perform these exams, with a growing number of exams being completed by contractors. Starting in 2017, VBA contracted examiners conducted about half of all exams. GAO was asked to review the performance and oversight of VBA’s disability medical exam contractors. The Veterans Benefits Administration (VBA) has limited information on whether contractors who conduct disability compensation medical exams are meeting the agency’s quality and timeliness targets. VBA contracted examiners have completed a growing number of exams in recent years (see figure). VBA uses completed exam reports to help determine if a veteran should receive disability benefits. VBA reported that the vast majority of contractors’ quality scores fell well below VBA’s target—92 percent of exam reports with no errors—for the first half of 2017. Since then, VBA has not completed all its quality reviews, but has hired more staff to do them. VBA officials acknowledged that VBA also does not have accurate information on contractor timeliness. VBA officials said the exam management system used until spring 2018 did not always retain the initial exam report completion date, which is used to calculate timeliness. In spring 2018, VBA implemented a new system designed to capture this information. This report examines (1) what is known about the quality and timeliness of VBA contracted exams; (2) the extent to which VBA monitors contractors’ performance; and (3) how VBA ensures that its contractors provide qualified and well-trained examiners. GAO analyzed the most recent reliable data available on the quality and timeliness of exams (January 2017 to February 2018), reviewed VBA and selected contract documents and relevant federal laws and regulations, and interviewed agency officials, exam contractors, an audit firm that checks examiners’ licenses, and selected veterans service organizations. What GAO Recommends GAO recommends VBA (1) develop a plan for using its new data system to monitor contractors’ quality and timeliness performance, (2) analyze overall program performance, (3) verify that contracted examiners complete required training, and (4) collect information to assess the effectiveness of that training. The Department of Veterans Affairs agreed with GAO’s recommendations. View GAO-19-13. For more information, contact Elizabeth Curda at (202) 512-7215 or curdae@gao.gov. Disability Compensation Medical Exams Completed by Contractors, Fiscal Years 2012-2018 VBA monitoring has addressed some problems with contractors, such as reassigning exams from contractors that did not have enough examiners to those that did. However, the issues GAO identified with VBA’s quality and timeliness information limit VBA’s ability to effectively oversee contractors. For example, VBA officials said they were unable to track the timeliness of exam reports sent back to contractors for corrections, which is needed to determine if VBA should reduce payment to a contractor. The new system implemented in spring 2018 tracks more detailed data on exam timeliness. However, VBA has not documented how it will ensure the data are accurate or how it will use the data to track the timeliness and billing of corrected exam reports. VBA also has no plans to use the new system to analyze performance data to identify trends or other program-wide issues. Without such plans, VBA may miss opportunities to improve contractor oversight and the program overall. A third-party auditor verifies that contracted examiners have valid medical licenses, but VBA does not verify if examiners have completed training nor does it collect information to assess training effectiveness in preparing examiners. While VBA plans to improve monitoring of training, it has not documented plans for tracking or collecting information to assess training. These actions could help ensure that VBA contractors provide veterans with high-quality exams and help VBA determine if additional training is needed. United States Government Accountability Office Contents Letter 1 Background VBA Reported Contractors Missed Exam Quality Targets, and VBA Could Not Accurately Measure Performance On Timeliness Targets Delayed Quality Reviews and Performance Reports, and Data Limitations, Hinder VBA’s Monitoring of Contractors Auditor Verifies Contracted Examiner Licenses, but VBA Does Not Verify Training Completion or Collect Information on Training Effectiveness Conclusions Recommendations for Executive Action Agency Comments and Our Evaluation 4 23 26 27 28 Appendix I Additional Information on Selected Methodologies 30 Appendix II Comments from the Department of Veterans Affairs 35 Appendix III GAO Contact and Staff Acknowledgments 38 Table 1: VBA Quarterly Performance Targets for Quality and Timeliness for Individual Contractors Completing Exams for Veterans’ Disability Compensation Claims 10 11 16 Table Figures Figure 1: Overview of VBA Disability Compensation Claims Process and Contractor Process for Completing Exams Figure 2: Overview of Current VBA Exam Contracts for Veterans’ Disability Compensation Claims Figure 3: VBA-Reported Quarterly Contractor Quality Scores for Exams for Veterans’ Disability Compensation Claims Figure 4: Number of Days Taken to Complete Contracted Exams for Veterans’ Disability Compensation Claims, February 2017 to January 2018 Page i 5 7 12 14 GAO-19-13 VA Disability Exams Abbreviations CAATS COR VA VBA VHA Centralized Administrative Accounting Transaction System Contracting Officer’s Representative Department of Veterans Affairs Veterans Benefits Administration Veterans Health Administration This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-19-13 VA Disability Exams Letter 441 G St. N.W. Washington, DC 20548 October 12, 2018 The Honorable David P. Roe Chairman Committee on Veterans’ Affairs House of Representatives In fiscal year 2017, the Department of Veterans Affairs (VA) paid about $72.4 billion to about 4.5 million veterans to compensate them for disabilities caused or aggravated by their military service. In that same fiscal year, the Veterans Benefits Administration’s (VBA) 57 regional offices processed more than 1.2 million veterans’ disability compensation claims to determine eligibility for benefits. To help determine this eligibility, VBA staff may request that the veteran undergo a medical examination to provide evidence of disabilities and their connection to military service. 1 Medical providers who work for the Veterans Health Administration (VHA) often conduct these exams. VBA also contracts with private firms to perform these exams. 2 According to VA officials, the demand for exams sometimes exceeds VHA’s capacity to perform them, and in these cases VBA assigns a contractor to perform the exam to avoid delays in processing the claim. In 2016, VBA awarded 12 exam contracts to five private firms for up to $6.8 billion lasting up to 5 years. The contracts cover exams conducted in both the United States and overseas, and cover benefit claims processed by VBA regional offices. In fiscal year 2017, VA spent $765 million on exams conducted by these VBA contractors. About 767,000 veterans received a total of about 1 million exams from VBA contracted examiners from January 1, 2017 to April 2018, which is about half of all disability exams during this time. 1 In this report, we refer to examinations as exams. 2 In general, VHA examiners are medical providers who are VHA employees. In contrast, examiners under VBA contracts are often sub-contractors or employees of the firms with which VBA has contracted. For purposes of this report, when referring to individual medical providers who perform exams under VBA contracts, we will use the term contracted examiners. When referring to individual VHA medical providers who perform exams, we will use the term VHA examiners or VHA providers. When referring to the firms who hold the contracts with VBA, we will use the term contractors. Page 1 GAO-19-13 VA Disability Exams You asked us to review VA’s oversight of the contract disability exam program to ensure the quality of disability exams for veterans. This report examines (1) what is known about the quality and timeliness of VBA contracted exams; (2) the extent to which VBA monitors contractors’ performance to ensure that they provide high quality and timely exams; and (3) how VBA ensures that its contractors provide qualified and welltrained examiners. To describe the quality of VBA contracted exams, we reviewed available scores and results of VBA’s quarterly quality reviews of exam reports completed in calendar year 2017 for each contract. 3 VBA has not finalized contractors’ quality scores and results for exam reports completed in calendar year 2017—particularly for reports completed in the last two quarters of that year. To describe the timeliness of contracted exams, we analyzed VBA data on exams completed between February 2017 and January 2018. 4 VBA’s timeliness data were sufficient for our purpose of calculating the total time it took to complete exams (see appendix I for additional details on our methodology). 5 In addition, we reviewed VBA aggregate data on the number of exam requests that had been accepted by contractors but not yet completed as of late June 2018. For context, we also reviewed VHA aggregate data on the quality and timeliness of exams conducted by VHA examiners during calendar year 2017. 6 We reviewed VBA and VHA documentation of methodology and data collection procedures, and interviewed officials knowledgeable about the systems used to collect and maintain the data. For example, we spoke with officials about how VBA uses available data to assess contractor 3 We use the term exam reports when referring to individual disability benefit questionnaires—the forms that both VHA examiners and VBA contracted examiners complete as part of their assessment of each medical condition claimed by the veteran. 4 Our period of analysis begins in February 2017 because that is first full month of data on completed exams for most of the current contractors. 5 VBA measures timeliness as the number of days between the date the contractor accepts an exam request and the date the contractor initially sends the completed exam report to VBA. VBA officials acknowledged that the exam management system the agency used until spring 2018 did not retain detailed data on exam completion dates, which are necessary to assess individual contractor performance against VBA’s timeliness targets outlined in the contracts. The timeliness values we calculated represent the total time taken to complete exams, which could include time taken by contractors to correct any issues identified by VBA after submitting the initial exam report. 6 Officials stated that VBA and VHA quality scores and exam timeliness are not directly comparable for a number of reasons, including differences in quality review criteria, workloads, and types of exams conducted. Page 2 GAO-19-13 VA Disability Exams performance against timeliness targets outlined in the contracts. We assessed the reliability of the data we collected through electronic testing for missing data and obvious errors, and by conducting interviews with VBA and VHA officials, and determined that the data were sufficiently reliable for our purposes. To evaluate VBA monitoring of contractor performance and VBA oversight of contracted examiners’ qualifications and training, we reviewed relevant federal laws, regulations, and VA guidance on the use of contracted examiners. To identify relevant contract provisions and requirements related to contractor performance, monitoring of such performance, licensing, and training, among other areas, we reviewed selected provisions of selected versions of 1) the 12 current VA Medical Disability Examination contracts originally awarded in 2016; 2) the 5 short-term contracts VA awarded in early 2017; and 3) 2 short-term contracts VA awarded in December 2017. 7 For more information about the methodology used to identify relevant contract provisions and requirements, see appendix I. We also reviewed related contract documents and reports to determine how contract provisions were implemented by VBA and contractors. We assessed VA monitoring against federal internal control standards on quality information and monitoring, and federal contracting guidance. In addition, we assessed VBA oversight of contracted examiners’ qualifications and training against GAO’s guide for assessing training and development. For context, we also reviewed relevant documents on training, licensing, and credentialing requirements for VHA providers who conduct disability compensation exams. We also interviewed officials of VBA’s Mandatory Contract Examination Program Office (exam program office), including Contracting Officer’s Representatives (COR); VA’s Strategic Acquisition Center in Frederick, Maryland, including the contract exam program Contracting Officer; VHA’s Office of Disability and Medical Assessment; each of the five contractors; a private firm that performs audits of VBA contracted examiners’ licenses; and three national veterans service organizations. 7 VBA awarded 5 short-term contracts in early 2017 after performance under 10 of the original 2016 contracts was delayed, in part due to multiple bid protests. Separately, in December 2017, VBA awarded 2 short-term contracts to obtain additional exam capacity after it determined that one of the original contractors was not able to meet the demand for exams. Page 3 GAO-19-13 VA Disability Exams We conducted this performance audit from July 2017 to October 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background VBA Disability Benefits Process VA pays monthly disability compensation to veterans with serviceconnected disabilities according to the severity of the disability. 8 VA’s disability compensation claims process starts when a veteran submits a claim to VA (see fig. 1). A claims processor then reviews the claim and helps the veteran gather the relevant evidence needed to evaluate the claim. 9 Such evidence includes the veteran’s military service records, medical exams, and treatment records from VHA medical facilities and private medical service providers. If necessary to provide support to substantiate a claim, VA will also provide a medical exam for the veteran, either through a provider at a VHA medical facility or through a VBA contractor. According to VBA officials, VBA monitors a VHA facility’s capacity to conduct exams and in instances when the facility may not have capacity to conduct a timely exam, VBA will send an exam request to one of its contractors instead. For exams assigned to a VBA contractor, VBA sends an exam request to the contractor, who then rejects or accepts the exam request. Once the contractor accepts the exam, it assigns a contracted examiner to conduct the exam and complete an exam report designed to capture essential medical information for purposes of determining entitlement to disability benefits. The contractors send the completed report to VBA, which uses the information as part of the evidence to evaluate the claim and determine whether the veteran is eligible for benefits. According to contractor officials, if they need clarification on an exam request, they 8 See 38 U.S.C. §§ 1110 et seq., 1131 et seq. VA’s ratings are awarded in 10 percent increments, up to 100 percent. 9 While veterans may file a claim at their local regional office, portions of the claim are generally processed at a number of offices based on VBA’s automated system for distributing claims workload nationally. Page 4 GAO-19-13 VA Disability Exams might reject the request and send it back to VBA who, in turn, will revise the request before sending it back to the contractor. Figure 1: Overview of VBA Disability Compensation Claims Process and Contractor Process for Completing Exams Note: The steps in the bottom flowchart represent the general process for exam requests assigned to and accepted by VBA contractors, but the flowchart does not include every step in the process. For example, it does not include exam requests that contractors reject and send back to VBA. Also, there is a separate process for exams assigned to VHA providers. VBA sends requests to contractors when a VHA facility is unable to provide exams in a timely manner. Use of Contracts to Complete Disability Compensation Exams VA has used contracted examiners—through VBA and VHA contracts—to supplement VHA-provided exams for at least two decades. VBA began using contractors to conduct disability compensation exams at 10 VBA regional offices in the late 1990s through a pilot program authorized Page 5 GAO-19-13 VA Disability Exams under federal law. 10 In 2014, federal law authorized VBA to expand the pilot to all its regional offices starting in fiscal year 2017. Before fiscal year 2017, VHA and VBA both administered disability exam contracts. However, since fiscal year 2017, all such contracts have been administered by VBA and none have been administered by VHA. VBA awarded 12 contracts to five contractors to begin providing exams in 2016. According to VA officials, performance under 10 of these contracts was delayed until late September 2017 due, in part, to multiple contract bid protests. 11 During this delay, VA officials told us that the agency awarded short-term contracts to allow existing contractors to perform exams until the bid protests were resolved. VBA’s current contracts cover exams for veterans in five U.S. geographic districts, one district for overseas exams, and one district for servicemembers participating in special programs, such as the Benefits Delivery at Discharge and Integrated Disability Evaluation System programs (see fig. 2). 12 VBA awarded two contracts in each of its five U.S. geographic districts and one contract each in districts 6 and 7, which include special programs and overseas exams, respectively. VBA also awarded two additional shortterm contracts in December 2017 to help address workload issues in districts 1-5. With the addition of these two contracts, VBA has a total of 14 contracts currently in place. 10 See Pub. L. No. 104-275, § 504, 110 Stat. 3322, 3341 (1996). In 2003, VA was given additional, temporary authority to use contractors to conduct disability exams. Pub. L. No. 108-183, § 704, 117 Stat. 2651, 2672. This temporary authority originally ran through December 31, 2009, but the expiration date has been extended numerous times. The law did not limit the number of VA regional offices that could obtain exams under the temporary authority. 11 Each contract is for a 12-month base period, with four 12-month option periods. VA initially awarded the contracts in March and September 2016. After several firms successfully protested these awards, VA took corrective action by reopening the procurement, reevaluating the offers, and making new awards in September 2016. Several firms again protested, this time unsuccessfully. Although the second round of protests was resolved in January 2017, the performance period under the contracts did not begin until later that year. 12 The Benefits Delivery at Discharge program allows servicemembers to submit a claim for disability compensation between 180 to 90 days prior to separation, retirement, or release from active duty or demobilization. The Integrated Disability Evaluation System is intended to streamline the military services’ evaluations of servicemember disabilities, and expedite the delivery of benefits to servicemembers separated due to disabilities. Page 6 GAO-19-13 VA Disability Exams Figure 2: Overview of Current VBA Exam Contracts for Veterans’ Disability Compensation Claims Note: San Juan is listed in district 2 in the original contracts signed in 2016, but not in the latest modifications of the contracts. District 5 also includes Manila, Philippines. a Because the latest available VBA data is through April 2018, the time period for the total cost of exams is from October 2017 through April 2018. b The time period for the total number of completed exams is from October 2017 through June 11, 2018. c While VBA awarded two contracts in each of the five U.S. geographic districts, the two additional short-term contracts signed in December 2017 also covered exams in those five districts with one of the short-term contracts covering all five districts and the other short-term contract covering four of the five districts. According to agency officials, because VBA wanted to update performance measures for its contractors, VA issued a Request for Proposals in May 2018 with plans to award new contracts in fall 2018 for its U.S. geographic districts. 13 Until it awards the new contracts, VBA will continue to use the current contracts. According to VBA officials, VA plans to continue using VBA contractors in the long term to conduct 13 The Request for Proposals outlines a one-year period of performance with nine subsequent one-year option periods, while the current one-year contracts have four oneyear option periods. According to VA officials, because it plans to use contractors in the long term, it modified the option years to lessen the administrative costs of resoliciting the contracts. Officials said VBA plans to exercise the option period for the contractor currently performing overseas exams, and to issue a Request for Proposals for its special program exams at a later date. Page 7 GAO-19-13 VA Disability Exams exams that exceed VHA’s capacity. In recent years, VBA contractors have completed an increasing number of exams, from roughly 178,000 in fiscal year 2012 to almost 600,000 in fiscal year 2017, according to VBAprovided data. 14 VA estimates that in fiscal year 2019, contractors will complete over 1.8 million exam reports for almost 800,000 veterans. 15 However, VBA officials noted that future projections for contracted exams might change based on the need to supplement VHA capacity to ensure timely exams. VBA Contract Exam Office and Requirements for Contractors In 2016, VBA established an exam program office to manage and oversee contractors, monitor their performance, and ensure that they meet contract requirements. For example, the contracts require that contractors develop plans outlining how they will ensure examiners are adequately trained. Contractors are also required to provide VBA with monthly exam status reports, which include the number of canceled, rescheduled, and completed exams, among other things. VBA also has an office dedicated to completing quality reviews of contractors’ exam reports, which are used to assess contractor performance. The contracts require that VBA conduct quality reviews of a sample of contractors’ exam reports. According to VA documents and officials, the results of these quality reviews, and contractor timeliness scores in completing exams, are included in quarterly performance reports. The contracts require that VBA provide these performance reports to the contractors. 16 VBA holds quarterly meetings with the contractors to discuss their quarterly performance based on these reports. 14 VBA reported the following number of exams completed by VBA contractors from fiscal year 2012 to fiscal year 2018: 2012-177,712 exams; 2013-193,965 exams; 2014-247,922 exams; 2015-316,995 exams; 2016-308,649 exams; 2017-592,831 exams; and 2018 through June 11-513,988 exams. 15 Because an exam report is based on the related disability benefit questionnaire, an examiner may complete multiple exam reports for each exam conducted for a veteran. 16 In this report, we refer to VBA’s reviews of the quality of a sample of contractor exam reports as quality reviews. We refer to VBA’s quarterly reports on contractors’ performance on quality and timeliness targets as quarterly performance reports. Page 8 GAO-19-13 VA Disability Exams VBA Licensing and Training Requirements for Contracted Examiners VBA contracts require that contracted examiners have full, current, valid, and unrestricted licenses, and current and valid State Medical Board certifications, before conducting any exams—the same requirements that apply to VHA medical providers. 17 According to agency officials, VBA also requires that contracted examiners complete the same training that VHA providers must take before they can conduct any disability medical exams. The required training consists of a set of online courses developed by VHA’s Disability Medical Assessment Office, such as courses on VA’s disability claims process and one on completing exam reports. In addition, examiners who provide some specialized exams, such as posttraumatic stress disorder exams and traumatic brain injury exams, are required to take additional courses. In addition to VHAdeveloped training, VBA contracts require that contractors provide examiners with a basic overview of VA programs. VBA Quarterly Contractor Performance Targets for Quality and Timeliness The contracts also outline quality and timeliness performance targets that VBA uses to assess contractor performance (see table 1). VBA can use contractors’ performance in meeting these targets to determine financial incentives. VBA’s performance measures are as follows: • Contractor quality: VBA calculates quality scores for each contractor based on a sample of exam reports that VBA’s quality office selects for review on a quarterly basis for each contract. According to VBA documents, the quality score represents the percentage of exam reports reviewed that had no errors as measured against specific criteria. Errors identified in quality reviews could range from incomplete information (e.g., an examiner’s medical specialty information is not listed on exam report) to completing the wrong exam report for a given condition. • Contractor timeliness: VBA calculates timeliness scores for each contractor based on the average timeliness of all exams completed in a given quarter for each contract. VBA measures timeliness as the number of calendar days between the date the contractor accepts an 17 VBA contracts outline slightly different requirements for nurse practitioners and physician’s assistants. The contracts also outline additional requirements for licensed psychologists and audiologists. VHA requirements are outlined in VHA Handbook 1100.19: Credentialing and Privileging, Department of Veterans Affairs, Veteran Health Administration, October 15, 2012. Page 9 GAO-19-13 VA Disability Exams exam request and the date the contractor initially sends the completed exam report to VBA. 18 Table 1: VBA Quarterly Performance Targets for Quality and Timeliness for Individual Contractors Completing Exams for Veterans’ Disability Compensation Claims Measure Unsatisfactory performance Expected performance Exceptional performance Quality-All a Districts Quality score of 90% or less for a sample of exams Quality score of 92% for a sample of exams Quality score of 94% or greater for a sample of exams Timeliness-U.S. Geographic b, c Districts 1-5 More than 20 calendar days on average 20 calendar days on average Fewer than 20 calendar days on average Timeliness-Special Programs and Overseas d Districts 6-7 More than 30 calendar days on average 30 calendar days on average Fewer than 30 calendar days on average Source: GAO analysis of selected Veterans Benefits Administration (VBA) disability compensation examination contract documents. GAO-19-13 a Quality scores for each contractor are based on a sample of exam reports selected for review on a quarterly basis for each contract. According to VBA documents, the quality score represents the percentage of exam reports that were reviewed and had no errors as measured against specific criteria. According to the contracts, contractors are expected to meet a target quality score of 92 percent. b VBA measures contractor timeliness as the number of days between the date the contractor accepts an exam request and the date the contractor initially sends the completed exam report to VBA. The contracts signed in 2016 included differing information regarding the start date of the timeliness measure. One provision of those contracts referred to the start date as the date VBA submits the exam request to the contractor. VBA later clarified that it uses the date the contractor accepts the exam request as the start date and that it revised its timeliness measure accordingly in contract modifications signed in December 2017 and January 2018. c VBA officials stated that the timeliness target for almost all exams in districts 1-5 is 20 days on average, with the exception of exams for incarcerated veterans, which is 30 days on average. d Special programs include programs for servicemembers, such as Benefits Delivery at Discharge and Integrated Disability Evaluation System. VBA officials stated that exams for special programs may take longer because veterans who are transitioning from military service may not be readily available for exams. Similarly, they said it may take longer to schedule exams with veterans living overseas. Though the district 1-5 contracts signed in 2016 listed timeliness targets for exams conducted under special programs and exams conducted overseas, VBA officials clarified that these types of exams are not conducted in districts 1-5. They stated that district 6 is for special program exams, district 7 is for overseas exams, and that both districts have a timeliness target of 30 days. 18 VBA officials stated that VBA requests that contractors accept exam requests within 3 days. The contracts signed in 2016 included differing information regarding the start date of the timeliness measure. One provision of those contracts referred to the start date as the date VBA submits the exam request to the contractor. VBA later clarified that it uses the date the contractor accepts the exam request as the start date and that it revised its timeliness measure accordingly in contract modifications signed in December 2017 and January 2018. Page 10 GAO-19-13 VA Disability Exams VBA Reported Contractors Missed Exam Quality Targets, and VBA Could Not Accurately Measure Performance On Timeliness Targets Contractors Missed Quality Targets in First Half of 2017; More Recent Data Are Not Yet Available for Most Districts VBA reported that almost all contractors missed VBA’s quality target of 92 percent in the first half of calendar year 2017, and more recent data are not yet available for most districts. More specifically, VBA-determined quarterly quality scores—the percentage of disability compensation exam reports with no errors as measured against VBA criteria—for the seven contracts used by VBA in calendar year 2017 showed that contractors were frequently well below the quality target. Quarterly quality scores ranged from 62 percent to 92 percent (see fig. 3). 19 According to VBA data, only one contractor’s quality score in one quarter met VBA’s target of 92 percent while the vast majority of contractors’ scores were classified by VBA as “unsatisfactory” performance. 20 19 According to VBA documents, for each quality score, VBA estimated the percentage of exam reports with no errors with a margin of error of 5 percentage points at the 95 percent confidence level. For context, VHA reported quarterly quality scores for VHA examiners in 2017 that ranged from 97 to 98 percent with a margin of error of 15 percentage points at the 95 percent confidence level. However, according to officials, VBA and VHA quality scores are not directly comparable due to differences in quality review criteria and types of exams conducted. 20 Based on VBA’s reported margin of error, the estimated quality score for one additional contractor overlapped with the target of 92 percent in three quarters of the year. Page 11 GAO-19-13 VA Disability Exams Figure 3: VBA-Reported Quarterly Contractor Quality Scores for Exams for Veterans’ Disability Compensation Claims a Quality scores for each contractor are based on a sample of exam reports selected for review on a quarterly basis for each contract. According to VBA documents, the quality score represents the percentage of exam reports reviewed that had no errors as measured against specific criteria. The sample was designed to produce a margin of error of, at most, 5 percentage points at the 95 percent confidence level. According to the contracts, contractors are expected to meet a target quality score of 92 percent. b The quarterly quality scores are for seven contracts under which exams were conducted during calendar year 2017. A total of 15 scores were available because VBA had not finalized the remaining 13 scores for the year. VBA has not yet completed all of the quality reviews used to calculate contractor quality scores, particularly for exams that were completed in the second half of 2017. VBA is hiring and training additional quality review staff to complete these reviews and help manage the workload moving forward. According to VBA officials, staff will complete the remaining quality reviews and finalize the quality scores for 2017 by December 2018. VBA Could Not Accurately Measure Contractor Timeliness Against Targets, but Our Aggregate Analysis Shows About Half of Exams Were Completed Within 20 Days According to agency officials, VBA has not calculated contractor timeliness as it is outlined in the contracts. VBA measures timeliness as the number of days between the date the contractor accepts an exam request and the date the contractor initially sends the completed exam report to VBA. According to officials, this measure does not include any time contractors may spend correcting an exam report returned to them by VBA. Returned exam reports are few in number, VBA officials said. 21 However, once a contractor submitted a corrected or clarified exam report, VBA officials said the exam management system did not preserve the date the exam was initially completed. At that point, the system only tracked the date VBA received the corrected or clarified report. As a result, the number of days in VBA’s system could include time contractors 21 VBA and contractor officials told us that these exam reports make up a small percentage of all exam reports. Also, officials from four of VBA’s five contractors reported estimates ranging from about 3 to 8 percent of exam reports that VBA returned to them for correction or clarification. Page 12 GAO-19-13 VA Disability Exams took to correct any issues identified by VBA after submitting the initial report. While VBA’s data does not allow it to reliably assess contractor performance against the targets in the contracts, VBA’s data can be used to measure timeliness in other ways. For example, we were able to use the data to calculate the entire amount of time it took to complete exams, which includes time contractors took to correct any issues identified by VBA. As such, the results of our analysis should not be interpreted as reflecting contractor compliance with timeliness targets under the contracts. However, to provide timeframes that are similar to VBA’s targets, we chose 20 days for districts 1-5 and 30 days for districts 6-7 as timeframes for our analysis. 22 Moreover, we analyzed timeliness across all contractors rather than for individual contractors. In particular, we analyzed VBA data on 646,005 contracted exams completed from February 2017 to January 2018, which included 575,739 exams in districts 1-5 and 70,266 exams in districts 6-7. 23 Our analysis of VBA data shows that 53 percent of exams were completed within 20 days for districts 1-5, and 56 percent were completed within 30 days for districts 6-7. However, some exams took at least twice as long to complete. For example, 12 percent of exams in districts 1-5 took more than 40 days to complete (see fig. 4). 24 Contractor officials described a number of reasons why exams might take longer in some cases. For example, they said that scheduling delays might occur due to 22 According to the contracts, contractors are not expected to complete all exams within VBA’s timeliness targets, but rather they should meet the target on average in a given quarter. 23 Our analysis included exams completed by contracted examiners between February 2017 and January 2018 that were requested on or after January 13, 2017 in districts 1-5 or on or after April 1, 2016 in districts 6-7 based on the periods of performance in the contracts for those districts. Our period of analysis begins in February 2017 because that is first full month of data of completed exams for most of the current contractors. However, the period of performance for one contractor did not start until April 2017. See appendix I for additional details on our methodology. 24 For context, according to VHA data, VHA examiners completed exams within an average of 20 days during calendar year 2017. However, officials stated that VBA and VHA exam timeliness are not directly comparable for a number of reasons, including differences in workloads, types of exams conducted, and the systems used to maintain exam data. Page 13 GAO-19-13 VA Disability Exams a veteran’s availability or severe weather, and that it can be challenging to find specialists for certain exam types in rural locations. 25 Figure 4: Number of Days Taken to Complete Contracted Exams for Veterans’ Disability Compensation Claims, February 2017 to January 2018 Notes: Percentages may not add up to 100 due to rounding. VBA officials stated that VBA requests that contractors accept exam requests within 3 days. We found that about 82 percent of exam requests were accepted within 3 days. We also calculated that the percentage of exams completed within 20 or 30 days could be as much as 3 percent lower if contractors had accepted every exam request within 3 days. Our analysis included exams completed between February 2017 and January 2018 for exam requests made on or after January 13, 2017 in districts 1-5 or on or after April 1, 2016 in districts 6-7 based on the periods of performance in the contracts for those districts. Our period of analysis begins in February 2017 because it is the first full month of data of completed exams for most of the current contractors. However, the period of performance for one contractor did not start until April 2017. a Special programs include programs for servicemembers, such as Benefits Delivery at Discharge and Integrated Disability Evaluation System. VBA officials stated that exams for special programs may take longer because veterans who are transitioning from the military may not be readily available for exams. Similarly, they said it may take longer to schedule exams with veterans living overseas. Our analysis of timeliness focused on exams that were completed, and it did not include exams that have been requested and not yet completed by a contractor. For example, a contractor may have accepted an exam request from VBA, but not yet scheduled an appointment with the veteran. Alternatively, a contractor may have conducted an exam with the 25 According to VBA officials, the agency restructured the service areas in its recent Request for Proposals from five U.S. geographic districts to four to help address the challenge of finding examiners in rural areas by reducing the number of these areas awarded to any one contractor and to more evenly distribute workload throughout each district. Page 14 GAO-19-13 VA Disability Exams veteran, but not yet sent the exam report to VBA. As of late June 2018, VBA-calculated data showed that 87,768 requested exams had not yet been completed, including 37,077 exams that had already exceeded VBA’s timeliness targets. 26 Tracking these exams is important because a large volume of such exams could ultimately increase the amount of time veterans have to wait for their claims to be processed. VBA officials stated that the agency closely monitors contractors’ workloads and helps expedite requested exams that have exceeded VBA’s targets for completing exams. 27 In addition, VBA included a performance measure in its May 2018 Request for Proposals to track the percentage of requested exams that have been with a contractor for more than seven days. Such a measure could help VBA identify whether contractors have a backlog of exams and better assess whether veterans are receiving timely exams. 26 According to a VBA official, the total number of exams that have not been completed could be slightly higher because there are a small number of exams in VBA’s exam management system with conflicting information on exam status. In particular, some exams appear to have been cancelled but still show an active status in the system. 27 While VBA’s system can track requested exams that have not been completed, it does not track exams through each precise phase of the exam process (e.g., assigned to examiner, appointment scheduled with veteran) between the time an exam is requested and completed. However, VBA’s new exam management system is designed to capture more details on the status of exams throughout the process. Page 15 GAO-19-13 VA Disability Exams Delayed Quality Reviews and Performance Reports, and Data Limitations, Hinder VBA’s Monitoring of Contractors VBA Identified Some Contractor Performance Problems but Was Delayed in Completing Quality Reviews and Performance Reports VBA Identification of Performance Problems VBA’s contract exam program office, primarily through its Contracting Officer’s Representatives (COR), has identified some contractor performance problems, such as delays in completing specific exams, through its oversight of contractor performance. This oversight includes day-to-day monitoring of contractor workloads and frequent contact with contractor officials. Through such contact and reviews of contractors’ daily and weekly exam status updates, the CORs work with contractor officials to identify ways to expedite disability compensation exams for veterans who have been waiting longer than VBA’s 20-day or 30-day targets. 28 In addition, VBA contract quality staff who review samples of contractor exam reports hold teleconferences with the CORs and contractor officials to provide feedback and discuss issues arising from their reviews, such as specific types of errors. The VBA contract exam program office also oversees and manages contractors through supplemental guidance memos, contractor site visits, 28 In its May 2018 Request for Proposals for new exam contracts, VBA describes requirements for contractors’ monthly exam status reports, including monthly reports of any exams older than 20 days, along with plans to resolve them. Page 16 GAO-19-13 VA Disability Exams and reviews of veteran customer satisfaction surveys. For example, in November 2017, VBA sent a supplemental guidance memo to all contractors to clarify guidance on conducting and documenting hearing loss exams. Further, VBA has conducted site visits to all five contractors’ headquarters or clinic sites since September 2017. Headquarters visits include reviews of contractors’ procedures, such as those for assigning exam requests, and contractors’ information systems, such as those for tracking the status of exams. VBA visits to contractor clinics focus on facility issues, such as accessibility and safety. 29 According to VBA officials, the CORs also review reports on satisfaction surveys completed by veterans after their exam appointments to identify veterans’ concerns regarding contractors and to follow up with contractors, when needed. 30 For example, in response to one veteran’s survey comment regarding a contracted examiner who did not show up to conduct a scheduled exam, VBA officials told us they followed up with the contractor and learned that the examiner’s car broke down. According to VBA, it reimbursed the veteran for round-trip transportation costs to the clinic. Additionally, VBA’s contract quality review staff have conducted special focused reviews to investigate concerns raised by veterans and by staff in VBA regional offices and VHA medical facilities. For example, VBA conducted a review of one contracted examiner who had high rates of diagnosing severe posttraumatic stress disorder. After reviewing this examiner’s reports, VBA found their overall quality to be poor. As a result, VBA requested that the contractor no longer use this examiner. In addition to identifying and addressing problems with individual exams and examiners, VBA has identified broader challenges faced by contractors in meeting VBA’s demand for exams and providing timely reports. For example, VBA identified two contractors who were not prepared to perform all of their assigned exams because they did not have enough examiners, particularly in rural locations, which led to delays 29 According to VBA officials, it has tested new site visit procedures and is updating its site visit checklists. VBA plans to implement the new site visit procedures in fiscal year 2019, and those procedures will include reviews of contractors’ clinical procedures, checks of contractors’ data systems, and reviews of contractors’ training materials, among other things. 30 A third-party contractor conducts the surveys and produces monthly, quarterly, and annual reports of the survey results. According to VA’s May 2018 Request for Proposals, VBA intends to use results from these surveys as part of its assessment of contractors’ performance when new contracts are awarded. Page 17 GAO-19-13 VA Disability Exams and a backlog of exam requests, according to VBA officials. VBA officials described how they worked with these contractors over several months to adjust and closely monitor the volume of exams sent to the contractors to address the backlog. However, according to VBA officials, by December 2017, VBA determined that one of the contractors was not able to meet the demand for exams, and the agency stopped sending new exam requests to this contractor. According to VBA, by late June 2018, it had discontinued all work with this contractor. 31 VA officials said that to obtain additional exam capacity to make up for the two contractors’ shortages, they awarded short-term contracts in December 2017 to two other contractors who were providing exams in other VBA districts. 32 VBA Delays in Assessing Quality and Completing Reports VBA has not completed all required quarterly quality reviews and accompanying quarterly performance reports on contractors, according to VBA officials. These reviews and reports are key components to effectively assessing contractor performance in a timely manner. Specifically, in late June 2018, VBA officials said that they had conducted almost all their quality reviews for contracted exams completed in districts 1-5 during the second half of 2017, but that they needed to finalize the quality scores. They also said that they were beginning their quality reviews for contracted exams completed in 2018. At the time of our review, VBA had released one quarterly performance report for the fourth quarter of calendar year 2017, and officials said they were drafting others. 33 VBA officials attributed delays in completing quality reviews and quarterly performance reports primarily to a lack of VBA quality review staff. The quarterly performance reports provide contractors with information on their performance against VBA quality and timeliness targets. For example, prior reports included detailed breakouts of quality errors by type and suggestions for performance improvements. As officials of one contractor said, delays in receiving quarterly performance 31 In March 2018, VA and the contractor agreed that the contractor would stop performing exams for VBA, except to complete certain exams that were already scheduled. 32 VA reported that it continues to use these short-term contracts, and will do so until the new contracts for its five U.S. geographic districts are awarded and implemented. VA officials also reported that it exercised the first option years in its other current exam contracts for its U.S. geographic districts. 33 As part of the process of preparing quarterly performance reports, VBA quality review staff provide contractors with preliminary review results. VBA officials also stated that the staff provide contractors with opportunities to seek reconsideration—and potential reversal—of VBA decisions on exam errors noted in VBA’s quality reviews. Page 18 GAO-19-13 VA Disability Exams reports limit VBA’s ability to provide contractors with timely and valuable feedback they can use to improve the quality of their exams. The delay in completing the quarterly reviews and reports also has implications for VBA’s ability to allocate exam requests across contractors and administer potential financial incentives across contractors. More specifically, VBA can use performance data to help determine how to allocate exams in each district that has two contractors, as outlined in the contracts. For example, VBA can decide to allocate more exams to the contractor with higher performance results. 34 Further, the contracts outline how VBA can use performance data to administer financial incentives linked to performance targets. For example, VA is to provide a bonus to a contractor who meets or exceeds the 92 percent quality standard for a quarter, and meets or exceeds the 20- or 30-day timeliness standard. However, because of its delays in completing quality reviews and the lack of reliable data on contractor timeliness, VA has not yet administered these incentives. VA officials told us that the agency will determine if it will administer the 2017 incentives after it completes its performance assessments of contractors. VBA officials said they are currently hiring more staff to address the lag in quality reviews and subsequent reports to contractors, as well as to provide more oversight of contractors. At the time of our review, VBA did not have its authorized level of 15 quality analysts and 2 senior quality reviewers, but VBA officials said that they expected to complete hiring to bring the quality reviewer staff up to 17 full-time positions by the end of fiscal year 2018. In addition, VBA officials acknowledged that they did not have enough CORs in VBA’s exam program office to oversee the 14 exam contracts (including the two short-term contracts). 35 As of April 2018, VBA officials said the office had 3 CORs, but hiring was expected to bring the number up to 14 by the end of fiscal year 2018. VBA officials said that they determined staffing levels for VBA’s contract exam program office—including CORs and exam quality reviewers—based on an assessment of the resources needed to expand the program, among other factors. Although VBA did not provide documentation outlining how 34 In districts 1-5, two contractors share the workload within each district. As stated in the contracts, VBA can determine how to allocate some exams between the two contractors based on each contractor’s performance, and its capacity to conduct exams. 35 VBA officials stated that for most of 2017 they were unable to hire any exam contract program office staff—including quality reviewers or CORs—due to a federal hiring freeze and an internal VA hiring freeze. Page 19 GAO-19-13 VA Disability Exams it determined its workforce needs, the agency provided us with updated organizational charts in June 2018 demonstrating increased staff levels for the exam program office. VBA’s Data Limitations Hinder Its Ability to Oversee Certain Contract Provisions, and VBA Has Not Conducted Comprehensive Performance Analysis VBA’s lack of reliable data on the status of exams, including insufficient exams—exam reports that VBA returns to contractors to be corrected or clarified—limits its ability to effectively oversee certain contract provisions. 36 VBA officials acknowledged that they could not calculate the number of completed exams that were once marked as insufficient or how long they had remained in that status due to the data limitations of the exam management system the agency used until spring 2018. The contracts require that contractors correct insufficient exams within a certain number of days and bill VBA for these exams at half price. 37 However, VBA’s lack of complete and reliable information on insufficient exams hinders its ability to ensure that either of these requirements is met. VBA officials also indicated that they were unable to fully assess individual contractor timeliness against VBA’s performance targets because the exam management system did not include the date the initial exam report was submitted to VBA, which is needed to calculate timeliness as outlined in the contracts. In March 2018, VBA began implementing a new exam management system designed to collect more comprehensive and accurate information on the status of exams. VA documentation on the new system shows that it will include detailed data on insufficient exams, which, according to VBA officials, should allow VBA to track whether contractors are properly discounting their invoices for those exams. However, in June 2018, VBA stated that three of its five contractors did not have complete functionality 36 VBA has separate classifications for inadequate and insufficient exams depending on whether information is missing, incorrect, or needs to be clarified. In the contracts signed in 2016, inadequate exams were defined as exams that do not meet exam report requirements. Insufficient exams were defined as exams returned to the contractor for clarification, correction, or completion. In later contract modifications, VBA amended the definitions of inadequate and insufficient exams. Inadequate exams are defined as exam reports with incomplete or missing information in required fields or reports with inaccurate medical information. Insufficient exams are defined as exam reports that, despite being complete and containing accurate medical information, require clarification for the purposes of reaching a disability rating decision. For the purposes of this report, we refer to all of these cases as insufficient exams, unless specified otherwise. 37 The contracts signed in 2016 required contractors to deduct 50 percent from the billing of the inadequate exams. The contracts were later modified to require contractors to deduct 50 percent from the billing of both inadequate exams and insufficient exams. Page 20 GAO-19-13 VA Disability Exams with VBA’s new exam management system. As a result, VBA officials said the agency still did not have complete data in the new system that would allow it to track insufficient exams. Officials said they were working to address these issues. More broadly, as described in VA system documents, the new system is designed to allow VBA to track more detailed data on exam completion dates and on other points throughout the exam process, such as dates for initial requests for clarification from contractors, and dates when appointments are scheduled. However, VBA is in the early stage of this transition, and agency officials stated that unexpected technical issues have affected communication between the new exam management system and other VBA systems. While they work to resolve the issues, VBA officials said that they have been manually moving some exam requests through the system each day. Further, VBA has not documented how it plans to ensure the additional data is accurate and use it to oversee contractor performance as outlined in the contracts, particularly for insufficient exams. 38 Federal internal control standards state that management should use quality information to achieve key objectives. 39 In addition, management should formulate plans to achieve those objectives. For example, agencies should assess collected data and ensure it is accurate so that it can be used to provide quality information to evaluate performance. In the absence of a plan for how it will capture and use data in its new exam management system to assess performance, VBA risks overpaying contractors for insufficient exams and continuing to inaccurately measure contractor timeliness. Further, according to agency officials, VBA has not conducted comprehensive analyses of performance data that would allow it to identify and address higher-level trends and program-wide challenges across contractors, geographic districts, exam types, or other relevant factors. Agency officials told us they have no plans to conduct such analyses. Federal internal control standards state that management should establish and operate monitoring activities and evaluate the 38 VBA officials stated that they have discussed with their data analysis team the types of measures that are needed to oversee contractor performance, but they have not taken additional steps to assess contractor timeliness based on those measures and have not developed monitoring tools. 39 GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: September 10, 2014), principle 13. Page 21 GAO-19-13 VA Disability Exams results of those activities. 40 In addition, management should evaluate deficiencies both at the individual and aggregate level. While VBA officials acknowledged that higher-level analyses could improve program oversight, they explained that analyzing performance data has been challenging due to the limitations of the exam management system. Thus, VBA has prioritized addressing contractor-specific problems and resolving long-standing pending exams over in-depth analysis of the performance data. However, with the expected improvements provided by VBA’s new exam management system and increased staff to manage the program and conduct quality reviews, VBA should be better positioned to conduct analyses of performance data in the future. By conducting higher-level analyses across contractors, geographic districts, exam types, or other relevant factors, VBA could make a more informed assessment of the challenges contractors and examiners face and where additional workload capacity and training may be needed. In addition, better analyses would allow VBA to determine if the contract exam program is achieving its quality and timeliness goals in a cost effective manner. 40 GAO-14-704G, principle 16. Page 22 GAO-19-13 VA Disability Exams Auditor Verifies Contracted Examiner Licenses, but VBA Does Not Verify Training Completion or Collect Information on Training Effectiveness VBA Uses a Third-Party Auditor to Verify Contracted Examiner Licenses VBA has a third-party auditor who verifies that all active contracted examiners have a current, valid, and unrestricted medical license in the state where they examined a veteran. The auditor provides regular reports of its audits to VBA. 41 Specifically, the auditor verifies the license numbers of all active contracted examiners in the states where they perform VA disability compensation exams; National Provider Identifiers; and any prior or current sanctions or restrictions resulting in a revoked or suspended license at the time of a VA exam. 42 In addition, contractors send VBA monthly reports of examiners’ medical license, specialty, and accreditation based on the contractors’ verification of this information. 43 Every 2 months, VBA sends the auditor a consolidated report of this information covering all five contractors. The auditor verifies examiners’ information in that report before sending a final audit report to VBA, noting if the auditor was or was not able to verify examiners’ licenses. After reviewing the report, VBA contacts the contractors to gather additional information to resolve any issues, and in cases in which licensing requirements are not met, VBA stops using the examiner and offers new exams to veterans who have been seen by the examiner. 41 For exams conducted before late 2017, the auditor checked a sample of active examiners. 42 National Provider Identifiers are standard national, unique, 10-digit identification numbers assigned to health care providers for the purpose of improving the efficiency and effectiveness of the electronic transmission of health information, such as billing. 43 Before awarding the exam contracts, according to VA officials, VA’s Contracting Office reviewed all contractors’ processes for verifying examiners’ licenses. Page 23 GAO-19-13 VA Disability Exams VBA and auditing firm officials noted that audit results show that almost all examiners have current and valid licenses, and contractors are required to stop using those who do not meet licensing requirements. VBA and auditing firm officials said that issues identified in the audits are usually due to typos or differences in how information is captured across different licensing databases. However, based on an audit, VBA provided an example of an examiner with a restricted medical license who had completed exams for one contractor. In this case, VBA notified the contractor, who then stopped using the examiner and said it was taking action to prevent errors in its license verification process from occurring again. In addition, the contractor reimbursed VBA for the cost of exams conducted by the examiner and also offered new exams to veterans who had been seen by the examiner. VBA Relies on Exam Contractors to Verify Training is Completed and Does Not Review Training Records for Accuracy VBA relies on contractors to verify that their examiners complete required training, and agency and contractor officials told us that VBA does not review contractors’ self-reported training reports for accuracy or request supporting documentation, such as training certificates, from contractors. 44 As required by the contracts, contractors must track and maintain records demonstrating each examiner has completed required training. Each of VBA’s five contractors has its own process for ensuring that required training is provided to and completed by their examiners, but generally, contractors export the courses from VA’s online training system into their own online training systems for their examiners to access. The contractors, rather than VBA, access the contractor training systems to verify that examiners have completed the required training before they are approved to conduct exams. When requested by VBA, contractors are required to send VBA reports demonstrating that their examiners have met training requirements. As stated in the latest version of the contracts, contractors must immediately stop using any examiner found to have not completed required training, notify VBA, and re-examine the involved veterans at no cost to VBA, if requested by the agency. Although VBA currently does not verify the accuracy of training selfreported by contractors to the agency, VBA officials said that they plan to enhance monitoring through spot checks of training records and a new training system. Specifically, in fiscal year 2019, VBA officials said they 44 The contracts require that contractors develop training plans that outline how they will ensure their examiners are adequately trained. VA officials told us they reviewed these plans before the start of all current contracts. Page 24 GAO-19-13 VA Disability Exams plan to start conducting spot checks of some examiners’ training records for accuracy and compliance during site visits to contractor headquarters and clinics. However, VBA has not provided details or documentation on these planned checks, such as how it will determine which records to review or the steps it will take to verify the accuracy of training records. VBA officials also said they are planning to develop an online system that would allow VBA to certify that examiners have completed required training, rather than relying on contractors for this information. However, as of July 2018, VBA had yet to determine when this system would be developed and had not documented plans to do so in order to use such information for monitoring training. VBA also said it would hire staff to manage contractor training, but has yet to do so. GAO’s prior work has emphasized tracking and other control mechanisms to ensure that all employees receive appropriate training. 45 While VBA said it would enhance its monitoring of training records, documenting and implementing a plan and processes to verify training could help ensure examiners have met training requirements. Without such a plan, VBA risks using contracted examiners who are unaware of the agency’s process for conducting exams and reporting the results, which could lead to delays for veterans as a result of poor-quality exams that need to be redone and insufficient exam reports that need to be corrected. VBA Does Not Collect Information to Determine if Training Effectively Prepares Examiners VBA does not collect information from contractors or examiners to help determine if required training effectively prepares examiners to conduct high quality exams and complete exam reports. VBA has provided additional guidance to contractors for some specialty exams. However, VBA identified these issues after some contractors requested guidance in monthly meetings, rather than through VBA efforts to proactively or regularly collect information from contractors or examiners to inform potential changes to training. VBA is considering including a component in the online training system that would collect information on the effectiveness of required training. However, VBA has not outlined additional details on collecting such information. VBA officials said that VBA did not collect such information in the past, in part, because staff were focused on program oversight. 45 GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: March 2004). Page 25 GAO-19-13 VA Disability Exams To assess progress toward achieving results and to make changes to training if needed, GAO has found that evaluation is a key component of any training program. 46 Given that VBA officials told us that the agency plans to issue new contracts in fall 2018, the number of contracted examiners who are new to VA processes may increase. Thus, collecting and assessing regular feedback on training from contractors and examiners, such as through surveys, discussion groups, or interviews, could help VBA determine if training effectively prepares examiners to conduct exams and complete exam reports. Further, information on the effectiveness of training could supplement data on contractor performance and results from VBA’s quality reviews to help assess if additional training courses are needed across contractors or for specific exam types. Conclusions As VBA increasingly relies on contractors to perform veterans’ disability compensation exams, it is important that the agency ensures proper oversight of these contractors. VBA’s lack of accurate and up-to-date data and reports on contractor performance hampers its ability to oversee the quality and timeliness of exams provided through contractors. VBA’s new exam management system provides opportunities to improve oversight through more comprehensive and accurate data. These improvements might be limited, however, without a plan to use the data to produce the quality information needed by VBA to monitor insufficient exams, ensure it pays contractors the correct amount for those exams, and help it accurately calculate contractor timeliness. Further, the new system provides an opportunity for VBA to conduct analyses that could identify high-level trends and challenges facing the program across contractors and districts, such as delays in completing exams in specific parts of the country or contractor performance issues related to specific exam types. Despite these capabilities, VBA has not outlined plans for using improved information in this manner. Without doing so, the agency may miss opportunities to improve the program and, ultimately, its service to veterans. VBA could better prepare contracted examiners for their role by taking actions to ensure required training has been completed and by collecting information to assess and improve training. Such actions could help 46 GAO-04-546G. Page 26 GAO-19-13 VA Disability Exams improve the quality of exams and exam reports, which could mitigate the need for exam rework and, ultimately, delays in determining veterans’ benefits. With VBA planning to award new contracts and potentially more new contracted examiners coming on board, verifying that required training is completed and collecting information on the effectiveness of training are critical. As VA continues to rely on contracted examiners, it is important that the agency is well positioned to carry out effective oversight of contractors to help ensure that veterans receive high-quality and timely exams. Recommendations for Executive Action We are making the following four recommendations regarding contracted disability compensation exams to VA. The Under Secretary for Benefits should develop and implement a plan for how VBA will use data from the new exam management system to oversee contractors, including how it will capture accurate data on the status of exams and use it to (1) assess contractor timeliness, (2) monitor time spent correcting inadequate and insufficient exams, and (3) verify proper exam invoicing. (Recommendation 1) The Under Secretary for Benefits should regularly monitor and assess aggregate performance data and trends over time to identify higher-level trends and program-wide challenges. (Recommendation 2) The Under Secretary for Benefits should document and implement a plan and processes to verify that contracted examiners have completed required training. (Recommendation 3) The Under Secretary for Benefits should collect information from contractors or examiners on training and use this information to assess training and make improvements as needed. (Recommendation 4) Page 27 GAO-19-13 VA Disability Exams Agency Comments and Our Evaluation We provided a draft of our report to the Department of Veterans Affairs (VA) for its review and comment. VA provided written comments, which are reproduced in appendix II. VA concurred with all our recommendations and described the Veterans Benefits Administration’s (VBA) plans for taking action to address them. Regarding our first recommendation, VA outlined improvements in the information collected through VBA’s new exam management system, and said that VBA is currently testing a mechanism to validate exam invoices submitted by contractors. We noted these improvements to the system in our draft report sent to the agency for comment. We maintain that it will be important for VBA to take the next step of developing and implementing a plan for how it will use information from the new system to ensure both accurate timeliness data and proper exam invoicing. Regarding our second recommendation, VA stated that VBA will use improved data in the new exam management system to regularly monitor and assess aggregate performance data, identify error trends, and monitor contractor performance and program-wide challenges. Regarding our third and fourth recommendations, VA stated that VBA plans to develop and implement a training plan for contractors that will include a mechanism to validate that required training has been completed and to assess the effectiveness of this training through feedback from trainees, contractors, and quality review staff in VBA’s contract exam program office. VA stated that VBA will use this data to improve the implementation and content of training. VA requested that GAO combine these two recommendations into one. However, we believe they are two distinct recommendations and have kept them as such. VBA could meet the intent of each recommendation with the development and implementation of one plan that covers both training verification and assessment, as outlined in its comments. Page 28 GAO-19-13 VA Disability Exams As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions, please contact me at (202) 5127215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in Appendix III. Sincerely yours, Elizabeth H. Curda Director, Education, Workforce, and Income Security Page 29 GAO-19-13 VA Disability Exams Appendix I: Additional Information on Selected Methodologies Appendix I: Additional Information on Selected Methodologies Review of VBA Contracts To evaluate VBA monitoring of contractor performance and VBA oversight of contracted examiners’ qualifications and training, we reviewed relevant federal laws, regulations, and VA guidance on the use of contracted examiners for disability compensation exams. To identify relevant contract provisions and requirements related to contractor performance, monitoring of such performance, licensing, and training, among other areas, we reviewed selected provisions of selected versions of the 12 current VA Medical Disability Examination contracts originally awarded in 2016, of 5 short-term contracts VA awarded in early 2017, and of 2 short-term contracts VA awarded in December 2017. 1 With regard to the 12 current contracts, we reviewed the selected provisions in the originally awarded contract from 2016 and in the most recently amended version of the contract (as provided to us by VBA officials). Based on our review of these two versions of the contract, the selected provisions appeared to remain in place, unless noted otherwise in this report. However, we did not review the various contract modifications that, according to VBA, occurred in the interim period to confirm whether the selected provisions we focused on in our review actually remained in place during the period between the original contract and the most recent amendment. With regard to the 2 short-term contracts awarded in December 2017, we reviewed the selected provisions in the original December contract. According to VBA officials, there have been no subsequent modifications to these short-term contracts. With regard to the 5 short-term contracts awarded in early 2017, we only reviewed selected provisions relating to contractor quality and timeliness performance. Thus, any statements in this report relating to other aspects of the contracts are not based on these short-term contracts. Further, we only reviewed such provisions in the originally awarded short-term contract, and we did not review the various contract modifications that, according to VBA, occurred subsequently, to confirm that those provisions remained in place over time. However, we found that those selected provisions were generally in place in all of the various contracts we reviewed. 1 VBA awarded 5 short-term contracts in early 2017 after performance under 10 of the original 2016 contracts was delayed due, in part, to multiple bid protests. Separately, in December 2017, VBA awarded 2 short-term contracts to obtain additional exam capacity after it determined that one of the original contractors was not able to meet the demand for exams. Page 30 GAO-19-13 VA Disability Exams Appendix I: Additional Information on Selected Methodologies Analysis of VBA Data on the Timeliness of Contracted Exams To answer what is known about the timeliness of VBA contracted exams, we analyzed VBA data on disability compensation exams completed by five contractors between February 2017 and January 2018. VBA’s Office of Performance Analysis and Integrity provided exam-level data that it maintains in the agency’s Enterprise Data Warehouse, including data on the exam request date, the date the contractor accepted the request, the date the contractor completed the exam, and the VBA district where the exam was conducted, among other information. 2 These data were created from data originally collected in VBA’s Centralized Administrative Accounting Transaction System (CAATS), which is the system that VBA used to request exams from contractors until spring 2018. 3 According to VBA officials, the status of exam requests (e.g., pending, completed, cancelled) was not always accurate in CAATS. To create more reliable data and identify the most current information on the status of exams, the Office of Performance Analysis and Integrity identified and replaced missing or incorrect data in CAATS by running checks against other VBA systems, including the Veterans Benefits Management System, which maintains veterans’ benefits claims records. We assessed the reliability of the data we received from VBA by conducting electronic testing for missing data and errors, and by interviewing VBA officials about their data collection and quality control procedures. We determined that the data were sufficiently reliable for our purposes of reporting the time it took to complete exams within districts. Our analysis included 646,005 contracted exams completed between February 2017 and January 2018. 4 We selected February 2017 as our starting point because it was the first full month of data available that 2 We also reviewed data on exam requests that were cancelled. However, we did not report data on cancellations because the reason for the cancellation was not always clear in the data. 3 VBA began implementing a new exam management system in spring 2018 as a new feature of the Veterans Benefits Management System, which is the electronic system VA uses to process veterans’ claims. 4 We calculated that the total universe of VBA contracted exams completed between February 2017 and January 2018 was 667,480 exams. The number of exams in our analysis was smaller primarily due to how we defined our population—based on the exam request date. We also excluded a small number of exams from our analysis that had missing or potentially inaccurate VBA district information because we could not determine which timeliness target should apply to those exams. Page 31 GAO-19-13 VA Disability Exams Appendix I: Additional Information on Selected Methodologies covered most of VBA’s current contractors. 5 To allow for 12 full months of data, we selected January 2018 as our ending point. In addition, we limited our population to include exams that were requested on or after January 13, 2017 in districts 1-5 or on or after April 1, 2016 in districts 67, based on the periods of performance in the contracts for those districts. 6 We calculated timeliness at the level of the exam request. We calculated the number of days between the date an exam request was accepted by the contractor and the date the exam report was completed by the contractor. The timeliness values we calculated may include additional time needed to request and receive contractors’ corrections or clarifications on previously submitted exam reports. In our report, we refer to these exams as “insufficient exams.” 7 VBA officials acknowledged that due to data limitations the new exam management system is intended to resolve, VBA’s CAATS system did not retain data on the number of exams that were once marked as insufficient or how long they remained in that status. 8 While VBA officials acknowledged that this data limitation affects the agency’s ability to assess individual contractor timeliness on VBA’s performance targets outlined in the contracts, the limitation did not prevent us from analyzing the timeliness of contracted exams overall. The overall timeliness values we calculated represent the total time taken to 5 Four out of five contractors had been conducting exams with VBA by mid-January 2017 or earlier. The other contractor began conducting exams for VBA in April 2017. 6 These timeframes were different based on bid protests that affected performance under the initial contracts for districts 1-5, which were initially awarded in 2016. As a result, our analysis for districts 1-5 included exams that were completed under short-term and initial contracts. 7 VBA has separate classifications for inadequate and insufficient exams depending on whether information is missing, incorrect, or needs to be clarified. In the contracts signed in 2016, inadequate exams were defined as exams that do not meet exam report requirements. Insufficient exams were defined as exams returned to the contractor for clarification, correction, or completion. In later contract modifications, VBA amended the definitions of inadequate and insufficient exams. Inadequate exams are defined as exam reports with incomplete or missing information in required fields or reports with inaccurate medical information. Insufficient exams are defined as exam reports that, despite being complete and containing accurate medical information, require clarification for the purposes of reaching a disability rating decision. For the purposes of this report, we refer to all of these cases as insufficient exams, unless specified otherwise. 8 Officials from four of VBA’s five contractors reported estimates ranging from about 3 to 8 percent of exam reports that VBA returned to them for correction or clarification. Page 32 GAO-19-13 VA Disability Exams Appendix I: Additional Information on Selected Methodologies complete exams regardless of whether additional time was needed for corrections. To put the timeliness values we calculated in context, we calculated the percentage of exams that were completed within VBA’s timeliness targets of 20 days for districts 1-5 and 30 days for districts 6-7 for the entire 12month period of our analysis. 9 We also calculated the percentage of exams that were completed within other timeframes (e.g., 21-40 days, more than 40 days). According to the contracts, contractors are not expected to complete all exams within the timeliness target, but rather should meet the timeliness target on average in a given quarter, so our analysis was different from one that VBA might conduct in order to determine contract compliance. Because VBA does not retain detailed data on exam completion dates necessary to assess contractor performance against VBA’s timeliness targets, and because we calculated timeliness across contractors, the percentages we calculated do not represent an assessment of whether contractors met VBA’s timeliness targets. GAO did not conduct a legal analysis of the various contractors’ compliance with the contract requirements. Alternate Timeliness Values Given that the start of VBA’s timeliness measure is the date the contractor accepts the exam request (rather than the date VBA requests the exam), we calculated alternate timeliness values to account for potential delays in accepting exam requests. VBA officials stated that VBA requests contractors accept or reject exam requests within 3 days. 10 For all exam requests that contractors took more than 3 days to accept, we calculated alternate totals that included the additional days. For example, if a contractor took 5 days to accept the exam request and completed the exam 20 days later, we calculated an alternate total of 22 9 VBA officials stated that the timeliness target for almost all exams in Districts 1-5 is 20 days. Though those contracts include a 30-day target for exams for incarcerated veterans, VBA officials stated that the agency does not have a variable to track the timeliness of those exams separately from exams with a 20-day standard in Districts 1-5. However, they noted that exams for incarcerated veterans account for a small minority of exams and should not materially affect a contractor’s average timeliness score. Further, though the District 1-5 contracts signed in 2016 listed timeliness targets for exams conducted under special programs and exams conducted overseas in the contract, VBA officials clarified that these types of exams are not conducted in Districts 1-5. Instead, they stated that District 6 is for special program exams, District 7 is for overseas exams, and that both districts have a timeliness target of 30 days. 10 VBA officials said that they initially expected to have a feature in CAATS that would require contractors to accept exam requests within 3 days, but that the system did not have this capability. Page 33 GAO-19-13 VA Disability Exams Appendix I: Additional Information on Selected Methodologies days to complete the exam. We used these alternate values to calculate adjusted percentages for each category presented in Figure 4 of our report. For example, using the alternate timeliness values, about 50 percent of exams in districts 1-5 would have been completed in 20 days and 53 percent in districts 6-7 would have been completed within 30 days, rather than the respective 53 percent and 56 percent shown in Figure 4. Moreover, we found that about 82 percent of exam requests during our period of analysis were accepted within 3 days. Pending Exams To report more recent data on exams that were accepted but not yet completed by contractors—pending contracted exams—VBA provided aggregate data on the number of pending exams as of June 25, 2018. For example, for districts 1-5, it provided data on the number of exams that had been pending for 20 days or fewer, 21-40 days, 41-60 days, 61100 days, and more than 100 days. We calculated percentages based on the VBA-provided totals. Page 34 GAO-19-13 VA Disability Exams Appendix II: Comments from the Department of Veterans Affairs Appendix II: Comments from the Department of Veterans Affairs Page 35 GAO-19-13 VA Disability Exams Appendix II: Comments from the Department of Veterans Affairs Page 36 GAO-19-13 VA Disability Exams Appendix II: Comments from the Department of Veterans Affairs Page 37 GAO-19-13 VA Disability Exams Appendix III: GAO Contact and Staff Acknowledgments Appendix III: GAO Contact and Staff Acknowledgments GAO Contact Elizabeth Curda, (202) 512-7215 or curdae@gao.gov. Staff Acknowledgments In addition to the contact named above, Nyree Ryder Tee (Assistant Director); Teresa Heger (Analyst-in-Charge); Alex Galuten; Justin Gordinas; and Greg Whitney made key contributions to this report. Also contributing to this report were James Bennett, Matthew T. Crosby, Teague Lyons, Sheila R. McCoy, Jessica Orr, Claudine Pauselli, Samuel Portnow, Monica Savoy, Almeta Spencer, and April Van Cleef. (102155) Page 38 GAO-19-13 VA Disability Exams GAO’s Mission The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to https://www.gao.gov and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. To Report Fraud, Waste, and Abuse in Federal Programs Contact: Congressional Relations Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548 Public Affairs Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Strategic Planning and External Liaison James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Website: https://www.gao.gov/fraudnet/fraudnet.htm Automated answering system: (800) 424-5454 or (202) 512-7700 Please Print on Recycled Paper.