From: Moodie, Andrea Sent: Friday, May 12, 2017 3:21 PM To: VHA 10H Action; VHA 10NF Action Cc: Holly, Nancy Subject: Scheduling Exit Conference - OIG Audit of the Health Care Enrollment Program at VHA Medical Facilities (Project No. 2016-00355-132-0013) Good Afternoon Request a representative from 10M and 10NF Audit Management) any other SMEs to attend GIG Entrance Conference. See attached Statement of Facts (SOF) and provide any comments to the SOF NLT Friday May 19, 2017. A negative reply is required. See information and proposed meeting date with times below: The VA Office of Inspector General (DIG) has completed fieldwork for the Audit of the Health Care Enrollment Program at VHA Medical Facilities (Project No. Our draft report is in the final stages of VAFDIG management review and approval, and once approved will be issued for comment and concurrence. An informal Statement of Findings was issued to key stakeholders on May 11, 2017 to facilitate a constructive conversation in advance of report issuance. At this exit conference, DIG will discuss the Statement of Findings and outline the facts and key information used during the audit to formulate our findings and recommendations. Meeting date, Friday, May 26th, 9-10:00am, 10?11:00am, ill-Noon, Noon?1:00pm, 1?2:00pm or 2- 3:00pm Please provide your availability and the name of attendee(s) by C05 16 May. If you have any questions or concerns, please contact me and Nancy Holly (202) 461-6773. Andrea Andrea E. Moodie Staff Assistant Management Review Service (10E1D) Phone: (702) 449?9911 STATEMENT OF FINDINGS Audit of the Health Care Enrollment Program at Medical Facilities VA Office of Inspector General (Project Number 2015-05379-D2-0306) STATEMENT OF FINDINGS May 11, 2017 This material is provided for discussion of facts and key information used to formuiate VA Of?ce of Inspector General?s findings and recommendations. This written material is subject to Office of Inspector General revisions; accordingly; recipients of this material must not, under any circumstances, show or release its contents for purposes other than review from those with responsibility for the subjects it discusses- It must be safeguarded to prevent publication or other improper disclosure of the information it contains. This material and all copies of it remain the property of the VA Office of inspector General. OIG Contacts: Steven Wise, Director, 202-461-4773;, Steven.Wise@va.gov Shawn Steele, Audit Manager, 202-632-6653; STATEMENT OF FINDINGS STATEMENT OF FINDINGS Objective: The purpose of the audit was to evaluate controls over the health care enrollment program administered at VA medical facilities and determine if enrollment actions were processed timely and supported by required documentation. Criteria: The Veterans Health Administration (VHA) provides comprehensive health care to eligible veterans. Most veterans must apply and be determined eligible to be enrolled for VA health care. Eligibility for enrollment is determined by evaluating evidence of qualifying military service and financial need. A veteran may submit an enrollment application through various methods. such as by mail, telephone, through website. or by applying at a VA medical facility. When an application is received at a facility, VHA policy requires enrollment staff to enter the application information into VistA, which creates a local registration and generates a preliminary enrollment record.I VistA transmits application information daily to the Enrollment System. If the veteranis military service or ?nancial need cannot be veri?ed at the time 1 van Directive 2012-001. Paragraph (4), and (5) 2 STATEMENT OF FINDINGS STATEMENT OF FINDINGS of application, the associated enrollment record will remain in a pending status while additional information is gathered. Facility enrollment staff should make every effort to obtain a veteran?s service record information.2 If the applicant cannot provide a record of military service, enrollment staff should query VistAis Hospital Inquiry module and the Veterans Information Solution application, which may enable facilities to obtain veteran eligibility information from other VA and Department of Defense systems.3 If eligibility cannot be determined, the medical facility is responsible for conducting periodic follow-up with the applicant. Once eligibility and income are confirmed, the applicant is enrolled in health care program. Incomplete applications may be closed 1 year after the applicant is noti?ed that additional information is needed to complete the enrollment. What We Did/Scope and Methodology: We conducted our audit work from December 2015 through May 2017. We reviewed application enrollment activities at VA medical facilities for the period from October 2014 through September 2015. fvna Handbook 160mm, Paragraph 7(a) 3' VHA Procedurc Guide l??lA?i, Chapter 3 'Verp?ttng Veteran?s Military Service, Section2 3 STATEMENT OF FINDINGS STATEMENT OF FINDINGS To accomplish the audit objective, we performed the following: I We reviewed a statistical sample of 275 Corporate Data Warehouse (CDW) FY 2015 health care enrollment record transactions. We performed automated testing procedures on the extracted data and obtained patient utilization history information from multiple VA information systems. We solicited each VA medical facilities associated with. our sample to obtain copies of health care applications and local standard operating procedures. This resulted in review of 106 out of 144 unique medical centers, including facilities under their administrative control, such as multiple divisions and clinics. We interviewed and obtained relevant testimonial information from over 100 employees in Member Services Division, the Health Eligibility Center (HEC), and various VA medical facilities nationwide. We performed site visits at eight VA medical facilities from January through June 2016 to the following locations: Washington, Tampa, Minneapolis, Memphis, Houston, TX, Salt Lake City, Seattle, and Beckley, WV. During our site visits, we conducted observations of the eligibility and enrollment sections to obtain an understanding of the local workflow processes. We also interviewed management and staff regarding topics related to our audit objective. 4 STATEMENT OF FINDINGS STATEMENT OF FINDINGS Dom Reliability We used computer-processed data from CDW, which were significant for our planned determination whether applications for enrollment were processed timely and supported by required documentation. By comparing these CDW data to applications obtained from VA medical facilities, we determined that only 127 of 275 records in the sample represented FY 2015 applications for enrollment. (The CDW indicator concerning whether a prior enrollment record existed was not reliable.) We used the con?rmed set of 127 records to estimate the universe of FY 2015 applications for enrollment and to assess processing timeliness. However, we determined that CDW data were not suitable for assessing whether these records were supported by required documentation. Except as previously noted, we believe the data we used were suf?ciently reliable for this report- Finding 1: Health Care Enrollment Program Needs Improvement What We Found: VHA did not provide effective governance necessary to ensure oversight and control over the health care enrollment program at its medical facilities.?4 4 According to OMB Circular A?l23, management is reaponsible for the establishment of a governance structure to effectively implement, direct? and oversee implementation of the Circular, and all the provisions of a robust process or risk management and internal control. 5 STATEMENT OF FINDINGS STATEMENT OF FINDINGS VHA delegated responsibility and required medical facilities to establish procedures for processing enrollment applications without implementing effective processes and structure to monitor the activities of its medical facilities. We determined that only 38 of 106 VA medical facilities sampled during the audit and only 5 of 8 visited had established local enrollment policies or procedures.5 However, many of the local practices con?icted with national policies. Applicant information was not always entered into VA systems and follow?up procedures were inconsistent. Con?icts between local practices and national policies occurred because VHA lacked effective governance over the health care enrollment program. VHA lacked appropriate guidance; oversight} and monitoring necessary to ensure a standardize enrollment process. In addition, formal training was not provided to eligibility and enrollment staff at VA medical facilities. As a result of local variances in processing health care applications? VHA does not have reasonable assurance that veterans receive consistent or timely enrollment decisions at VA medical facilities nationwide. 5 We reviewed local policies at each facility associated with our statistical sample of FY 2015 enrollment actions. This method resulted in review of 106 of 144 unique VA medical centers, including their multiple divisions and clinics when applicable. Additional information on our methodologies is available in Appendixes El and C. 6 STATEMENT OF FINDINGS 100 101 102 103 104 105 106 STATEMENT OF FINDINGS Enrollment-Program Net Consistently Implemented VHA policy required the health care facility director ensure local policies are in place outlining the requirements for processing enrollment applications in a timely We determined that only 33 of 106 VA medical facilities sampled during the audit and only 5 of 8 sites visited had established required local enrollment policies or procedures. The remaining 68 facilities did not establish required policies and relied on enrollment directive. However. that directive only established requirements for timely entry into VA systems and did not outline steps to process applications for health care enrollment. At locations that did have enrollment policies and procedures, we identi?ed signi?cant variations in the methods used to process health care enrollment applications. several of which were in con?ict with VA criteria. This included practices that delayed or avoided entering an applicant?s information into Vista and inconsistent followuup procedures. Applicants Not Recorded in Vista VHA policy required that all applications, regardless of method of submission, be processed into VistA.7 Entering an applicant?s information van Directive 2012?001. Paragraph VHA Directive 2012-001, Paragraph 40)) 7 STATEMENT OF FINDINGS 107 103 109 115 116 117 118 119 120 121 122 123 124 STATEMENT OF FINDINGS into the system creates an enrollment record and verifies the information against other automated resources, including shared VA and Department of Defense systems. We determined 9 of the 38 facilities with established local enrollment policies adopted practices that delayed or did not require entering an applicant?s information into VistA. By not entering the applicant?s information, enrollment staff delayed or, in some instances, may have inadvertently prevented obtaining available evidence to validate the applicant?s eligibility for VA health care. At two of the eight visited medical facilities, if the applicant did not provide adequate evidence of military service and evidence was not otherwise available to the facility, enrollment staff were instructed not to enter the application into VistA. The applicant?s intent to apply was not otherwise recorded and follow-up was not conducted. I The Salt Lake City, UT, VA Medical Center (VAMC) had a written policy in place for 10 years that specifically stated that veterans were not to be loaded into VistA until eligibility for health care can be established.3 The individual was essentially turned away if the application was incomplete. Enrollment staff would not enter the 3 Following our February 2016 site visit to Salt Lake City, VAMC management revised the policy to eliminate this practice and provided a copy of the new procedures to the DIG. 8 STATEMENT OF FINDINGS 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 STATEMENT OF FINDINGS applicant?s information into VistA and the facility kept no record of the applicanti intent to apply. I The Puget Sound Health Care System had a. standard practice of providing applicants with information cards that identi?ed resources for obtaining military information. Enrollment staff would not enter the applicant?s information into VistA and the facility kept no record of the applicant?s intent to apply. The applicant was essentially turned away and VA initiated follow-up would not occur. Another facility. the Minneapolis Health Care System, delayed health care enrollments by not interviewing individuals that applied in-person and deferring the entry of the applicants? information into VistA to a later date. VHA Procedure Guide 1601A outlines procedures for interviewing an individual during an in-person application for enrollment};I Once interviews are completed, VHA Directive 2012-001 states that local policies must ensure all applications completed during a face?to-face . . . . . 10 encounter with a veteran are immediately processed into thA. All applications submitted to the Minneapolis facility were organized in a file cabinet and processed in the order in which they were received. Enrollment staff would then enter applications into VistA at a target rate 9 VHA Procedure Guide 16011401, Chapter 2, Section B, Paragraph 1(d) vna Directive 2012?00 1, Paragraph 9 STATEMENT OF FINDINGS 144 145 146 147 148 149 150 151 152 153 154 155 156 157 153 159 160 STATEMENT OF FINDINGS of four applications per hour. This processing method was used even if the individual applied in?person at the facility. Enrollment staff did not interview the applicant as required and did not enter the registration information into VistA while the applicant was present. As a result of this practice, initial processing was delayed several weeks and the applicant?s opportunity to provide additional information while at the facility was lost. Thus. this non-veteran-centric process allowed the applicant to leave the facility without knowing his or her preliminary enrollment status or understanding what additional information was needed to complete their enrollment. VHA lacked a National Policy for Follow? Up Activities VA has the authority to close incomplete applications 1 year after the applicant is notified that additional information is needed to complete the enrollment.11 This authority was not implemented by national policy.12 VHA Procedure Guide 1601A states that if an application cannot be processed to a ?nal determination: it should be coded as pending during a period of follow-up where the applicant may provide additional information 11 Title 33 use 5102 12 A February 10, 2016:, memorandum noted that the HEC had developed a process to execute the authority. Nationwide guidance was not otherwise implemented. 10 STATEMENT OF FINDINGS 161 162 163 164 165 166 167 168 169 170 1?1 172 173 174 175 176 177 178 179 STATEMENT OF FINDINGS in support of his 01? her enrollment.? Of the 33 medical facilities with established enrollment policies, 9 had also implemented follow?up practices. We found that 5 of those 9 facilities established practices that, contrary to VA authority, closed incomplete applications before the applicant?s 1 year period expired. For example, at the Minneapolis Health Care System, enrollment staff provided applicants a return mail envelope with instructions on how to obtain evidence of military service. The entire burden of follow-up was transferred to the applicant and no other contact was made by facility enrollment staff. Conversely at the Tampa VAMC, enrollment staff tracked follow-up activities and contacted applicants up to four times via mail in a 90?day period. While the facility made aggressive attempts at follow?up, the facility truncated the codified allowance of 1 year down to 90 days. Additionally, three medical facilities had procedures for coding an applicant?s record as a status other than pending after a short or unsuccessful period of follow-up. For example: I A Palo Alto Health Care System memorandum directed enrollment staff to provide the applicant with a 30-day follow-up period, after which the application would no longer be considered. The memorandum further 13 VHA Procedure Guide 1601A.02, Chapter 2, Section 1, Paragraph 11 STATEMENT OF FINDINGS 180 181 182 183 184 185 186 187 188 189 190 191 192 193 STATEMENT OF FINDINGS stated that if the facility was not in receipt of supporting documentation within 60 days, enrollment staff may delete messages containing the veteran?s data or otherwise dispose of the ?le. i The Tampa VAMC had a local policy memorandum to send the veteran a final letter and place the veteran in a Humanitarian status if no response was received within 90 days.14 Humanitarian status, intended for emergent care, could result in a veteran incorrectly receiving a bill for care. The policy had a qualifying statement if the veteran was actively obtaining documentation, the record should remain open. I The Puget Sound Health Care System had an unwritten 90?day rule. If the facility was not in receipt of supporting documentation within a 90?day period, the record was changed from pending to ineligible or humanitarian. A status of ineligible could prevent applicants from receiving care. 14 Title 33 U.S.C. 1734 authorizes VHA to provide hospital care or treatment as a humanitarian service to persons having no eligibility and in need of emergency care while at a VA medical facility. 12 STATEMENT OF FINDINGS 194 i 195 196 197 198 199 200 201 202 203 204 205 205 207 208 209 210 211 STATEMENT OF FINDINGS Why this occurred? - VHA Locked E??ective Governance Inconsistent processing of health care applications by VA medical facilities occurred because VHA lacked effective governance over the health care enrollment program. The Of?ce of Management and Budget Circular A-123 requires management to establish and maintain internal controls necessary to achieve the objectives of effective and ef?cient program operations. Management is responsible for establishing control activities such as policies and procedures; monitoring of the program operations; and a control . . . . 1' environment that Includes approprtate staf?ng and trammg. 3 Inadequate Guidance and Oversight VHA did not have national guidance standardizing the application receipt and entry process at medical facilities. VHA has provided national guidance on requirements for health care eligibility and enrollment priority groups, timeliness standards for application processing, and the requirement for each 16 VA medical facility to have an enrollment coordinator. However. we did not identify sufficient guidance for eligibility and enrollment staff that establishes the minimum steps to take from the time a veteran submits an 15 OMB Circular A423, Section II. . 1'5 Public Lavv 104-262, VHA Directive 2012?001. and VHA Directive 1175, respecttvely. 13 STATEMENT OF FINDINGS 212 213 214 215 216 217 213 219 220 221 222 223 224 225 226 227 223 229 230 231 STATEMENT OF FINDINGS application. through data entry: and follow-up. Establishing well?de?ned guidance would help ensure veterans receive equitable consideration for health care nationwide. In addition, Member Services Division did not provide adequate oversight or monitoring of the health care enrollment program administered at VA medical facilities. Monitoring was largely deferred to the medical facility leadership and the HEC. In addition, the only performance metric we identi?ed that was tracked and reported nationally was the timeliness of online applications for enrollment. We did not identify any other means of national oversight. The Acting Executive Director of Member Services Division acknowledged that they did not review enrollment procedures or otherwise monitor the health care enrollment process nationwide. The Director further stated that VHA lacked active case management for health care enrollments and the process could be improved by implementing standardized tools. Inadequate Training for Enrollment Sta)?? VHA did not ensure that formal training was provided to eligibility and enrollment staff who processed applications for health care at VA medical facilities. The HEC academy offered training sessions for ?eld facility staff responsible for eligibility and enrollment activities. Training recipients were 14 STATEMENT OF FINDINGS 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247r 248 249 250 STATEMENT OF FINDINGS expected to train others at their facilities. At six of the eight facilities we visited. at least one staff member had attended the HEC academy. However. in many instances, the attendee was not a front line staff member who processed enrollment applications on a day-to-day basis. For example. at one facility, the Director of Health Plan Management attended the HEC academy. When interviewed. the individual could not recount the standard procedures for entering a veteran?s application into VistA. Given that those who attended the HEC academy were expected to train other staff at the facility. day-to-day eligibility and enrollment staff likely would have received greater bene?t from attendance. Enrollment training at VA medical facilities primarily consisted of on?the?j ob training during live application registrations. We noted that facility enrollment staff modeled many inappropriate practices in this manner, which often became norms for the local process. For example, the HEC required the use of ?Register a Patient? function in VistA when uploading application data because it prevented duplicate record creation and provided quicker access to veteran information that may already be in the system. However. we found several facilities were using the prohibited ?Load! Edit Patient Data? function because it was the method taught through 15 STATEMENT OF FINDINGS 251 252 253 254 255 256 257 258 259 260 261 262 263 2644 265 266 267 STATEMENT OF FINDINGS on?the-job experience.? Locations that used this function may have created duplicate records in VistA that ultimately impacted the individual?s final enrollment decision. What Resulted? Enrollment Process Not Veteran Centric Ineffective governance of the health care enrollment program resulted in f? ?T?icient service activities that were not veteran?centric and did not provide 3 to those seeking access to health care. As a result of decentralized guidance, veterans? health care enrollment experiences, such as the extent of assistance provided and the timeliness of enrollment decisions, varied based upon the medical facility Where the veteran submitted his or her application. In addition, due to the lack of adequate training, facility enrollment staff did not use correct procedures when processing health care applications that could ultimately affect the enrollment decision. Unless a national process is implemented and enrollment staff are trained to follow those procedures, VHA cannot ensure that all veterans are provided equal treatment when applying for enrollment in VA health care. The ?Register a Patient? function queries other systems, including the Master Veteran Index and the Enrollment System, and links to preexisting information. The ?Loadx?Edit Patient Data? function. does not make these queries and generates a unique record. 16 STATEMENT OF FINDINGS 268 269 270 2'21 272 273 274 275 276 27? 278 279 280 281 282 283 284 285 286 287 STATEMENT OF FINDINGS Conclusion: VHA needs to implement effective governance to ensure VA medical facilities apply healthcare enrollment procedures uniformly nationwide. VHA has relied on medical facilities to establish their ovvn procedures for processing enrollment applications. However. many facilities did not implement local policies. Additionally? the absence of standard guidance. adequate oversight controls: and formal training permitted individual medical facilities to implement local practices that were not veterandcentric, such as delaying entry of an applicantis request for healthcare enrollment into VistA and reducing follow?up periods for pending applications. Potential Recommendations 1. We recommended the Under Secretary for Health develop standardized national procedures for the health care enrollment program at VA medical facilities. 2. We recommended the Under Secretary for Health implement national oversight of the health care enrollment program to continually review operations and performance of VHA medical facilities. 3. We recommended the Under Secretary for Health provide mandatory and standardized training on eligibility and enrollment to ensure health care applications are processed accurately and timely. STATEMENT OF FINDINGS 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 STATEMENT OF FINDINGS Finding 2: VHA Needs to Improve the Reliability of Health Care Enrollment Data What We Found? The reliability of health care enrollment data needs improvement. VHA data systems did not permit the identi?cation of new enrollment applications or provide the basis for independent testing or conclusions on timeliness or supporting docmnentation. This occurred because VistA is decentralized and cannot reliably identify new applications for enrollment versus other registration actions involving previously enrolled veterans. Unless internal controls and program data are strengthened, VHA leadership cannot accurately track enrollment performance nationally or make reliable program-level decisions using this data. VistA Enrollment Data Not Reliable We identified a population of 427,000 records from VistAis patient enrollment ?le that should have represented applications submitted during FY 2015. Based on available data definitions; these records were coded as not having a prior enrollment history. However? we reviewed a statistical sample of these records against original applications obtained from VA medical facilities and identi?ed that many records were prior enrollment actions and were not FY 2015 health care applications. 18 STATEMENT OF FINDINGS 308 309 310 311 312 313 314 315 315 317 318 319 320 321 322 323 324 325 326 327 STATEMENT OF FINDINGS Speci?cally, we found that only 127 (46 percent) of 275 records reviewed were associated with FY 2015 health care applications. Overall, we projected that only about 197,000 of 427,000 records in our universe actually represented FY 2015 health care applications. Other sample records included inter?facility consults and facility transfers for previously enrolled veterans, as well as compensation and pension medical exams, according to VA of?cials. We concluded that VistA fields designed to identify prior enrollments was not reliable. Further, because it was necessary to use supporting documentation to identify a valid sample for review, we determined that VistA data were not suitable for assessing whether these records were supported by required documentation. We also could not draw conclusions on the timeliness of health care applications. Only 67 (53 percent) of 127 documents associated with 2015 health care applications had a timestamp necessary to evaluate the time from receipt to entry into VistA. Why this occurred? VH4 Locked System Functionality and Internal Controls The lack of tranSparency in the health care enrollment program occurred because VHA did not ensure that quality data and processes were available to track application processing performance. 19 STATEMENT OF FINDINGS Of?ce of Management and 328 329 330 331 332 333 334 335 335 337 338 339 340 341 342 STATEMENT OF FINDINGS Budget Circular A-123, issued December 2004, states that relevant and reliable information should be communicated to personnel at all levels Within an organization. Also, assessing the effectiveness of internal controls should occur in the normal course of business, in addition to periodic reviews, . . . . rs reconciliations, or comparisons of data. VistA is a decentralized system that does not automatically share data between facilities. An individual needs to be entered into a specific facility?s VistA system in order to receive care regardless of previous history with VA at other locations. Facilities may share patient data, but this requires VistA users to actively pull the data from one facility into another. In addition, according to VA enrollment personnel, registration actions such as facility transfers or registrations in Choice Program may cause an individual?s record to appear new. In these cases, VistA data cannot reliably distinguish new applications for enrollment versus transactions affecting previously established enrollment records. It OMB Circular A-123, Section II, and a. 20 STATEMENT OF FINDINGS 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 353 359 STATEMENT OF FINDINGS What Resulted? VHA Cannot Monitor Enrollment Activities The limitations of the VistA system signi?cantly impacted ability to collect and maintain accurate enrollment data. as well as identify systemic issues in the enrollment. program. Further, VistA fed data into the authoritative system of record. the Enrollment System, further impacting ability to produce reliable data. Unless enrollment data are improved. VHA leadership cannot accurately track and report enrollment program performance nationally. Conclusion The reliability of health care enrollment data needs improvement. Limitations in the Vista system did not allow us to make a conclusive determination if FY 2015 enrollment applications were timely or supported by required documentation. This occurred because VHA lacked effective internal controls over health care enrollment data. Unless program oversight and VistA enrollment data are improved, VHA leadership cannot make reliable program-level decisions using the data. 21 STATEMENT OF FINDINGS 360 361 362 363 STATEMENT OF FINDINGS Potential Recommendation 1. We recommended the Under Secretary for Health develop and execute a process to distinguish new applications for health care enrollment in VistA from other registration data. 22 STATEMENT OF FINDINGS 364 365 366 367 363 369 370 371 372 373 374 375 376 377 378 379 330 331 382 383 STATEMENT OF FINDINGS Appendix A Statistical Sampling Methodology To evaluate if enrollment actions were processed timely and supported by required documentation, we conducted a simple random sample. For each sample case, we requested supporting documentation from VA medical facilities and attempted to trace application dates to assess the accuracy of data in the CDW. Population The Enrollment System is authoritative system of record for veterans? health care enrollment data. However, the Enrollment System receives data feeds from VA medical facilities using an enrollment module in VistA. This is the prirnary point-of-entry system for enrollment staff who process applications in the field. As such, the scope of this audit focused on enrollment data originating through VistA. CDW is a national repository of data from VistA and several other VHA clinical and administrative systems. enrollment data consists of information such as enrollment status, application date, priority group, and originating facility. Historical changes to enrollment status are captured by adding a new entry to the individualis record each time a change is made. For audit purposes, we planned to focus on the population of CDW records associated with FY 2015 enrollment applications that were submitted 23 STATEMENT OF FINDINGS 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 STATEMENT OF FINDINGS through VA medical facilities between October 1, 2014, and September 30, 2015. This sampled population, based on available data descriptions, consisted of 426,657 records. However, we identi?ed that the population included records for veterans who were previously enrolled. We estimated a population of about 197,000 applications for enrollment for FY 2015. Sample Design We reviewed 275 randomly selected enrollment records for applications submitted between October 1, 2014, and September 30, 2015. However, we identi?ed that only 127 (46 percent) records in our sample met the parameters of our objective. As a result, our analysis and conclusion were based on that subset of sample data. Weights We calculated estimates in this report using weighted sample data. Sampling weights are computed by taking the product of the inverse of the probabilities of selection at each stage of sampling. We used WesVar to calculate population estimates and associated sampling errors. WesVar employs replication methodology to calculate margins of error and con?dence intervals that correctly account for the complexity of the sample design. 24 STATEMENT OF FINDINGS 404 405 406 407 408 409 410 411 412 413 STATEMENT OF FINDINGS Projections and Margins ofError The margins of error and con?dence intervals are indicators of the precision of the estimates. If we repeated this audit with multiple samples. the con?dence intervals would differ for each sample. but would include the true population value 90 percent of the time. Our review indicated that only 197,000 enrollment records actually represented FY 2015 health care applications. The 90 percent con?dence interval for the estimate of the population of FY 2015 health care applications is between 176,000 and 218,000. For our projection, we used the midpoint of the 90 percent con?dence intervals. 25 STATEMENT FINDINGS