California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR The Data Standards and Quality Control Unit (DSQC) of the California Cancer Registry (CCR) performed a reabstracting audit remotely from the CCR offices in Sacramento in 2011. The audit utilized the Electronic Medical Record (EMR) remotely. Two facilities were audited; a rural community hospital in Region 2 and a large urban university medical center in Region 5. The audit design evaluated the efficacy of performing an audit remotely while performing a routine reabstracting audit. The audit had two separate and equally important elements; audit performance and data quality. The audit performance component evaluated the efficiency of remote access, reliability and completeness of the EMR information accessed, and quality of the audit being performed in this manner. The data quality component evaluated the quality and completeness of the data reported. There were 52 data items reabstracted from each source record for a total of 2600 possible discrepancies. The list of audited data items are found on Table 1. This audit focused on diagnosis year 2010 cases. Depending on the facility, completion of 2010 cases, each hospital had approximately 55 recently abstracted and transmitted cases selected for this audit. The selected cases consisted of 20 colon and rectum (C18.0 – C20.9) which included 16 colon cases (C18.0 – C19.9) and 4 rectal cases (C20.9). There were 20 lung and bronchus (C34.0 – C34.9), and the remaining cases divided between malignant melanoma of the skin (C44.0 – C44.9) and kidney cases (C64.9). This breakdown is demonstrated in Table 2. Performance Methodology Per audit protocol, each facility was sent an audit initiation letter which outlined the request for access to the EMR as well as the specific elements within the EMR that needed to be reviewed and reabstracted. Each facility required a two week notice prior to commencing the audit in order to set up remote access. Each auditor was required to sign and submit a Confidentiality Statement for their respective facility. One facility required a list of cases that were to be abstracted, while the other facility gave the auditor access to the facilities entire database. Each auditor used CITRIX to access each facility’s electronic medical record (EMR). The rural community hospital EMR consisted of software named Mosaqi, Signature, and Inmet. The large urban university medical center utilized CERNER PowerChart and HPF. Once approval was obtained and access granted, the auditors was given access to their respective EMR’s for a period of 30 days. December 28, 2011 1 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR While it was requested that each facility provide the auditor with instructions on how to log into each system, how to navigate between the two systems, and how to find the documents that were needed to complete the audit, this was unfortunately only performed by one of the facilities audited. Audit Performance Results Hospital A This facility is a medium sized community hospital in a rural location in the central valley. Hospital demographics include: • • • Approximately 900 cases abstracted and submitted to the CCR annually ACoS accreditation status: Not approved Cancer registry staff is hospital personnel working remotely (off site) The EMR at this hospital was difficult to navigate and it was time-consuming locating information required to reabstract cases for the audit. There was a significant amount of miscommunication between the reporting facility contact personnel, IT Department and the Cancer Registry. This caused an initial delay in gaining access to the EMR. There are advantages and disadvantages to performing this audit remotely. Some of the advantages to performing this reabstracting audit remotely for this facility include: • • Cost savings in the form of travel. Ability to perform other tasks and assignments, while conducting the audit Disadvantages of performing reabstracting audits remotely on this facility include: • • • Communication with the facility assigned to an administrative assistant in the oncology business services department was difficult at times because she did not have a complete understanding of the EMR and the registry and therefore did not have the experience to know what exactly the auditor needed. Communication with the cancer registry representative was also frustrating and they provided little orientation of the EMR. The cancer registry liaison eventually sent an email with a basic overview of where to look for documents. Several different medical record systems needed to be accessed as different information was housed in different systems. December 28, 2011 2 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR • • Lack of orientation to the EMR, which was due to above noted issues, resulted in an extensive amount of time attempting to locate documents such as lab reports or pathology reports. In some instances, these documents were never located. For example: Patients frequently had over 100 lab results that required opening individually, one at a time, in order to review. This hospital’s EMR is not user friendly and therefore an on-site audit would not be an option because the auditor would need several days if not weeks to completely audit a sufficient number of cases to provide adequate results. Hospital B This facility is a large university medical center located in urban Southern California. The hospital demographics include: • • • Approximately 2400 cases abstracted and submitted to the CCR annually ACoS accreditation status: Approved as a Teaching Hospital Cancer Program Cancer registry staff is hospital personnel located on site The experience at this hospital is different than of hospital A. The EMR at this hospital was easy to navigate and to identify required information to completely reabstract cases for the audit. There are advantages and disadvantages to performing this audit remotely. Some of the advantages to performing this reabstracting audit remotely for this facility include: • • Cost savings in the form of travel. Ability to perform other tasks and assignments, while conducting the audit Disadvantages of performing reabstracting audits remotely on this facility include: • Substantial amount of documents and information available resulted in a slower than usual review of chart and the completion of the audit. December 28, 2011 3 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Data Quality Overall – Both Facilities There were a total of 190 discrepancies identified on this audit which resulted in an accuracy rate of 96.9%. A majority of the discrepancies occurred in the colon and rectum cases of which there were 67 (35.3%) discrepancies noted. Kidney cases had the second highest number of discrepancies with 47 (24.7%); Lung was next with 45 (23.7%) discrepancies, and melanoma of the skin had the least number of discrepancies with 31 (16.3%). This is demonstrated in Table 3 and Graph 1. The data items that make up the top discrepancies are: • • • • • • • • • • Surgery Primary Site CS Extension Diagnostic Stage Summary Date of Diagnostic Staging Date of Diagnosis CS Tumor Size Date Primary Surgery Diagnostic Stage Hospital Histology Type ICD-O-3 Scope Regional Lymph Node Surgery 16 (8.4%) 12 (6.3%) 12 (6.3%) 11 (5.8%) 11 (5.8%) 10 (5.3%) 10 (5.3%) 10 (5.3%) 10 (5.3%) 8 (4.2%) The distribution of all discrepancies identified on this audit can be found on Table 4. Surgery of the Primary Site had the most discrepancies noted with 16 discrepancies. This represented 8.4% of all discrepancies noted on the audit. Seven (43.8%) of these discrepancies identified occurred in the colon cases. There were three (18.8%) discrepancies noted in each of the remaining sites; the lung, melanoma of the skin, and kidney cases. Of interest, 14 (87.5%) discrepancies in Surgery of Primary Site occurred in one facility. There were seven (43.8%) cases that were recoded from a specific surgery (such as surgery code 40- hemicolectomy) code to a high, more specific surgical code (41- hemicolectomy with resection of small bowel). There were five (31.3%) cases that were changed from surgery code 00 (no surgery) to a specific surgery code such as code 26-polypectomy in the colon cases, or December 28, 2011 4 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR surgery code 23-cyrosurgery in the kidney cases. All five of the discrepancies that were recoded from 00 to a specific surgical code were situations where the abstractor had originally coded the surgery as an incisional biopsy in another field. There were two (12.5%) cases that were recoded to a lower, less specific surgical code. The remaining two (12.5%) cases were recoded from a specific surgical code to 00- no surgical procedure. The data item CS Extension had the second number of discrepancies noted with 12 (6.3%) discrepancies. Five (41.7%) discrepancies occurred in the colon cases, four (33.3%) were identified in the lung cases, two (16.7%) occurred in the melanoma of the skin, and the remaining one (8.3%) discrepancy occurred in the kidney case. Four (33.3%) of these discrepancies were the result of the CS Extension code being recoded to 999 (Unknown) due to a lack of information in order to code a specific extent of disease. An additional four (33.3%) cases were recoded form a higher, more specific code to a lower less specific extension code due to either a lack of information or misunderstanding of the definition of the code. These four cases excluded a recode to 300, which is discussed below. There were two (16.7%) discrepancies in which were recoded from a lower, less specific code to a higher, more specific code. The remaining two (16.7%) discrepancies were recoded to CS Extension 300. One of these recodes occurred in the lung cases and was recoded from a higher extension code to an extension code 300, which is defined as localized, NOS. In this case, there was limited information available to code any specific extent of disease code. The other discrepancy occurred in a melanoma of the skin case, in which the original code was a lower code and was recoded to the higher code 300, which is defined as “Reticular Dermis Invaded; Clarks Level IV.” The information available was a statement of “Clarks Level IV” located in the pathology report. There were 10 (5.3%) discrepancies noted in the data item Histology. Of interest is that six (60%) of these cases occurred in one facility and in one site, kidney. These were the result of not properly identifying and coding papillary renal cell carcinoma (8260) and clear cell carcinoma (8310). Of the remaining four discrepancies, two (20%) occurred in colon cases, one (10%) in lung cases, and one (10%) last case occurred in melanoma of the skin which resulted in the case being deemed non-reportable and the case was deleted. December 28, 2011 5 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Hospital A Of the 190 discrepancies noted on this audit, Hospital A had 73 (38.4%) discrepancies and the top discrepancies were: • • • • • • • • CS Extension Date of Diagnosis CS Tumor Size CS Site Specific Factor #4 CS Site Specific Factor #1 Histology Type ICD-O-3 CS Lymph Nodes CS Site Specific Factor #3 9 (12.3%) 7 (9.6%) 6 (8.2%) 6 (8.2%) 5 (6.8%) 4 (5.5%) 4 (5.5%) 4 (5.5%) Of the 73 discrepancies, 35 (47.9%) were identified in the colon cases. There were 20 (27.4%) discrepancies noted in lung cases, 11 (15.1%) in melanoma of the skin, and the remaining seven (9.6%) cases in the Kidney cases. The distribution of the discrepancies by site for Hospital A are demonstrated in Table 5 and Graph 2. Hospital B There were 117 (61.6%) discrepancies identified at Hospital B. The top discrepancies were: • • • • • • Surgery Primary Site Diagnostic Stage Summary Diagnostic Stage Hospital Date Diagnostic Staging Date Primary Surgery Histology Type ICD-O-3 14 (12%) 12 (10.3%) 9 (7.7%) 9 (7.7%) 8 (6.8%) 6 (5.1%) Of the 117 discrepancies, 40 (34.2%) discrepancies were noted in the kidney cases. There were 32 (27.4%) discrepancies identified in the colon cases, 25 (21.4%) discrepancies noted in lung cases, and the remaining 20 (17.1%) in melanoma of the skin. The distributions of the discrepancies by site for Hospital B are demonstrated in Table 6 and Graph 3. December 28, 2011 6 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Resource Allocation, Benefit, and Core Team • • Projected Hours to Complete Project: Estimated at 225 hours from project kick-off to Final Report submission. Actual Hours to Complete Project: Approximately 405 hours. The business benefit of this project is that a data quality baseline has been established for these types of cases in the CCR data base. The core team included Kyle L. Ziegler, CTR, Cheryl Moody, CTR, and Lois Inferrera, CTR CONCLUSION Performing audits remotely can be challenging depending on the type of EMR that is utilized. As experienced in this audit, one EMR was unconventionally difficult to utilize and navigate and the auditor found it frustrating to located required documents needed to conduct the audit. The other auditor had a much different experience. The other EMR was effortless and straightforward and the needed information was readily accessible. Nevertheless, auditing remotely is currently the most efficient and cost effective method of auditing. The data quality results of this audit is concerning. The data items Surgery of the Primary Site, CS Extension, and Histology are of particular concern. The data item Surgery of the Primary Site is not currently a visually edited data item and there has been no training or educational presentations made on this data item in many years. The effects of a lack of training for this data item are beginning to show. The discrepancies in this data items are novice errors and need attention. It is recommended that a two pronged approach be taken to resolve these types of discrepancies. First, training needs to be considered and an educational module needs to be created that outlines common coding errors and how to properly code surgical procedures. The second approach is to recommend the CCR reinstitute visually editing of treatment fields, and at a minimum, surgery of the primary site. An educational module should also be created for the data items CS Extension and coding Histology, emphasizing histologies commonly found in the primary site kidney. December 28, 2011 7 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Data Items by Category Tumor Data Social Security Number Date of Birth Gender Race 1 Race 2 Race 3 Data Item Place of Diagnosis Date of Diagnosis Primary Site (including subsite) Laterality Histology Type (ICD-O-3) Behavior Grade Race 4 Race 5 Spanish/Hispanic Origin Marital Status Diagnostic Confirmation Date of Surgery Surgery of Primary Site Scope of Regional Lymph Nodes Class of Case Surgery of Other Site CS Tumor Size CS Extension CS Lymph Nodes CS Mets at Diagnosis DX Stage Summary DX Stage Hospital Date Diagnostic Stage Systemic Surgery Sequence Date Radiation Therapy Regional Radiation Treatment Modality CS Site Specific Factor 1 CS Site Specific Factor 2 CS Site Specific Factor 3 CS Site Specific Factor 4 CS Site Specific Factor 5 CS Site Specific Factor 6 Tumor Marker California 1 Lymph Nodes Positive/Examined Treatment Stage Demographic Data Item Regional Radiation Boost Date Chemotherapy Chemotherapy Summary Date Hormone Hormone Summary Date Immunotherapy Immunotherapy Summary Date Other Treatment Other Treatment Date Trans Endo Trans Endo Summary Table 1 December 28, 2011 8 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Number of Cases Reabstracted by Primary Site with Distribution of Discrepancies by Case and Primary Site Cases Primary Site Total Number of Cases Reabstracted Number of Cases with Discrepancies C18.0 - C18.9 C19.9 C20.9 C34.0 - C34.9 C44.0 - C44.9 C64.9 Total 20 5 11 35 21 28 120 14 4 5 21 15 22 81 Total Cases with Discrepancies 23 21 15 22 81 Table 2 December 28, 2011 9 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Distribution of Discrepancies by Primary Site with Discrepancy and Accuracy Rates Discrepancies Discrepancy and Accuracy Rate Number of Possible Discrepancies Number of Discrepancies Discrepancy Rate Accuracy Rate C18.0 - C18.9 C19.9 C20.9 1020 255 561 44 6 17 1.1% 98.9% C34.0 - C34.9 1785 45 0.7% 99.3% C44.0 - C44.9 1071 31 0.5% 99.5% C64.9 1428 47 0.8% 99.2% Total 6120 190 3.1% 96.9% Primary Site Table 3 December 28, 2011 10 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR All Discrepancies by Primary Site - Both Facilities Combined n=190 45 47 31 Colon Rectum (C18.9 - C20.9) Kidney (C64.9) Lung (C34.0 - C34.9) Melanoma of the Skin (C44.0 - C44.9) 67 Graph 1 December 28, 2011 11 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Distribution of Discrepancies – Both Facilities Data Item Surgery Primary Site CS Extension Diagnostic Stage Summary Date Diagnostic Staging Date of Diagnosis CS Tumor Size Date Primary Surgery Diagnostic Stage Hospital Histology Type ICD-O-3 Scope Regional Lymph Node Surgery CS Lymph Nodes CS Site Specific Factor #1 CS Site Specific Factor #4 CS Site Specific Factor #6 Date Chemotherapy Class of Case CS Site Specific Factor #3 Regional Lymph Nodes Examined CS Site Specific Factor #5 Laterality Marital Status Social Security Number Spanish/ Hispanic Origin Chemotherapy Summary CS Mets at Diagnosis CS Site Specific Factor #2 Date Radiation Primary Site Race 1 Birth Date Histology Grade Radiation Boost Radiation TX Modality Regional Lymph Nodes Positive Discrepancy Percent 16 8.4% 12 6.3% 12 6.3% 11 5.8% 11 5.8% 10 5.3% 10 5.3% 10 5.3% 10 5.3% 8 4.2% 7 3.7% 7 3.7% 7 3.7% 7 3.7% 5 2.6% 5 2.6% 5 2.6% 5 2.6% 3 1.6% 3 1.6% 3 1.6% 3 1.6% 3 1.6% 2 1.1% 2 1.1% 2 1.1% 2 1.1% 2 1.1% 2 1.1% 1 0.5% 1 0.5% 1 0.5% 1 0.5% 1 0.5% Table 4 December 28, 2011 12 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Hospital A Distribution of Discrepancies Data Item CS Extension Date of Diagnosis CS Tumor Size CS Site Specific Factor #4 CS Site Specific Factor #1 Histology Type ICD-O-3 CS Lymph Nodes CS Site Specific Factor #3 Regional Lymph Nodes Examined Scope Regional Lymph Node Surgery Class of Case CS Mets at Diagnosis CS Site Specific Factor #6 Date Primary Surgery Surgery Primary Site Date Diagnostic Staging Chemotherapy Summary Spanish/ Hispanic Origin Primary Site CS Site Specific Factor #2 CS Site Specific Factor #5 Diagnostic Stage Hospital Date Radiation Radiation TX Modality Radiation Boost Discrepancy 9 7 6 6 5 4 4 4 3 3 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 Percent Table 5 December 28, 2011 13 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Discrepancies by Primary Site - Hospital A n=73 20 11 Colon Rectum (C18.9 - C20.9) Kidney (C64.9) Lung (C34.0 - C34.9) 7 Melanoma of the Skin (C44.0 - C44.9) 35 Graph 2 December 28, 2011 14 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Hospital B Distribution of Discrepancies Data Item Surgery Primary Site Diagnostic Stage Summary Diagnostic Stage Hospital Date Diagnostic Staging Date Primary Surgery Histology Type ICD-O-3 CS Site Specific Factor #6 Scope Regional Lymph Node Surgery Date Chemotherapy Date of Diagnosis CS Tumor Size Social Security Number Marital Status Class of Case Laterality CS Extension CS Lymph Nodes Race 1 Spanish/ Hispanic Origin Regional Lymph Nodes Examined CS Site Specific Factor #1 CS Site Specific Factor #5 Birth Date Primary Site Histology Grade Regional Lymph Nodes Positive CS Site Specific Factor #2 CS Site Specific Factor #3 CS Site Specific Factor #4 Date Radiation Discrepancy Percent 14 12 9 9 8 6 5 5 5 4 4 3 3 3 3 3 3 2 2 2 2 2 1 1 1 1 1 1 1 1 Table 6 December 28, 2011 15 California Cancer Registry 2011 Remote Reabstracting Audit CCR Project ID #300.65 Final Report Prepared by Kyle L. Ziegler, CTR Discrepancies by Primary Site - Hospital B n=117 40 Colon Rectum (C18.9 - C20.9) Kidney (C64.9) 25 32 Lung (C34.0 - C34.9) Melanoma of the Skin (C44.0 - C44.9) 20 Graph 3 December 28, 2011 16