An Audit Report on

Performance Measures at the Juvenile
Justice Department
August 2018
Report No. 18-042

State Auditor’s Office reports are available on the Internet at http://www.sao.texas.gov/.

 An Audit Report on

Performance Measures at the Juvenile
Justice Department
SAO Report No. 18-042
August 2018

Overall Conclusion
The Juvenile Justice Department (Department)
had significant weaknesses in a system that it
uses to manage its youth population. Those
weaknesses were a result of the Department
not following a change management process
when implementing a change within its
Correctional Care System (CCS).
As a result of that change, certain data in CCS
was unreliable. While auditors observed youth
and verified that the current location recorded
in CCS was correct for a sample of youth,
auditors identified certain errors in other key
data fields such as youth admission dates.

Background Information
Pursuant to Senate Bill 653 passed by
the 82nd Texas Legislature and signed
by the Governor, the Juvenile Justice
Department (Department) was created
on December 1, 2011, and the existing
Texas Juvenile Probation Commission
(TJPC) and Texas Youth Commission
(TYC) were abolished. Operations of
both TJPC and TYC were transferred to
the Department on its inception.
Agencies report results for their key
measures to the Legislative Budget
Board’s budget and evaluation system,
which is called the Automated Budget
and Evaluation System of Texas (ABEST).
Sources: The Department’s Web site and

the Guide to Performance Measure
The Department also uses information from CCS
Management (State Auditor’s Office
to calculate and report certain key
Report No. 12-333, March 2012).
performance measure results that are then
entered in the Automated Budget and
Evaluation System of Texas (ABEST). As a
result, factors prevented certification for six of the nine key performance
measures audited.

The Department also reported inaccurate results for two key performance
measures audited due to errors in the information it used to calculate and report
the performance measure data. One key performance measure tested was certified
with qualification.
In addition, the Department did not have adequate policies and procedures for its
performance measure collection, calculation, and reporting, and it did not conduct
sufficient reviews of its collection and calculations for the performance measures
audited. Auditors also noted user access weaknesses for systems that the
Department used in its performance measure reporting process.
Table 1 on the next page presents a summary of the findings discussed in Chapter 1
of this report and the related issue rating.

This audit was conducted in accordance with Texas Government Code, Section 2101.038.
For more information regarding this report, please contact Audrey O’Neill, Audit Manager, or Lisa Collier, First Assistant State
Auditor, at (512) 936-9500.

 An Audit Report on
Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042

Table 1

Summary of Chapters and Related Issue Ratings
Issue Rating a

Chapter

Title

1

The Department Has Significant Weaknesses in the System It Uses to Manage Its
Youth Population; However, Information Related to Current Youth Location for a
Sample of Youth Was Accurate

Priority

a A subchapter is rated Priority if the issues identified present risks or effects that if not addressed could critically affect the audited
entity’s ability to effectively administer the program(s)/function(s) audited. Immediate action is required to address the noted concern
and reduce risks to the audited entity.

Auditors communicated other, less significant issues separately in writing to
Department management.
Table 2 summarizes the certification results, discussed in Chapters 2 and 3 of this
report, for the nine performance measures tested for fiscal year 2017 and, as
applicable, the first quarter of fiscal year 2018.

Table 2

Performance Measure Results for the Juvenile Justice Department (Agency No. 644)
Related
Objective or
Strategy
Classification

Description of Performance Measure

B.1.4,
Outcome

Turnover Rate of Juvenile Correctional
Officers

2017

40.81%

Certified with Qualification

B.1.8,

Reincarceration Rate: Within One Year

2017

23.38%

Factors Prevented
Certification

B.1.9,
Outcome

Reincarceration Rate: Within Three Years

2017

41.78%

Factors Prevented
Certification

B.1.3.1,

Juvenile Per Direct Supervision Juvenile
Correctional Officer Staff Per Shift

2017

7.32

Inaccurate

Explanatory
B.1.4.1,
Output

Average Daily Attendance in JJD-operated
Schools

2017

988.89

Inaccurate

First Quarter 2018 b

961.79

Inaccurate

B.1.6.1,
Efficiency

Cost of Health Care Services Per Juvenile
Day

2017

$18.65

Factors Prevented
Certification

B.1.7.1,
Efficiency

Cost of Mental Health (Psychiatric)
Services Per Juvenile Day

2017

$1.65

Factors Prevented
Certification

B.1.8.1,
Output

Average Daily Population: General
Rehabilitation Treatment

2017

1,046.65

First Quarter 2018 b

1,030.20

Factors Prevented
Certification

B.1.8.2,
Output

Average Daily Population: Specialized
Treatment

2017

869.57

First Quarter 2018 b

791.68

Fiscal Year

Results
Reported in
ABEST

Outcome

ii

Certification Results

Factors Prevented
Certification

a

 An Audit Report on
Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042

Performance Measure Results for the Juvenile Justice Department (Agency No. 644)
Related
Objective or
Strategy
Classification

Description of Performance Measure

Fiscal Year

Results
Reported in
ABEST

Certification Results

a

a A performance measure is certified if reported performance is accurate within plus or minus 5 percent of actual performance and if it
appears that controls to ensure accuracy are in place for collecting and reporting performance data.
A performance measure is certified with qualification when reported performance appears accurate but the controls over data
collection and reporting are not adequate to ensure continued accuracy. A performance measure is also certified with qualification
when controls are strong but source documentation is unavailable for testing. A performance measure is also certified with qualification
if agency calculation of performance deviated from the performance measure definition but caused less than a 5 percent difference
between the number reported to ABEST and the correct performance measure result.
A performance measure is inaccurate when the actual performance is not within 5 percent of the reported performance, or when there
is a 5 percent or greater error rate in the sample of documentation tested. A performance measure also is inaccurate if the agency’s
calculation deviated from the performance measure definition and caused a 5 percent or greater difference between the number
reported to ABEST and the correct performance measure result.
A factors prevented certification designation is used if documentation is unavailable and controls are not adequate to ensure accuracy.
This designation also will be used when there is a deviation from the performance measure definition and the auditor cannot determine
the correct performance measure result.
b The Department reported this performance measure in ABEST on a quarterly basis; therefore, auditors tested this performance
measure for fiscal year 2017 and the first quarter of fiscal year 2018.

Summary of Management’s Response
At the end of each chapter in this report, auditors made recommendations to
address the issues identified during this audit. The Department agreed with the
recommendations in this report.

Audit Objectives and Scope
The objectives of this audit were to determine whether the Department:


Is accurately reporting its performance measures to ABEST.



Has adequate controls in place over the collection, calculation, and
reporting of its key performance measures.

The scope of this audit included six key performance measures that the
Department reported for fiscal year 2017 (September 1, 2016, through August 30,
2017) and three key performance measures that the Department reported for fiscal
year 2017 and the first quarter of fiscal year 2018 (September 1, 2017, through
November 30, 2017).
Auditors expanded the scope of the audit to include on-site visits to five
Department facilities and one halfway house due to the issues related to the
reliability and completeness of data within CCS (see Chapter 1 for discussion of
those issues).

iii

 Contents
Detailed Results
Chapter 1

The Department Has Significant Weaknesses in the
System It Uses to Manage Its Youth Population;
However, Information Related to Current Youth
Location for a Sample of Youth Was Accurate .................... 1
Chapter 2

The Department Reported Unreliable Results for Eight
Key Performance Measures Tested ................................. 5
Chapter 3

The Department Reported Reliable Results for One Key
Performance Measure Tested, But It Should Strengthen
Its Review Process ................................................... 11

Appendices
Appendix 1

Objectives, Scope, and Methodology ............................. 13
Appendix 2

Issue Rating Classifications and Descriptions .................... 18

 Detailed Results
Chapter 1

The Department Has Significant Weaknesses in the System It Uses to
Manage Its Youth Population; However, Information Related to
Current Youth Location for a Sample of Youth Was Accurate
Chapter 1
Rating:
Priority 1

The Juvenile Justice Department (Department) did not follow a change
management process when it implemented changes to its Correctional Care
System (CCS).

In January 2016, the Department implemented
changes to CCS and the associated database
system. The Department uses CCS to help track
and manage its youth population. Those
changes to CCS were intended to modify the
manner in which data for certain youth was
stored in the system; the project to make those
changes was referred to as the Multiple
Commitments Project (see text box).

Multiple Commitments Project
The purpose of the Department’s Multiple
Commitments Project was to update CCS
to make it possible to track instances in
which youths have multiple, simultaneous
state commitments.
That update was intended to improve the
Department’s monitoring of the youths’
treatment progress and length of stay
requirements and to enhance the
Department’s general external reporting
capabilities.
Source: The Department.

However, the Department did not use a change
management process when it implemented
those changes (see text box for details about
change management for information systems).
As a result, certain data in CCS was incorrectly
changed. The Department asserted that over the
next several months it identified errors in the
data related to the system changes; however,
the Department did not restore CCS with
previous data.
In addition, the Department asserted that it
attempted to determine the cause of those
system errors on at least three occasions
between December 2016 and May 2017.
However, it was unable to determine the cause,
which limited its ability to correct those errors.

1

Change Management
ISACA, an independent association that
certifies professionals in information
technology governance, defines change
management as:
“All changes, including emergency
maintenance and patches, relating to
infrastructure and applications within the
production environment are formally
managed in a controlled manner. Changes
are logged, assessed, and authorized prior
to implementation and reviewed against
planned outcomes following the
implementation. This assures mitigation of
the risks of negatively impacting the
stability or integrity of the production
environment.”
Source: ISACA’s Control Objectives for
Information and Related Technology
(COBIT), version 4.1.

The risk related to the issues discussed in Chapter 1 is rated as Priority because the issues identified present risks or effects
that if not addressed could critically affect the audited entity’s ability to effectively administer the program(s)/function(s)
audited. Immediate action is required to address the noted concern and reduce risks to the audited entity.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 1

 As a result of changes the Department made to CCS, certain data in that system
was not reliable.

There were errors in key data fields in the reports generated by CCS that the
Department uses to monitor youth information. One of those key data fields
was average daily population, which the Department uses to calculate some
performance measure results. Because the Department could not provide
sufficient evidence showing that the data in CCS was complete and accurate,
auditors selected a sample of youth records for additional testing. Auditors
identified certain errors in the CCS data for the 61 youths tested in five
secure facilities and one halfway house. For example, information related to
youth admission dates was inaccurate for 19 (31 percent) of the 61 youths
tested. In addition, the Department identified more than 75 data fields that
it determined were adversely or unexpectedly affected by the system
change.
Auditors verified that the current youth’s location listed in CCS was correct.
Specifically, the Department provided supporting documentation for all
youths tested. In addition, during on-site visits to the facilities, auditors
observed the youths still assigned at those facilities and confirmed the
accuracy of the location information for those youths.
The data accuracy and completeness issues identified increase the risk that
the Department could make decisions related to youth based on inaccurate
or incomplete information.
The Department implemented a process to fix the data errors for reporting
purposes; however, it could not ensure that the process identified and
corrected all errors for performance measure reporting.

The Department implemented a process to help improve the accuracy of the
reports it uses to calculate performance measure information. Specifically,
for those reports, the Department created a query to correct individual youth
records it identified as having errors. However, that process did not correct
the underlying data in CCS and the Department could not ensure that the
process identified and corrected all errors in those reports. Instead, that
process focused on identifying errors based on certain risk factors the
Department identified.
The Department did not adequately restrict access to CCS.

The Department did not terminate in a timely manner system access to CCS
for certain users who no longer required that access. Not appropriately
restricting user access increases the risk that unauthorized changes could be
made to CCS.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 2

 The Department has begun the process of replacing CCS.

The Department provided auditors with documentation showing that it
received transfer authority from the Legislative Budget Board for the purpose
of replacing CCS. The Department stated that it was in the early stages of
planning that procurement.
Recommendations

The Department should:


Consistently follow a change management process and ensure that it
manages all changes in a manner that protects the integrity of its data.



Develop and implement a process to ensure that its current and historical
youth-related data is accurate and complete. That should include:





A process to verify the accuracy of data in its current system; and



A data migration process to ensure that all youth data is accurately
transferred to and stored in any new system.

Complete user access reviews in a timely manner to ensure that it
removes those users who no longer need access to Department systems.

Management’s Response

Responsible Party: Chief Information Officer
The Texas Juvenile Justice Department agrees with the recommendations.


Efforts to define short term improvements to the change management
process within the agency will begin August 1, 2018, while the agency
works to more fully define IT change management policies, procedures,
roles and responsibilities, and develops the necessary training of IT and
agency staff on the revised practices. Additionally, the identification,
capture and measurement of key performance metrics will help the
agency assess its improvement efforts and better identify any further
modifications to the process. These reports will be coordinated and
reviewed with agency Executive Management to ensure effectiveness and
transparency.
Timeframe: While the agency will begin the effort on August 1, 2018, to
identify short term improvements to the change management process,
the broader review, policy development, and implementation will be
completed by July 31, 2019.
An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 3

 

The agency also agrees that special attention must be paid to the process
of verifying the accuracy of current and historical data within the
Correctional Care System, to ensure the reliability of this information as
the agency prepares to migrate to a new youth case management
application. Furthermore, the agency will incorporate audits and
verification checkpoints of the migrated data as the process evolves, to
ensure the continued accuracy and reliability of the converted data.
Leading up to the implementation of a new youth case management
system, the agency expects to operate both the existing application and
the new platform simultaneously, for a preliminary period, as an added
means of validating the accuracy of the converted data.
Timeframe: The agency has not yet defined the specifications for the new
youth case management system, however, staff will coordinate with the
selected vendor during the implementation phase to ensure a high degree
of priority is placed on the verification and migration of data from the
current system to the new system. Additionally, the agency will begin the
review of current and historical data as of September 1, 2018, and plans
to have this review completed, and audit functionality in place, by July 31,
2019. The agency expects to begin the purchase process by June 1, 2019,
and complete the implementation of the new youth case management
system by May 31, 2020, and will operate both systems in parallel
through November 30, 2021.



Although the agency currently conducts annual user access reviews with
the designated application owners, the agency will review its existing
operational procedures with regard to account management and access
authorization to identify opportunities to strengthen the controls already
in place as well as provide for an added level of review of the core aspects
of these operations.
Timeframe: The agency will complete the review of this process and
implement identified enhancements by December 31, 2018.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 4

 Chapter 2

The Department Reported Unreliable Results for Eight Key
Performance Measures Tested
The Department reported unreliable results for eight key performance
measures tested for fiscal year 2017. Specifically, factors prevented
certification for six performance measures, and the Department reported
inaccurate results for two performance measures tested.
Furthermore, three of those eight performance measures were required to
be reported on a quarterly basis. For the first quarter of fiscal year 2018,
factors prevented certification for two of those performance measures, and
the Department reported inaccurate results for the third performance
measure tested.
In addition, the Department did not have adequate policies and procedures
or perform adequate reviews for the eight performance measures tested.
Factors Prevented Certification
As a result of the data reliability issues discussed
in Chapter 1, auditors could not determine the
correct results for the performance measures that
relied on that data. Specifically, the Department
used data from CCS to track key youth
information, such as data elements used to
monitor the average daily population of youth, in
its secure facilities. The Department used that
CCS data in its calculation for six performance
measures tested. Therefore, factors prevented
certification (see text box) for the following six
key performance measures:


Reincarceration Rate: Within One Year



Reincarceration Rate: Within Three Years

Factors Prevented Certification
A factors prevented certification
designation is used if documentation
is unavailable and controls are not
adequate to ensure accuracy. This
designation also will be used when
there is a deviation from the
performance measure definition and
the auditor cannot determine the
correct performance measure result.
Source: Guide to Performance
Measure Management (State Auditor’s
Office Report No. 12-333, March
2012).

for fiscal year 2017: This performance
measure represents the number of youth who re-enter a secure facility
within one year as a percentage of the number of youth who were
released from a secure program to a non-secure program, parole, or
discharge.
for fiscal year 2017: This
performance measure represents the number of youth who re-enter a
secure facility within three years as a percentage of the number of youth
who were released from a secure program to a non-secure program,
parole, or discharge.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 5

 

Cost of Health Care Services Per Juvenile Day



Cost of Mental Health (Psychiatric) Services Per Juvenile Day



Average Daily Population: General Rehabilitation Treatment



Average Daily Population: Specialized Treatment

for fiscal year 2017: This
performance measure represents the total dollars expended for health
care services for each juvenile per day in the Department’s residential
programs.
for fiscal year 2017:
This performance measure represents the total dollars expended for
mental health services for each juvenile per day in the Department’s
residential programs.
for fiscal year 2017
and the first quarter of fiscal year 2018: This performance measure
represents the average number of juveniles served daily in the
Department’s general rehabilitation treatment programs.
for fiscal year 2017 and the
first quarter of fiscal year 2018: This performance measure represents
the average number of juveniles served daily in the Department’s
specialized treatment programs.

Inaccurate Performance Measures
Juvenile Per Direct Supervision Juvenile Correctional Officer Staff Per Shift

For fiscal year 2017, the Department reported inaccurate results for the
Juvenile per Direct Supervision Juvenile Correctional Officer Staff Per Shift
performance measure (see text box for inaccurate performance
Inaccurate
measure information). This performance measure represents the
A performance measure is
ratio of juveniles in secure facilities per Juvenile Correctional Officer
inaccurate when the actual
staff for each shift.
performance is not within 5 percent
of the reported performance, or
when there is a 5 percent or greater
error rate in the sample of
documentation tested. A
performance measure also is
inaccurate if the agency’s
calculation deviated from the
performance measure definition and
caused a 5 percent or greater
difference between the number
reported to ABEST and the correct
performance measure result.
Source: Guide to Performance
Measure Management (State
Auditor’s Office Report No. 12-333,
March 2012).

The Department used population control logs to track the number of
youth and juvenile correctional officers present in the Department’s
juvenile dormitories for each shift. The Department then recorded
that information in a summary document, which it used to calculate
the performance measure results. The Department results were
within 5 percent of the results calculated by auditors. However, the
performance measure results were inaccurate because the
supporting information in 6 (10 percent) of the 61 logs tested was
not correctly recorded in the summary document.

Not ensuring that the population control logs are accurately recorded
in the summary document increases the risk that the Department could
continue to report inaccurate results.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 6

 Average Daily Attendance in JJD-operated Schools

For fiscal year 2017 and the first quarter of fiscal year 2018, the Department
reported inaccurate results for the Average Daily Attendance in JJD-operated
Schools performance measure. This performance measure represents the
average number of juveniles attending school taught by Departmentemployed teachers each day.
The Department reported results that were within 5 percent of the results
calculated by auditors. However, the performance measure results were
inaccurate because auditors identified errors in the automated query that
the Department used to calculate the average daily attendance. Specifically,
for a sample of days for which the Department tracked its population for
fiscal year 2017 and the first quarter of 2018:


Nineteen (31 percent) of 61 days tested for fiscal year 2017 incorrectly
included or excluded certain students.



Thirteen (21 percent) of 61 days tested for the first quarter of fiscal year
2018 incorrectly included or excluded certain students.

Not ensuring that the query used to calculate the performance measure
results accurately captures all required attendance information increases the
risk that the Department could continue to report inaccurate results.
In addition, the Department did not terminate system access to the internetbased Texas Computer Cooperative System (iTCCS), which the Department
uses to record attendance information, for certain users who no longer
required that access. Not appropriately restricting user access increases the
risk that unauthorized changes could be made to that system.
Policies and Procedures and Reviews
The Department did not have adequate policies and procedures for the
performance measures tested. Specifically, for seven of the eight
performance measures for which the Department reported unreliable
performance results, the Department was unable to provide any policies and
procedures related to collecting, calculating or reporting those performance
measures. For the remaining measure, the policies and procedures were not
in place for the first eight months of fiscal year 2017.
The Guide to Performance Measure Management (State Auditor’s Office
Report No. 12-333, March 2012) requires state agencies to clearly document
all steps performed in the collection, calculation, review, and reporting of
performance measure data. Written policies and procedures will help the
Department increase the consistency of collection, calculation, and review of
performance measure information. Additionally, the Department did not
An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 7

 perform sufficient reviews for the eight performance measures for which it
reported unreliable results. While the Department reviewed the final data
that it submitted into ABEST, it did not review the collection and calculation
of the performance measure data.
Recommendations

In addition to the recommendations made in Chapter 1 related to youth data
in CCS used for performance measure reporting, the Department should:


Ensure that the population control logs are accurately recorded in the
summary document used to calculate the Juvenile per Direct Supervision
Juvenile Correctional Officer Staff Per Shift performance measure.



Ensure that automated queries used in the calculation of performance
measure results are accurate and reviewed.



Remove access for users who no longer require access to iTCCS.



Develop and implement policies and procedures related to the collecting,
calculating, and reporting of performance measures.



Develop a documented review process to verify the accuracy of
performance measure data and calculations.

Management’s Response

Responsible Party: As Noted for each bullet point
The Texas Juvenile Justice Department agrees with the recommendations.


TJJD will 1) implement a control process to ensure the summary document
used to calculate ratios contains accurate information by August 31,
2018; and 2) develop a formal procedure within the Institution Operations
Manual that specifies steps for reporting the youth to JCO ratio accurately
by January 1, 2019.
Person Responsible: Director of State Operated Programs and Services



Immediately after auditors brought this error to the Department’s
attention, the Research department revised its performance measure
calculation for average daily attendance for the third quarter of fiscal
year 2018 to ensure accurate results. The Department will develop a
plan to implement additional reviews of queries used in the calculation of
performance measures. This will include annual review of queries by the

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 8

 appropriate department(s) in fiscal year 2019. The reviews will be
documented and saved electronically.
Person Responsible: Director of Research and Planning; Chief of Staff;
Chief Operating Officer
Date of Implementation: December 31, 2019


The District PEIMS Coordinator reviewed the list of active iTCCS Users and
requested removal of everyone that no longer needs access on July 23,
2018 through the Region 13 Education Service Center. Subsequent review
of current active Users on July 25, 2018 confirms that the request through
Region 13 has been completed. Going forward, an iTCCS access review
will occur at the end of every grading cycle and EDU Policy 05.21 (3)(C)
will be enforced. Region 13 will be notified to inactivate users as needed.
Person Responsible: Senior Director of Education Services
Date of Implementation: July 25, 2018



The Department developed a Performance Measure Manual for State
Education Measures for fiscal years 2018-2019 in September 2017 and is
in the process of formalizing and implementing it. The manual includes
data entry, measure calculation, reporting, and controls. The Department
will expand the performance measure manual to include all measures
included in the agency’s State performance measures.
Person Responsible: Director of Research and Planning; Chief of Staff;
Chief Operating Officer
Date of Implementation: December 31, 2020



While the Department had reviews for both data and calculation in place
during the audit period, these processes should be expanded and formally
documented. The Department is planning a data integrity workgroup
starting fiscal year 2019 that will facilitate data accuracy across
appropriate divisions. In an effort to further improve data integrity, the
Department hired a CCS System Support Analyst starting July, 2, 2018.
This person has started conducting reviews, identifying potential data
issues, documenting current processes, and planning front-end user
training. The intent is to develop documentation of operational,
technical, and user support requirements for a new case management
system that will ensure data integrity. The Department will also
implement a plan for annual review of performance measure calculations.
This will include all departments involved with the measures.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 9

 Person Responsible: Director of Research and Planning; Chief of Staff;
Chief Operating Officer
Date of Implementation: December 31, 2020

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 10

 Chapter 3

The Department Reported Reliable Results for One Key Performance
Measure Tested, But It Should Strengthen Its Review Process
Certified with Qualification
A performance measure is certified with
qualification when reported performance
appears accurate but the controls over
data collection and reporting are not
adequate to ensure continued accuracy. A
performance measure is also certified
with qualification when controls are
strong but source documentation is
unavailable for testing. A performance
measure is also certified with
qualification if an agency’s calculation of
performance deviated from the
performance measure definition but
caused less than a 5 percent difference
between the number reported to ABEST
and the correct performance measure
result.
Source: Guide to Performance Measure
Management (State Auditor’s Office
Report No. 12-333, March 2012).

The Department accurately reported the results for the
Turnover Rate of Juvenile Correctional Officers performance
measure for fiscal year 2017, which represents the termination
rate of Juvenile Correctional Officers within the fiscal year. That
measure was certified with qualification because the
Department did not have policies and procedures in place for
the first eight months of fiscal year 2017. (See the text box for
certified with qualification information.)
In addition, while the Department documented the review that
it performed to verify that its summary documentation was
accurately entered into ABEST, it did not have a documented
review in place for the collection or calculation of the
performance measure data.

Recommendation

The Department should implement a documented review process for
collecting and calculating the Turnover Rate of Juvenile Correctional Officers
performance measure.
Management’s Response


The Texas Juvenile Justice Department agrees with the recommendations.
The Department developed informal procedures in FY 2017 which will be
expanded and formalized. For the period audited, the Department
routinely reviewed data collected and calculated for JCO employees and
terminations against available numbers published by the State Auditor’s
Office prior to reporting. While there is some lag in SAO reporting, the
reported numbers have been within .1% since the Department was
created. In July 2018, the Department moved to CAPPS HR/Payroll to
replace the state’s Uniform Statewide Payroll/Personnel System (USPS).
Research and Planning will be reviewing information and documentation
in CAPPS HR/Payroll. Once the new source data is available, the
Department will move forward with recreating the turnover calculation.
A review process will be formally documented and reviews will be saved
electronically.
An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 11

 Person Responsible: Director of Research and Planning
Date of Implementation: September 30, 2019

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 12

 Appendices
Appendix 1

Objectives, Scope, and Methodology
Objectives
The objectives of this audit were to determine whether the Juvenile Justice
Department (Department):


Is reporting accurate performance measure results to the Automated
Budget and Evaluation System of Texas (ABEST).



Has adequate controls over the collection, calculation, and reporting of
its performance measures.

Scope
The scope of this audit included six key performance measures that the
Department reported for fiscal year 2017 (September 1, 2016, through
August 30, 2017) and three key performance measures that the Department
reported for fiscal year 2017 and the first quarter of fiscal year 2018
(September 1, 2017, through November 30, 2017).
Auditors expanded the scope of the audit to include on-site visits to five
Department facilities and one halfway house due to the issues related to the
reliability and completeness of data within the Correctional Care System
(CCS) (see Chapter 1 for discussion of those issues).
Methodology
The audit methodology included auditing the Department’s reported
performance measure results for accuracy and adherence to performance
measure definitions and evaluating controls over the Department’s
performance measure calculation processes. Auditors also assessed the
reliability of the Department’s performance measure-related data obtained
from the Uniform Statewide Payroll/Personnel System (USPS), the Internetbased Texas Computer Cooperative Software (iTCCS), CCS, and internally
maintained Excel spreadsheets, which supported the reported performance
measure results. Auditors also tested support in the Department’s hard-copy
master files for the youth in its facilities to verify key data in CCS.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 13

 Data Reliability and Completeness

Auditors assessed the reliability of the data from CCS related to the following
performance measures:


Reincarceration Rate: Within One Year.



Reincarceration Rate: Within Three Years.



Cost of Health Care Services Per Juvenile Day.



Cost of Mental Health (Psychiatric) Services Per Juvenile Day.



Average Daily Population: General Rehabilitation Treatment.



Average Daily Population: Specialized Treatment.

Auditors also assessed the reliability of the data from:


USPS related to the Turnover Rate of Juvenile Correctional Officers
performance measure.



iTCCS related to the Average Daily Attendance in JJD-operated Schools
performance measure.



Excel spreadsheets that the Department internally maintained for the
Juvenile Per Direct Supervision Juvenile Correctional Officer Staff Per Shift
performance measure.

To do that, auditors (1) determined population completeness and
reasonableness; (2) reviewed the process to generate data related to the
calculation of the performance measures; (3) interviewed and obtained
information from Department staff; (4) reviewed source documentation for
performance measure data; and (5) evaluated information technology
general controls, including user access and change management for the
Department’s CCS system.
Auditors determined that for fiscal year 2017 and the first quarter of fiscal
year 2018, the USPS data, the iTCCS data, and data from the Excel
spreadsheets internally maintained by the Department was sufficiently
reliable for the purposes of this audit.
However, auditors determined that for fiscal year 2017 and the first quarter
of fiscal year 2018, the CCS data was not sufficiently reliable for the purposes
of this audit. As discussed in Chapter 1, certain data in that system was not
reliable as of a result of system changes the Department made to CCS.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 14

 Sampling Methodology

For the Turnover Rate of Juvenile Correctional Officers, Average Daily
Attendance in JJD-operated Schools, and Juvenile Per Direct Supervision
Juvenile Correctional Office Staff Per Shift performance measures, auditors
selected non-statistical samples through random selection. The sample
items were not necessarily representative of the population; therefore, it
would not be appropriate to project the test results to the population.
Auditors used the samples to test whether controls over the performance
measures were operating effectively to ensure that performance measure
results were accurate and to determine whether the Department was
accurately reporting its performance measures in ABEST.
Additionally, to verify key data in CCS, auditors selected a risk-based sample
of youths’ records in CCS for testing. The sample items were generally not
representative of the population and, therefore, it would not be appropriate
to project those test results to the population. Auditors used the sample to
verify the youth’s physical presence at Department facilities and to test other
key data fields in CCS.
Information collected and reviewed included the following:


Performance measure information reported in ABEST.



The Department’s summary and source documents used for calculating
the performance measure results tested.



Reports generated from the Department’s internal payroll system used to
calculate the performance measure results tested.



The Department’s hard-copy master files for the youth in its facilities.

Procedures and tests conducted included the following:


Interviewed Department staff to gain an understanding of the processes
used to calculate the performance measures tested.



Reviewed performance measure calculations for accuracy and to
determine whether the calculations were consistent with the definitions
on which the Department, the Legislative Budget Board, and the
Governor’s Office of Budget, Planning, and Policy agreed.



Tested a sample of source documents to verify the accuracy of reported
performance and the effectiveness of controls.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 15

 

Interviewed individuals responsible for data entry into CCS at
Department facilities to gain an understanding of any data accuracy and
data use issues.



Tested a sample of youth in Department facilities to verify the accuracy of
certain data in CCS.



Assessed performance data results in one of the four categories:
Certified, Certified with Qualification, Inaccurate, or Factors Prevented
Certification.

Criteria used included the following:


The Guide to Performance Measure Management (State Auditor’s Office
Report No. 12-333, August 2012).



ABEST performance measure definitions.



The Department’s policies and procedures.



Title 1, Texas Administrative Code, Chapter 202.

Project Information
Audit fieldwork was conducted from December 2017 through June 2018. We
conducted this performance audit in accordance with generally accepted
government auditing standards. Those standards require that we plan and
perform the audit to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our audit
objectives. We believe that the evidence obtained provides a reasonable
basis for our findings and conclusions based on our audit objectives.
The following members of the State Auditor’s staff performed the audit:


Jeffrey D. Criminger (Project Manager)



Joseph T. Frederick, CPA (Assistant Project Manager)



Philip Stringer, CPA (Assistant Project Manager)



Michael Bennett



James Collins



John Felchak



David Garcia Benitez

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 16

 

Arnton Gray



Justin H. Griffin, CISA



Nathan Stein



Richard Wyrick



Dana Musgrave, MBA (Quality Control Reviewer)



Audrey O’Neill, CIA, CFE, CGAP (Audit Manager)

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 17

 Appendix 2

Issue Rating Classifications and Descriptions
Auditors used professional judgement and rated the audit findings identified
in this report. Those issue ratings are summarized in the report
chapters/sub-chapters. The issue ratings were determined based on the
degree of risk or effect of the findings in relation to the audit objective(s).
In determining the ratings of audit findings, auditors considered factors such
as financial impact; potential failure to meet program/function objectives;
noncompliance with state statute(s), rules, regulations, and other
requirements or criteria; and the inadequacy of the design and/or operating
effectiveness of internal controls. In addition, evidence of potential fraud,
waste, or abuse; significant control environment issues; and little to no
corrective action for issues previously identified could increase the ratings for
audit findings. Auditors also identified and considered other factors when
appropriate.
Table 3 provides a description of the issue ratings presented in this report.
Table 3

Summary of Issue Ratings
Issue Rating

Description of Rating

Low

The audit identified strengths that support the audited entity’s ability to
administer the program(s)/function(s) audited or the issues identified do
not present significant risks or effects that would negatively affect the
audited entity’s ability to effectively administer the
program(s)/function(s) audited.

Medium

Issues identified present risks or effects that if not addressed could
moderately affect the audited entity’s ability to effectively administer
program(s)/function(s) audited. Action is needed to address the noted
concern(s) and reduce risks to a more desirable level.

High

Issues identified present risks or effects that if not addressed could
substantially affect the audited entity’s ability to effectively administer
the program(s)/function(s) audited. Prompt action is essential to address
the noted concern(s) and reduce risks to the audited entity.

Priority

Issues identified present risks or effects that if not addressed could
critically affect the audited entity’s ability to effectively administer the
program(s)/function(s) audited. Immediate action is required to address
the noted concern(s) and reduce risks to the audited entity.

An Audit Report on Performance Measures at the Juvenile Justice Department
SAO Report No. 18-042
August 2018
Page 18

 Copies of this report have been distributed to the following:

Legislative Audit Committee
The Honorable Dan Patrick, Lieutenant Governor, Joint Chair
The Honorable Joe Straus III, Speaker of the House, Joint Chair
The Honorable Jane Nelson, Senate Finance Committee
The Honorable Robert Nichols, Member, Texas Senate
The Honorable John Zerwas, House Appropriations Committee
The Honorable Dennis Bonnen, House Ways and Means Committee

Office of the Governor
The Honorable Greg Abbott, Governor

Juvenile Justice Department
Members of the Juvenile Justice Board
Mr. Scott W. Fisher, Chair
The Honorable John Brieden, III
The Honorable Carol Bush
Ms. Becky Gregory
Ms. Jane Anderson King
The Honorable David “Scott” Matthew
Ms. Mary Lou Mendoza
The Honorable Laura Parker
Dr. Rene Olvera
Mr. Riley Shaw
The Honorable Jimmy Smith
Mr. Calvin Stephens
Mr. David Reilly, Executive Director

 This document is not copyrighted. Readers may make additional copies of this report as
needed. In addition, most State Auditor’s Office reports may be downloaded from our Web
site: www.sao.texas.gov.
In compliance with the Americans with Disabilities Act, this document may also be requested
in alternative formats. To do so, contact our report request line at (512) 936-9500 (Voice),
(512) 936-9400 (FAX), 1-800-RELAY-TX (TDD), or visit the Robert E. Johnson Building, 1501
North Congress Avenue, Suite 4.224, Austin, Texas 78701.
The State Auditor’s Office is an equal opportunity employer and does not discriminate on the
basis of race, color, religion, sex, national origin, age, or disability in employment or in the
provision of services, programs, or activities.
To report waste, fraud, or abuse in state government call the SAO Hotline: 1-800-TX-AUDIT.