Employment Security Department WASHINGTON STATE Unemployment Tax and Benefits Project QUALITY ASSURANCE CLOSEOUT REPORT MARCH 21, 2017 SIGHTLINE March 21, 2017 Dale Peinecke Commissioner Employment Security Department PO Box 9046 Olympia WA 98504-9046 RE: Unemployment Tax and Benefits Project Quality Assurance Closeout Report – March 2017 Dear Mr. Peinecke, Thank you for the opportunity to provide Quality Assurance (QA) services to the Employment Security Department (ESD) Unemployment Tax and Benefits (UTAB) system project. I am attaching a final closeout report summarizing the project and providing lessons learned that can be applied to future projects at ESD. The UTAB team should be commended for their work in implementing UTAB, as they made substantial progress in meeting the intended business needs. The team is working to transition the system to ongoing maintenance and operations (M&O) beginning in April 2017. The success of the project includes several positive outcomes. For example:  The system is generally working as designed.  The project implemented a new system within two years, and $11 Million under budget.  During the initial troubled cutover phase, the vendor and state team worked in unison to quickly resolve outstanding issues. The following lessons were gathered through interviews, documentation review, and observation. 1. The project management structure was atypical, but generally provided adequate support to the project team. 2. Greater engagement of operations by the project team in decision-making may have provided ESD additional insight and increased buy-in of executive decisions. 3. The project schedule was aggressive and more resources may have improved testing and organizational change management activities. Project Management Tools 4. ESD’s supplemental supporting project documentation was essential for tracking progress and understanding deployment readiness. 5. ESD created necessary planning documents prior to the start of the vendor engagement that provided a strong foundation for project success. 855 TROSPER RD. SW, #108-147, TUMWATER, WA 98512 360.264.7715 SIGHTLINELLC.COM Dale Peinecke March 31, 2017 Page 2 6. Development and implementation of a defect prioritization process prior to go-live would have improved clarity by providing a structure for ensuring the most critical defects were repaired first. 7. Use of industry standard tools and ESD installed toolsets could have improved project processes and better supported ESD in the maintenance and operations phase. Team Structure and Organizational Considerations 8. Including agency staff in all project phases provides a more stable foundation for the maintenance and operations phase. 9. Involving dedicated, full-time ITBI resources in all project phases (procurement to closeout) is important to ensure a successful integration of tools and a smooth transition to M&O. Organizational Change Management, Staff Training, and Operational Readiness 10. Implementing a strong organizational change management strategy that spans the length of the project provides a much greater probability of user acceptance and decreases user anxiety. 11. Development of a maintenance and operations transition strategy prior to cutover provides for a smooth transition to long-term support. 12. Conducting user training on a version of the system that closely reflects the system that will be in place at production increases usability at go-live. 13. Desk-side support at cutover provided operation staff access to experienced system users, which provided critical support in completing system tasks. Technical Management Processes 14. The UTAB system is a COTS product and is not compatible with ESD’s installed infrastructure. This may increase risks to the long-term maintenance activities and will likely require increased support from the vendor. 15. The state’s enterprise security infrastructure is complex and will likely require ongoing technical and operations support. 16. WaTech was not prepared to support ESD’s SAW/SEAP implementation. 17. The level of difficulty associated with interface coordination and management was underestimated and started too late in the implementation process. 18. It is important that cutover activities be documented and communicated to responsible parties to ensure a smooth cutover process. 19. Mock conversions provided insight into data issues so that issues could be addressed prior to cutover, and confirmed conversion timelines to ensure data migration could be run within a specific timeline. 20. On projects using an iterative methodology, testing needs to start early and continue until golive to ensure that the system is fully and well tested. 21. Contracting with an expert usability firm to conduct usability and accessibility testing helped to uncover usability issues with the external-facing application. Page 2 Dale Peinecke March 31, 2017 Page 3 22. Performance and load testing was inadequate, which resulted in significant performance issues at cutover. The final closeout report summarizes the UTAB project and provides details regarding lessons learned. More specifically this report contains three sections: I. General Background – an overview of the UTAB project. II. Project’s Lifecycle Summary – a summary of the various phases completed as part of the UTAB project. III. Lessons Learned – A list of lessons learned as part of the UTAB project. Thank you again for the opportunity to support the UTAB project. Regards, Kathleen Nolte, Principal Sightline, LLC Page 3 UTAB Quality Assurance Closeout Report I. GENERAL BACKGROUND ESD pays unemployment insurance (UI) benefits to unemployed workers in Washington State. A collection of computer systems and business processes receive claims, process claims and pay benefits according to defined benefit rates. Prior to the implementation of UTAB, when a claim was received, it was processed and managed through the department’s UI benefit information system, known as the General Unemployment Insurance Development Effort system (GUIDE). GUIDE’s base architecture and code were originally developed in the 1980s for the state of Wyoming. In 1996, the system was transferred to Washington and modified to meet its unique business needs. This mainframe system worked well for ongoing processing. However ensuring compliance with current statutes, responding to changes in law or policy, and meeting customer needs for more modern computing had become very difficult in this aging system. Further, finding the skilled staff to support the outdated programming language was also becoming increasingly difficult since many of the staff trained in older programming languages are reaching retirement age. The GUIDE system was a workhorse. In calendar year 2016, approximately $1.02 billion was paid to more than 210,550 unemployment benefit claimants. Failure to receive a weekly check, or even receiving it one or two days late, creates real hardship for claimants. Functionality and failure in ESD's benefits systems are highly visible to the public. Replacement of the state’s UI systems has been a work in progress for several years. ESD completed implementation of its Next Generation Tax System (NGTS) Project, which replaced the previous UI tax payment system with a modern technology platform. The Unemployment Tax and Benefit (UTAB) system, developed to replace the benefits functionality contained in GUIDE, includes the following functions.  Eligibility and payment functions that calculate benefit amounts and duration for all UI programs and entitlements;  An extension of the subsidiary ledger accounting service that has been developed for the NGTS system;  Benefit charging functions that support the payment structure as well as the employer tax structure;  An initial claims module that guides applicants through the process of applying for benefits on the Internet and is used by staff in UI benefits centers;  On-line weekly benefits certification that guides claimants through the process of certifying continuing eligibility for UI benefits;  Issue resolution functions to help UI staff determine eligibility when an application or benefits certification has issues;  Collections services that streamline the billing process and leverage functions developed for NGTS for instituting legal actions for collections;  Additional tools (especially data mining) will help staff detect fraud;  An electronic mailing function that supports communication between both claimants and employers; and Page 1 UTAB Quality Assurance Closeout Report  A reports engine that streamlines the development and maintenance of reports and makes reports readily available to staff. II. PROJECT PHASE SUMMARY Planning and Initiation In April 2015, ESD entered into a contract with Fast Enterprises, LLC (FAST) to provide a commercialoff-the-shelf (COTS) product to manage the benefits-related functionality currently performed by GUIDE and its ancillary systems. Initial work focused on establishing project management systems and setting up the technical environments. A decision was made early in the project to use FAST’s Workbench tool to the extent possible to track project tasks and progress. Although Workbench may function well for FAST to document and track development, it is still maturing as a project management tool and the project team was required to develop additional supporting tools to provide sufficiently robust project tracking and management. Other foundational plans were also put into place to guide project work, such as the communications plan, risk and issues tracker, and the project management plan. These were well thought out plans that provided a useful framework for project activities. Initial work was bifurcated between configuration of the Claimant and Wage portal, and development of the benefits system. Project subject matter experts (SMEs) worked with FAST developers to define the scope of the system through a series of definition sessions that provided both FAST and ESD a base understanding of project processes. Execution Claimant and Wage Portal ESD implemented a Claimant and Wage portal to introduce ESD to the FAST approach and methodology, and to provide a quick win for the department. The Claimant Wage phase included deployment of an external web portal that provides UI recipients with general information and status about their claims. The portal was developed using the FAST methodology and was in many ways successful in introducing a sub-set of ESD employees to the FAST approach. The portal was configured and readied for implementation but the portal deployment was delayed by about four months as the ESD team completed new user access security requirements. ESD worked with Washington Technology Solutions (WaTech) to implement appropriate security measures through the state’s standard security tools, the SecureAccess Washington (SAW) / SAW Enabled Agency Portal (SEAP). ESD’s original plan was to migrate to the new enterprise SAW/SEAP solution when the full UTAB system rolled out, however shortly before go-live, ESD discovered it would be required to implement SAW/SEAP with the Claimant and Wage Portal. As a result, the deployment of the portal was put on hold until the agency’s SEAP solution was implemented. The portal was ultimately placed into production in October 2015. Page 2 UTAB Quality Assurance Closeout Report Unemployment Insurance Tax and Benefits (UTAB) Once definition was complete, the team moved into configuration of the system and development of site-specific code. Development of the UTAB system occurred over several months. ESD staff were actively engaged in the process and generally were comfortable making decisions at the team level. One of the hallmarks of the FAST approach is for team members to make incremental decisions so the system can be configured as quickly as possible. This early configuration is then used to show how the system functions in a given scenario. Because it is done early and often, the decisions can be revisited and changed if the system is not meeting the business need. Eventually, through incremental configuration sessions, the system is readied for the test and implementation phases. User training was developed and delivered by the project team. ESD staff participated in a series of training sessions presented using a three-tiered training approach and using a standardized grouping method structured to teach people who perform similar jobs together in the same classes: Tier 1 – Computer-based training provided a basic overview of system functionality; Tier 2 - Instructor led courses with practice exercises designed to address the basic functionality of UTAB; and Tier 3 Instructor led courses with practice exercise based on business related scenarios designed to provide a more structured look and concentration on specific tasks. While Tier 1 and Tier 2 training went well and was well received by staff, Tier 3 training did not provide staff a sufficiently comprehensive endto-end view of how business processes would be completed in the system. Additional labs were set up to help staff work through their questions. Deployment Deployment of the UTAB system was initially planned for October 2016. In September, ESD determined that development, testing, conversion, and interface tasks were not sufficiently complete and a revised deployment plan was developed. FAST and ESD defined a release strategy that allowed for core functions to be released in November (Release 2.0) and the remaining to be released in the Spring 2017 (Release 2.1). As the November deployment date neared, the team determined that several critical pieces of functionality were not completely tested, and additional time was needed to ensure the system, business processes and staff were ready. During November, the team worked to identify all remaining work and then to establish a new target date for the R2.0 deployment. Implementation was rescheduled for January 3, 2017. In the first few days following launch, UTAB system performance was problematic and peripheral systems did not manage the volume of users attempting to access the system in a timely manner. This resulted in a partial system shutdown during the first two days of operations (eServices was temporarily disabled to allow the team to fix system issues and manage claimants who were able to gain access to services). While UTAB has stabilized significantly, other peripheral systems continued to have issues, which has affected ESD’s ability to manage the claim and call volume.  Users are still experiencing a low response success rate using the state’s single sign-on security infrastructure, resulting in an increased number of phone calls to the agency.  The Interactive Voice Response (IVR) system experienced performance issues because the number of callers far exceeded expected numbers. It is unlilkely that any phone system could Page 3 UTAB Quality Assurance Closeout Report have managed the call volume on the first couple of days (ESD received nearly 750,000 calls on the first day of operations.)  Because the UTAB system is new, it will take longer for operations staff to process claims so phone wait times unrelated to the single sign-on process are likely to stay high for the near future. Once the technical performance issues were addressed, the system has been generally working as expected. During the first 10 weeks in production, UTAB created more than 110,000 new eServices accounts, more than 543,675 people filed an initial application for benefits, approximately 783,000 weekly claims requests were processed and more than $283 million in payments were made. As is typical in any large system implementation, users have identified defects and enhancements that need to be assessed, prioritized and fixed. ESD implemented a defect review process that is continuing to mature. Release 2.1 Release 2.1 included several tasks that were not critical for the initial release and some additional reports. The team planned to implement the remaining functionality for this phase prior to February 28. Release 2.1 was completed in March. CLOSEOUT ESD is in the process of moving the project into maintenance and operations, with day-to-day warranty tasks being managed by operations and information technology staff. A cross-agency product team has been established and will be housed in the UI Benefits division. ESD plans for the official maintenance and operations phase to begin on April 1. FAST continues to maintain an on-site development team. This team is responsible for completing fixes for prioritized defects and enhancements. After the UTAB system was put into production, the Director of the UI Benefits division identified functionality previously provided by GUIDE that has not yet been developed in UTAB, or that was developed differently than expected. FAST has agreed to complete these tasks during the warranty period. Page 4 UTAB Quality Assurance Closeout Report III. LESSONS LEARNED This report highlights some of the major lessons learned for the UTAB project. The following lessons were gleaned from QA observations and interviews with project team members and stakeholders. Project Management Processes 1. The project management structure was atypical, but generally provided adequate support to the project team. ESD named a business sponsor from the policy division and a project director to manage project controls. The business sponsor performed several duties that would normally be the responsibility of the project director (e.g., vendor management, decision-making processes, etc.). The project director maintained the project controls including the traceability matrix, project schedule and managed the technical aspects of the FAST contract. FAST provided a project manager and technical leads responsible for managing the FAST team and monitoring FAST progress. FAST leads brought strong knowledge of the FAST product and systems needed. While the structure resulted in a successful system implementation there were some lessons identified by staff.  The business sponsor and policy liaison were skilled decision makers, which was a critical success factor for the project. Escalated decisions were generally timely, although the availability of the sponsors was limited.  Project staff felt supported and able to make project decisions related to configuration.  In hindsight, project staff believe that the number of issues identified at go-live could have been decreased had operations managers been more involved in the day-to-day configuration and development discussions to address broader operations policy impacts. 2. Greater engagement of operations by the project team in decision-making may have provided ESD additional insight and increased buy-in of executive decisions. Project leadership established an Executive Steering Committee in the beginning of the project. While the steering committee meetings provided a status reporting venue, they were not used to discuss policy issues or make executive level decisions. Instead, decisions requiring executive input were made in closed executive sessions rather than at steering committee meetings. This approach resulted in a lack of transparency to overall decisionmaking and did not allow for a full discussion of issues or include oversight entities or vendor staff in the discussions. Typical steering committee structures allow for parties to openly discuss concerns or issues, thus leading to greater buy-in and openness of project decisions. The number of issues raised to the ESC level were limited as project management encouraged and empowered project team members to make decisions at the lowest level of the organizational structure as possible. Fortunately, the business sponsor and the business manager were very supportive and made timely decisions to keep the project on track. FAST’s methodology supports the notion that decisions should be made quickly because the system is flexible enough that functionality can be modified if needed. In the majority of cases, the decisions were thoughtful, timely and correct. In some instances, decisions needed to be revisited which caused rework and frustration at the project team level. Several issues impacted the decision making process: Page 5 UTAB Quality Assurance Closeout Report  The business sponsor and the business manager were available as much as possible. The number of needed decisions was large, which made it difficult for them to be involved in every decision. In some cases, a suboptimal decision was made because they were not readily available to the team.  Project team members did not always feel comfortable making decisions on policy related matters, yet felt pressure from the vendor to make them expeditiously.  Other than the small number of decisions made at the steering committee level, project level decisions and finalized requirements were not documented for future reference. Without documentation, there was no record to support either the vendor or project staff when there was a disagreement. This eroded some trust among the team members. The lack of documentation also limited the amount of information that could be passed to operational staff for reference in maintenance and operations (M&O). This situation improved once the team began testing as the decisions were fully documented and used to explain how and why a particular function was developed.  The functional teams were silo’d. This did not lend itself to making sound decisions that crossed functional areas. In some cases, one team’s decisions did not meet the needs of another team. This also caused rework and frustration. 3. The project schedule was aggressive and more resources may have improved testing and organizational change management activities. The original schedule was proposed by FAST during the procurement process. The proposed 18-month schedule was seen by ESD and FAST as aggressive, but doable. A schedule was prepared and the team began to drive towards a September 2016 go-live date. When it became apparent that interim tasks were slipping, ESD began implementing mitigation strategies that would allow them to meet the original schedule. Although adjustments to team size and work assignments were made, it became apparent in the summer of 2016 that testing and development dates could not be met. The team completed an assessment and determined that an additional two months would be needed. The schedule was realigned with a new November go-live date. The team also determined that several tasks could be completed later and they were removed and placed into a second release (Release 2.1 [R2.1]) that was scheduled for completion in February 2017. As the November date approached, project management became concerned that additional training and end-to-end testing were needed to ensure a smooth deployment. Unfortunately, this put the project at a disadvantage as it left few dates available for deployment. Data migration and cutover activities needed to occur over a three-day period leaving only a few weekends available. The project chose January 3 because it provided sufficient time to complete the cutover list, and it simplified reporting processes as it occurred at the end of a yearly, quarterly, and monthly reporting cycle, thus some work to combine data from the old and the new system was eliminated. In hindsight, implementation on January 3 proved to be problematic. The number of unemployed individuals is at its peak for the year, which significantly increased calls to ESD. Page 6 UTAB Quality Assurance Closeout Report Because the system was functioning poorly for the first few days, ESD encountered public scrutiny that might have been avoided with a later release. More specifically,  Additional system testing including testing of the NGTS interfaces may have uncovered issues with out-of-sync converted data.  Staff could have benefited from additional training and time for developing end-to-end business process flows that would have helped them navigate the system better at cutover. Project Management Tools 4. ESD’s supplemental supporting project documentation was essential for tracking progress and understanding deployment readiness. ESD chose a system and vendor whose iterative methodology relies on little documentation. FAST brings a proprietary development and project management tracking tool, called Workbench, that provides some insight into all of the development tasks necessary to complete the project. However, Workbench does not provide robust progress monitoring and communication capabilities. For example, the tool contains a list of tasks and assignments but does not provide the level-of-effort, dependency tracking, or traceability. Additionally, it was often not updated to reflect true status, so it was difficult for ESD to determine whether development and configuration were tracking to the schedule. To mitigate this risk, the ESD project team developed several tracking and monitoring tools outside of Workbench to provide more precise monitoring. For example, the project developed:  Requirements traceability tools that linked the tasks in Workbench with the business requirements contained in the bidder’s response to the proposal.  Test tracking tools that aligned test scenarios with each of the requirements and development tasks.  An integrated schedule that included FAST tasks and ESD tasks that resided outside of the vendor’s responsibility. All of these tools were necessary in order to track progress and to ensure ESD’s contractual obligations were being met. 5. ESD created necessary planning documents prior to the start of the vendor engagement that provided a strong foundation for project success. ESD spent significant time preparing for the procurement process by defining requirements and business rules. This work was necessary to be able to meet aggressive timelines. Although the vendor did not use the list of requirements in developing their task list, the UTAB project management ensured that the requirements were met. The team also learned that completion of some technical tasks (development of the inventories of correspondence [letters and notices], interfaces and data sources) during the initiation phase would have saved significant staff time during the implementation phase. The vendor needed to understand how big this work effort was, and the delayed development of the inventories caused some delay. Page 7 UTAB Quality Assurance Closeout Report 6. Development and implementation of a defect prioritization process prior to go-live would have improved clarity by providing a structure for ensuring the most critical defects were repaired first. ESD developed a transition plan that included a defect prioritization process prior to cutover. However, at cutover, the process was not implemented because the teams were overwhelmed with critical bugs that needed to be fixed immediately. For the first week, operations personnel recorded defects that were not readily prioritized. In essence, the “squeaky wheel” defects were fixed first, which may or may not have been the most critical or impactful. Several changes to the prioritization process would likely have improved the process during the first month of implementation.  Operations staff were confused by the process. They were asked to document an Incident Report (IR) in the current ESD defect tracking system. A UTAB team member would then review the IR and, if appropriate, it would be logged as an SQR (FAST’s label for service request) in FAST’s proprietary defect tracking tool. Communications to the operations staff were not always timely so operations staff did not know if the defect was logged as an SQR or whether it was considered to be something not system-related (training issues, for example). While this process was put in place to reduce duplicate SQRs, increased and formal communications would have increased confidence at the operational level that issues were being tracked and fixed.  The team was overwhelmed in the first few days with the number of issues they encountered. Appropriately, the team focused on making sure the system was operable. This meant that many defects were not reviewed timely. The backlog of issues remained high and at the end of stabilization, some defects had not been formally reviewed (although UTAB team members completed a cursory review to weed out any issues they saw as critical). This increased risks that a defect had been incorrectly prioritized and in fact should have been fixed sooner than later.  The process to assign tasks to developers was not fully mature. The team assigned the most critical defects (referred to as the top ten list) to developers, but there were times when developers ran out of work and were not authorized to work on tasks outside the top ten list. This was particularly true for state developers who could have supported lower level defects that were not approved for repair. The process was improved at the end of stabilization when the state operations team began to manage the defect list. 7. Use of industry standard tools and ESD installed toolsets could have improved project processes and better supported ESD in the maintenance and operations phase. The Workbench tool does not provide the functionality that is needed during the maintenance and operations phase. Unfortunately, project documentation related to bugs, enhancements and changes are only documented in the Workbench tool. ESD has an installed set of tools that it uses to manage system maintenance. The use of a proprietary tool requires that ESD staff develop and maintain two different toolsets. Additionally, retrieving data from Workbench is not intuitive and may require vendor support as long as the system is in use. During the project phase, the team developed the master work plan in MS Project. Additionally, the team used SharePoint as the document repository. These tools will be easier Page 8 UTAB Quality Assurance Closeout Report to maintain as needed as they are standard to ESD’s infrastructure and can be supported longterm. Team Structure and Organizational Considerations 8. Including agency staff in all project phases provides a more stable foundation for the maintenance and operations phase. ESD established an integrated team of technology, operations, business, and project management staff. FAST also provided developers and project management staff who were co-located with the ESD project team. Generally, staff reported that this integrated team model was very beneficial. It allowed for the development of close working relationships throughout the project and improved the team’s ability to prioritize features and communicate challenges. Subject matter experts (SMEs) from the business area were integrally involved in all project processes. The UI organization selected experienced employees for the project team so that the system would benefit from deep program knowledge. While this was a good strategy, it did not completely meet the needs of the project. Several lessons were learned as the project progressed.  While the operations staff sent to the project were skilled in their particular area, the skill set did not necessarily transfer to a project skill set. Some staff were good at managing claims, but did not understand system testing or analysis. This is a different skill set and is not always teachable. The project did not readily realize that some staff were struggling to make decisions and complete their project work. Reassessing staff skill sets periodically and making necessary staff changes may have improved velocity in completing project tasks. This would have allowed the team to switch out project staff or support staff in gaining the new skill sets.  It was difficult to determine the appropriate team size. Some people believed the number of SMEs was too high, while others thought it was too low. ESD was trying to ensure that the right people needed to be in the room to make decisions. One way to measure whether the team size was too big is to assess whether decisions were being made timely. Some functional teams were taking inordinate amounts of time to make decisions, partly because the team was large and they were relying on consensus-based decision making. Smaller teams did not have as difficult a time making decisions. It is likely that the number of SMEs on this project was high, which increased schedule risks as decisions lingered or were changed frequently when the teams did not have full consensus.  The initial team included an operations management representative on the project team, but she was redirected to manage the training team early in the project. This caused a gap in operations knowledge at the team level. A second operations manager was added to the team on a part-time basis several months into the project, providing a needed broad operations management perspective. This was extremely helpful to guide system development, but it was too late and needed to be a full-time assignment.  Some staff were matrixed to the project, but still maintained some responsibilities from their positions outside of the project team. Although some staff were matrixed because they were not needed on a full-time basis, other staff could have been used more fully if Page 9 UTAB Quality Assurance Closeout Report they were available. As a result, some project work was impacted because appropriate skill sets were not readily available and at times this matrixed approach caused staff to choose between project work and their “normal” work. It is best to assign staff to the project on a full-time basis, rather than rely on part-time resources.  While much of the project focus is on configuring the UTAB system, other issues need to be considered. It was important to document and assign responsibilities for policy level initiatives (ADA compliance, security compliance, etc.) so that the system when configured would meet Washington’s technology policies.  The project included a dedicated communications expert on the project team who was responsible for internal and external communications. The communications expert was colocated with the team, which allowed her the opportunity to gain valuable project knowledge in crafting targeted communications. Issues identified at cutover required that an additional communications expert be assigned to the project to manage the external communications with the press and Governor’s office. The communications staff worked together closely to keep the messages on target and consistent. Additional communications during development between operations and the project could have increased user buy-in and surfaced issues that were encountered after go-live. One critical challenge for the communications staff was that the go-live date remained in flux so it was difficult to get information to the right people at the right time. Because the system remained in development (e.g., screens were changing, directions were changing, etc.) it was difficult to keep instructions up to date and timely.  FAST brought experienced staff as technical leads. They knew the system well and were able to resolve issues quickly. Some of the team leads were new to FAST and did not understand the system as well. Because they lacked in-depth knowledge of the system and processes, they were not able to quickly answer ESD staff questions. This led to slower than planned progress during configuration. It would have been helpful to intervene sooner with additional support when a work group was falling behind. 9. Involving dedicated, full-time ITBI resources in all project phases (procurement to closeout) is important to ensure a successful integration of tools and a smooth transition to M&O. The UTAB team included a technical manager, an interface manager, and four developers from the Information Technology and Business Integration (ITBI) division. ITBI expected that these resources would gain enough experience to supplement M&O activities post-project. While knowledge has increased significantly, some of the developers did not gain enough knowledge and experience to support the system without FAST support. Several lessons related to knowledge transfer were learned.  FAST is contractually obligated to deliver a system to ESD. They look for the most efficient way to complete their tasks. It is to their benefit to bring experienced staff to the project. In the absence of a specific contractual obligation to meet knowledge transfer metrics, they do not prioritize training ESD staff to perform at a specific level. Some developers were more successful in navigating the training than others. ESD will continue to train remaining developers and has the opportunity to hire more staff to support the product. In general, knowledge transfer approach was not successful in ensuring ESD can manage the system in the future. Ongoing knowledge transfer is planned. Page 10 UTAB Quality Assurance Closeout Report  ITBI originally planned to use a “dynamic” staffing pattern that provided staff resources to the project team as they were needed, rather than assigning full-time staff. This strategy was chosen to take advantage of the available resources and skill sets in ITBI. There was considerable staff attrition in ITBI during the project timeframe, and the remaining ITBI staff were needed to manage the current systems. It became clear that the dynamic staffing strategy was not adequate for the project because technical staff found it difficult to maintain their assigned workload on top of project tasks. ITBI increased the number of fully committed staff to the project once it was clear that the strategy was not workable. Organizational Change Management, Staff Training, and Operational Readiness 10. Implementing a strong organizational change management strategy that spans the length of the project provides a much greater probability of user acceptance and decreases user anxiety. ESD included organizational change management (OCM) as part of the FAST implementation contract. Although FAST has provided this service in other projects, it is not one of their core offerings and their definition of OCM is limited to staff readiness. FAST hired a contracted OCM resource who was skilled and experienced. She created an OCM plan that included staff readiness activities to prepare staff to accept change. She met with workers throughout the state and laid a solid foundation. The plan included additional assessments and work to create an atmosphere for change that relied on ESD to implement. While this initial work was helpful, other OCM activities did not extend to the end of the project. It would have been helpful if additional OCM activities were more robust and constant throughout the implementation and stabilization phase. For example:  Workflows and work queues were defined late in the process and were not appropriately socialized with operations staff. Staff did not have a solid foundation of the end-to-end business process and how work would flow or route within the system. This work is just now starting which has contributed to staff frustration in the operations centers.  Training did not include any documentation of workaround processes. In fact there was no indication that there were any workarounds until the system was live. Most of the necessary workarounds were needed because of the data conversion issues, as some data did not convert accurately and needed to be manually processed. These workarounds were developed after cutover so it left little time to train and prepare staff, thus increasing the amount of support the team needed to provide to staff after cut-over. Page 11 UTAB Quality Assurance Closeout Report 11. Development of a maintenance and operations transition strategy prior to cutover provides for a smooth transition to long-term support. The stabilization period relied on the same defect management and prioritization process used during implementation. Resources responsible for decision making were not the same people who will manage the system long-term. The transition from stabilization to warranty period was approximately 90 days. A transition approach was confirmed near the end of the stabilization period, leaving very little time for staff to absorb new responsibilities and understand their roles. This also caused significant angst because staff did not know if they would be transitioned to other positions. Developing a transition plan before cut-over would have allowed staff responsible for M&O activities to use the cutover period to learn the new processes and roles. Fortunately, the staff assigned to M&O have UTAB project experience, which will mitigate many of the risks. 12. Conducting training on a version of the system that closely reflects the system that will be in place at production increases usability at go-live. In early December, concerns regarding claims operations staff readiness were raised. Some staff reportedly did not feel confident in their ability to work in the new system competently. The team discovered several issues that were increasing staff concerns.  Tier 3 training (detailed functionality training) was completed in November 2016. The classes were broken into topics such as intake, initial claims, ongoing claims, and so on. Each session trained to a specific topic. While staff reported that the quality of the topical sessions was good, they did not feel that it provided an overall picture of the end-to-end business processes. Rather, they understood functions in a silo.  The training environment was based on a “snapshot in time” and was considerably outdated by the time training occurred. In addition, the training environment did not contain converted data. It was difficult for staff to understand how “real” data would be used in a production-level system. Ensuring that staff are trained on a near-production quality data snapshot would have improved training quality and consistency.  Throughout training, the system was still being configured. As changes were made to the system, training materials were updated to match the current system configuration. Training materials continued to evolve and those receiving training later in the training cycle may have been trained differently that those earlier in the cycle. To mitigate these risks the team implemented several strategies.  To help staff become familiar with the system, training labs were available in November for staff to practice on. The labs were initially unstructured and the training team found that staff did not use them. To increase staff participation in the labs and to increase user confidence, the team extended the labs through December. The training team developed structured end-to-end scenarios to orient staff to the system features. Because staff were assigned to specific lab sessions, participation increased and staff gained valuable insight into the system processes.  The training environment was updated and included converted data and a nearly exact copy of production data (a few changes occurred as a result of defect fixes.) Page 12 UTAB Quality Assurance Closeout Report  The training team developed help content that provides detailed instructions, and tips and hints. If others are needed, they can be completed quickly by ESD staff. Training materials have been updated and put onto an internal drive. The training materials did not include detailed process flows and desk manuals. ESD is just now developing the more detailed desk manual procedure and business processes. FAST assumed that operations staff would use the same business processes in the new system, which proved to be an inaccurate assumption. The lack of comprehensive training (including both system and business process training) will remain an issue until the new materials are available. 13. Desk-side support at cutover provided operation staff access to experienced system users, which provided critical support in completing system tasks. FAST and ESD project staff provided desk side support to users during the first two weeks of implementation. This was used to supplement the training materials and to provide hands-on training when questions arose. There were many benefits to the support, but also some unanticipated issues. The project team had a deep understanding of the system, but not the business processes that linked the major functions together. There were a higher number of operations staff from Spokane involved in the project so there were more staff available for desk side support in Spokane. There were challenges in getting trained staff to be available in the Lacey operations center. Additionally, staff report that they would have liked the support to continue longer than two weeks as they were not well prepared to navigate the system on their own. Technical Management Processes 14. The UTAB system is a COTS product and is not compatible with ESD’s installed infrastructure. This may increase risks to the long-term maintenance activities and will likely require increased support from the vendor. The UTAB system is built using older technology that ESD does not currently support. The system was chosen through the RFP process, because it met the business needs. Although the technical needs were discussed during the procurement process, the need for business software was a higher priority. This is disconcerting to ITBI managers, as it requires a skill set they do not have readily available or need for other systems. They believe UTAB will be harder to maintain in the long-term and will rely upon the vendor for ongoing support. Additionally, there is an inconsistent understanding of the functions that are referred to as “core” which, because this is a COTS system and core functionality is largely proprietary, typically cannot be easily modified by the agency. Some staff believe that some core functions are not actually core and that they can be changed if needed. 15. The state’s enterprise security infrastructure is complex and will likely require ongoing technical and operations support. As required by policy, ESD implemented the state’s shared security infrastructure, known as Secure Access Washington (SAW) / SAW Enabled Agency Portal (SEAP) in October 2015 with the Claimant and Wage Portal. Customers with existing claims could set up their credentials in advance of UTAB’s release in January 2017. This process was available to existing claimants, but it was not used extensively, and it was not available for newly unemployed individuals. Page 13 UTAB Quality Assurance Closeout Report Therefore, the load placed on the sign-on process at cutover was far higher than ESD had anticipated. Moreover, although the SAW/SEAP processes reside outside the UTAB system, they impeded users’ ability to get to the UTAB system and process their claims. ESD and Washington Technology Solutions (WaTech) have made improvements to the SAW/SEAP processes, but some issues remain.  ESD implemented adaptive authentication (AA) which uses a service from LexisNexis. The AA services are triggered during a claimant’s first sign-on attempt. The LexisNexis service asks questions of claimants that will provide greater surety that the claimant is the person they claim to be. While this is a standard practice for many sectors such as banking, it is relatively new to government services, and ESD is currently one of a small number of Washington state agencies using AA extensively. The AA process requires two steps. First, LexisNexis needs to identify the claimant using some basic identification information so that it can provide follow-up questions to authenticate the claimant. The second step requires the claimant to accurately answer the questions that are posed to them. Both steps have been challenging in the ESD implementation. Many claimants cannot be identified by LexisNexis, which stops the automated process. Other claimants have had a difficult time answering questions correctly. These claimants must call ESD for manual verification, which increases call volumes. Ultimately, LexisNexis reports that 90% of the claimants are getting through the verification process. While 90% is reportedly a good success rate, there is no data available to determine how many of these individuals required manual intervention to pass the AA stage.  Once the claimant has passed the AA process, they must be authenticated through the Multi-factor Authentication (MFA) process, which is used for ongoing authentication in the SAW/SEAP portal. Claimants choose a password and username and are asked to enter a code that will be sent to them through email. Two remaining issues are still impacting customer service and system access. - The adapter used in SEAP allows the user to indicate that the device being used is a private device and that it will be used on an on-going basis. The advantage of this feature is that as long as the same device is used, the claimant should not be “challenged” or asked to confirm an email code when they log in. If a user is logging in from a new device or public computer (e.g., at a public library) the device should not be remembered, and the user should be challenged. However, the adapter has a known defect that does not always “remember” the device, which leads to some users being challenged every time. While this is not a security issue, it is a usability issue, as users may not have full access to their email at the time they are signing into the system. - Additionally, the current implementation of MFA does not include a cross-agency unlock feature. If a user is locked out of any agency system, they will have to contact each agency where they have system access through SAW to be unlocked in order to gain access. This has increased phone calls to ESD (and other agencies). WaTech is aware of the issue and is working on a potential solution. In the meantime, this will remain an ongoing workload for ESD. The issues encountered with SAW/SEAP and AA increased phone calls to WaTech and ESD during the cutover period. ESD estimated the number of calls it expected based upon previous Page 14 UTAB Quality Assurance Closeout Report call volumes, and added 20 additional employees to manage the expected call volume. However, on the first day of cutover, the phone volume was significantly higher than expected and the phone system could not keep up with the volume, which increased customer and staff frustration and resulted in negative press coverage. After two months in production, ESD continues to see increased phone calls for login support. It is clear that the SAW/SEAP functionality is not self-service functionality for the UI customer base. 16. WaTech was not prepared to support ESD’s SAW/SEAP implementation. WaTech’s support team is very small and supports the entire state SAW/SEAP infrastructure. The team does not have the capacity and in-depth knowledge needed to support SAW/SEAP. Technical questions were relayed to WaTech’s system vendors, but answers to the questions were slow to materialize. Call volume data (handle times, answer rates, failure rates, etc.) were not directly available from the systems and it took months to get data for ESD to analyze and recommend changes to the installed security infrastructure. For example, ESD noticed a particularly high failure rate for the AA services. Lexis/Nexis needed to provide data to ESD that showed which questions were confusing customers. This data was delivered several weeks after the request. ESD changed some questions that were being missed by ESD’s customers and the AA rate improved slightly. Access to data and technical support may have reduced ESD’s failure rate sooner which, in turn, would decrease the number of phone calls to the agency. 17. The level of difficulty associated with interface coordination and management was underestimated and started too late in the implementation process. FAST’s methodology focuses on completing the configuration and development tasks before implementing interfaces, reports, and correspondence. While this strategy is not atypical of other system development efforts, some of the interfaces were complex and should have been started earlier. The project chose to implement an interface strategy that assumed the interfaces would not change. This “like for like” approach did not take into account that the interfaces were going from mainframe-based connections to server –based connections. Because of this, the effort to change the interfaces was more complicated than anticipated. ESD hired an additional staff person who was responsible for the coordination work. This proved to be very valuable and eventually resulted in a relatively smooth interface process. The interface coordinator implemented a revised interface strategy that was tailored to the type of interface partner.  State agency partners – ESD developed a list of contact information for each agency partner. They started with general communications with the chief information officer. The team then contacted the operational contacts within each agency and educated them about any impacts related to changing the interfaces. Most of ESD’s partners were able to modify and test their interfaces with little effort. One large agency was particularly impacted. The late start in addressing interfaces caused a significant workload for the agency, and very little time to complete the work. While the interface work was completed prior to cutover, it was one reason why the agency was not ready to implement in the initial timeframe. The lateness of the request also caused a lack of credibility for ESD that needed to be managed very carefully. Page 15 UTAB Quality Assurance Closeout Report  Federal partners – Several of ESD’s interfaces are with federal agencies and some of these proved to be challenging. Development of the interfaces was not the most challenging issue, as FAST has developed the interfaces for other states. However, gaining appropriate permissions and specifications was very difficult. Phone calls were not returned, information was not available, and credentials were slow to materialize. It would have been helpful to start this “pre-work” at the very beginning of the project. After cutover, several federal reporting interfaces did not work as planned. It became apparent that testing by the federal agency had not been completed, even though they signed an agreement stating that testing was complete. It is valuable for ESD to have confirmation that the testing had been done. Additional work on the interfaces was needed so that ESD could transmit the appropriate data accurately.  Internal ESD Interfaces – Most of the internal interfaces were implemented with little trouble. One benefit of implementing internal interfaces is that the agency has control over the internal resources and can prioritize work appropriately. The greatest challenge with the internal interfaces concerned data. The GUIDE system was old and some data was suspect. Other ESD systems contained data that did not match the GUIDE data. While many of these issues were resolved, the agency has struggled to reconcile some wage and address data, as they do not match within the internal systems. There were conversion issues that have been challenging to manage and are causing pay issues for some UI claimants. Understanding system of record issues, and determining the “right” data source has been difficult. Additional work will be needed to sync this data between systems. 18. It is important that cutover activities be documented and communicated to responsible parties to ensure a smooth cutover process. Cutover activities were well planned and the transition to the new system was executed smoothly. The project team included business and IT staff in creating the plan. They assigned primary and secondary responsible parties, met weekly to discuss progress and roadblocks and used appropriate email distribution lists to ensure all parties had status. Some staff noted that timelines for task completion, particularly on cutover weekend were not communicated until too late in the process. It was unclear whether the tasks would fit within a cutover window. Major tasks such as data conversion were timed in advance but cutover tasks associated to GUIDE were less precise, and there were some discoveries (e.g., existence of yearly and monthly runs that had not been identified) the week of cutover that were going to impact the cutover weekend timelines. In the end, staff were able to execute cutover tasks within the cutover window. 19. Mock conversions provided insight into data issues so that issues could be addressed prior to cutover, and confirmed conversion timelines to ensure data migration could be run within a specific timeline. FAST’s methodology brings with it a mature data conversion process. Conversion activities are started early in the project, which was important given the age and poor quality of GUIDE data. Mock conversions provided the agency with reports detailing data discrepancies and issues that need to be fixed. As was expected, the mock conversions found many data anomalies that were assigned to project team staff for correction. Most of the anomalies were fixed in GUIDE, which is the preferred approach. A few issues were addressed during the Page 16 UTAB Quality Assurance Closeout Report cutover phase so that the data in UTAB would be correct. Some issues were not discovered until the system was live. These issues are identified as defects and worked depending on the priority. Having dedicated resources assigned to data clean-up was critical to ensure that the new system had the cleanest data possible. 20. Testing needs to start early and continue until go-live to ensure that the system is fully and well tested. FAST’s methodology calls for testing to begin during the development cycle as is typical in projects using an iterative approach. The methodology does not include a robust test strategy (e.g., regression testing is not included as part of FAST’s test approach) so ESD developed supplemental testing strategies to ensure some regression testing was completed. ESD also required an additional “end-to-end” or validation test after development completed. The initial end-to-end testing occurred while the system was still under development. Therefore, ESD did not feel that the system had undergone a thorough validation test that included all developed and tested functionality. Because development tasks are not clearly organized or coordinated in Workbench, it was not always clear which tasks were complete and ready to test. Additionally, ESD relied on SMEs to develop test scenarios and system testing. The SMEs were not familiar with these testing processes so preparing the test scenarios took more time than expected. Additionally, the requirements were not updated as changes were made, which relied on the SMEs memory to determine what decisions had been made. By the end of the project, the testing processes had matured considerably and the SMEs were very familiar with the processes needed to manage a test scenario (e.g., scenario writing, test data preparation, results documentation, etc.). The learning curve was significant and contributed to the schedule delays. In future projects, ESD should consider adding professional testers and automated testing to the project teams. Regression testing in particular was not well organized and is not included in FAST’s methodology. Some repaired defects broke other functionality, which was not always visible during the testing phase. This was frustrating for testers and difficult to manage. Some testers reported that there was significant pressure to pass tests that were not fully testable. For example, interfaces, reports, and correspondence were developed later in the development process. If a test scenario relied on this functionality, it should not have been successfully passed, as it was not possible to test the entire business process. It is possible that tests were passed that were not ready, or that failed later when all related items were developed. Frequently, development work was promoted to the system over the weekends. Testers worked during the week, which resulted in an onslaught of work assignments every Monday morning. There was pressure for the testers to get through the testing quickly. Staff suggested that it may have been more effective if they adjusted their hours to meet the development cycle so that they had better access to developers if questions arose. Page 17 UTAB Quality Assurance Closeout Report 21. Contracting an expert usability firm to conduct usability and accessibility testing helped to uncover external usability issues with the external-facing application. The UTAB project contracted for an expert level usability testing firm to conduct usability and accessibility testing for the external website. The testing revealed some improvement opportunities to make the system easier to use. Many of the changes were incorporated during the project phase. Other changes were identified that will require changes to the core system or could not be completed in the project timeline. These changes are documented but should be included in the backlog of items to be looked at as future enhancements. Additionally, a final report outlining accessibility opportunities should be produced so that the backlog accurately reflects actions ESD can take to improve accessibility compliance. Because of time constraints and scope concerns, the internal staff portal was not included in usability. Some staff are concerned that the system is not user friendly. ESD might want to consider conducting a usability study on the internal facing system to improve workflow and business processes. 22. Performance and load testing was inadequate, which resulted in significant performance issues at cutover. FAST conducted a load test on the UTAB system. The test did not include all systems connected to the UTAB system such as the SAW/SEAP portal or telephony systems. During cutover weekend, the system performed very well with no load on it. There was no indication the system would have issues once customers began to use the system externally. However, when customers began using the system it became very apparent that the load, which was unexpectedly high, could not be managed by the telephony system, the UTAB system, or the SAW/SEAP portal. The team quickly began diagnosing the issues with all three systems and by the end of the first day, changes were made to all three systems and performance improved dramatically. While call volumes and customer needs remain high, the UTAB system is managing the volumes well. Conclusion Overall, the UTAB Project was a success. The team carefully managed scope, schedule and budget factors that allowed it to be implemented in a relatively short time, and under budget. Ensuring support during the stabilization period is an important mitigation strategy that should provide sufficient support to ESD. There is continuing concern about how much support the project will need during the operational phase until some of the critical issues are resolved. It is likely that additional operations support will be necessary until the systems stabilize. Page 18 Appendix A Recommendation Matrix SIGHTLINE UTAB Quality Assurance Closeout Report UTAB Recommendations and Findings Summary Recommendations Findings Date Offered Date Closed QA Status 1 Implement a decision tracking tool and process to document and manage decisions made during the configuration sessions. Decisions are not being tracked during configuration sessions. 8/31/2015 4/30/2016 Configuration sessions are now complete. Decisions will still be made and will need to be documented throughout the testing phase. 2 Determine which tool will be used to document and manage issues, risks, and decisions. The FAST workbench does not provide robust functionality. There is no consensus on whether to use it or not. 8/31/2015 9/30/2015 The project will use Workbench to document risks. Issues are being tracked within the risk log. 3 Reconsider the decision to use the dynamic staffing approach for the Applications Architect and Data Architect positions. The dynamic staffing approach may not be the best suited approach for critical project positions. 8/31/2015 9/30/2015 The Solutions Architect will be refocused to fill both roles. 4 Develop a long-term customer support plan for SAW/SEAP services that resides outside of the project boundaries. The project is taking on additional tasks that will need to have long-term operational support. 11/30/2015 4/30/2016 Current calls are being managed. The agency may need to look for additional resources should call volume increase as the number of users increases. This recommendation is closed. 5 Determine impacts of the SAW/SEAP process (including multi-factor lock-out procedure and Lexis/Nexis impacts) and estimate potential customer support needs. [Revised] The currently planned multifactor authentication lockout procedure may not be supportable by ESD in the long term. 12/31/2015 3/31/2017 ESD specific issues are being addressed by WaTech. Other global issues (global unlock feature, revised questions) are still in progress. 6 Refine the project plan to include additional details around the interfaces and other technical tasks that fall on the critical path. Detailed interface tasks are not included in the project work plan. 12/31/2015 7/31/2016 Additional tasks have been added to the plan. A tracking sheet is used to manage more detailed progress. 7 Implement a standard code review process for configuration and development tasks. Code reviews are not being held with state developers resulting in a lost opportunity for knowledge transfer 2/29/2016 5/31/2016 FAST has implemented a weekly meeting between the development lead and ESD developers. This time is used to ask specific questions or review code. Revised 1/31/2017 Appendix A - 1 UTAB Quality Assurance Closeout Report Recommendations Findings Date Offered Date Closed QA Status 8 Develop a process for managing items on the pull list. Items are being added to a pull list with little understanding about what that means or who is responsible for managing the items. 3/31/2016 6/30/2016 A process has been documented. 9 Identify roles and responsibilities for ensuring interface partners are able to receive accurate and complete data through the interfaces. Roles and responsibilities for working with partner agencies are not clearly defined. 7/31/16 9/30/2016 Business staff in all agency partners have been contacted to confirm the partner is able to test and ensure data is accurately transmitted. Some challenges remain but the right individuals have been identified and work is being completed. 10 Develop a prioritized list of correspondence and reports and assign those that are of high priority to developers for immediate development. The schedule is aggressive for report and correspondence development and it is likely that not all of them can be developed within the schedule timeline. 7/31/16 8/31/2016 The team has prioritized all development tasks including reports and correspondence. 11 Ensure discrepancies are documented in the reconciliation report and communicated to project leadership. Discrepancies in the reconciliation report have not been defined. 9/30/2016 10/31/2016 Discrepancies are being documented in the reconciliation reports. 12 Complete a full final validation test (previously referred to as end-to-end test) cycle that includes all required interfaces and converted data. Not all interfaces have been fully tested through an endto-end test cycle. 10/31/2016 12/31/2016 A final validation test was completed. 13 Develop criteria and a review process for promoting any change, including defect repairs, into the system during the final validation test period and code freeze. FAST and ESD do not have common agreement regarding final changes needed for the system. 11/30/2016 12/31/2016 An interim process was defined and generally followed during validation testing and will be implemented for the initial stabilization period. The next step is to transfer control of SQRs/Change requests to business staff. Appendix A - 2 UTAB Quality Assurance Closeout Report Recommendations Findings Date Offered Date Closed QA Status 14 Supplement currently identified operations maintenance staff with seasoned ESD staff knowledgeable about UTAB functionality, testing, policy, processes and staff resource capabilities. Current operations maintenance staff are new to ESD and will need ongoing mentorship. Additionally, there is no clear plan regarding longterm business analysts skill sets available to the team. 1/31/2017 3/31/2017 Roles and responsibilities for M&O have been approved. 15 ESD staff should develop agreement on exit criteria for project closeout. ESD staff do not have clear agreement on exit criteria needed to close the project phase. 2/28/2017 3/31/2017 Business sponsors agreed on exit criteria for project closeout. 16 Confirm responsibilities and processes for monitoring performance standards as agreed to between ESD and FAST. Responsibility for gathering data and monitoring performance is not understood by all parties. 2/28/2017 Tools for monitoring performance are not entirely in place. Some reports from FAST are available but need to be analyzed to ensure they meet the contract’s intent. The M&O agreement needs to be negotiated and signed. Greyed out recommendations are closed. Appendix A - 3 APPENDIX OCIO CRITICAL INDICATORS DASHBOARD SIGHTLINE UTAB Quality Assurance Closeout Report CRITICAL INDICATORS DASHBOARD The Quality Assurance Dashboard provides an assessment of the effectiveness of the Technical Project and assesses the risk areas of the project that have the greatest potential for impacting or delaying the project in accordance with the OCIO Critical Indicators of Project Performance. UTAB Quality Assurance Risk Dashboard Assessment Areas Overall Project Health Previous Risk Level Current Risk Level Comments The project is in the warranty period. The UTAB application is working well and is generally processing appropriately. The team remains focused on implementing defect repairs and enhancements. Risks are elevated as M&O processes are still maturing. YELLOW YELLOW Scope Management GREEN GREEN Time Management GREEN GREEN Governance GREEN GREEN Cost Management GREEN GREEN Quality Management GREEN GREEN The team continually refines processes and tools to monitor project progress. Human Resource Management YELLOW GREEN The team developed a staffing plan for ongoing maintenance. Communications Management GREEN GREEN A Communication structure has been implemented and is being managed appropriately. YELLOW YELLOW The number of customer phone calls is decreasing although they remain higher than normal. This risk will remain elevated until staff become more comfortable with the system, and phone volumes become more manageable. YELLOW GREEN Stakeholder Management Integration Management The project scope was implemented as planned. The team is working to transition the system into M&O in April 2017. The project is implementing a governance structure to be used in M&O. Staff assigned to M&O are knowledgeable about the system and business processes. The project manager maintains a detailed project budget that is used for reporting to the Executive Steering Committee. The budget was underspent by $11 Million. ESD selected a solution that is highly configurable and is expected to meet the agency’s business requirements. Integration with NGTS has been challenging, and is likely to remain challenging during M&O. UTAB Quality Assurance Closeout Report Assessment Areas Previous Risk Level Current Risk Level Risk Management GREEN GREEN The project maintained a robust risk management process. The risks were reviewed periodically and updated as needed. YELLOW ESD plans to acquire appropriate levels of support from FAST to manage the infrastructure during M&O. Issues related to Washington’s security infrastructure (SAW/SEAP) and its configuration for ESD is keeping risks elevated. Call volumes would indicate that customers still require support in accessing the system through SAW/SEAP. Standard Infrastructure YELLOW Comments Procurement and Vendor Management GREEN GREEN The project managed contracts appropriately. Contract provisions were reviewed for deliverable approval processes to ensure that all contractually required exit criteria were complete. Formal Methodology GREEN GREEN The FAST methodology is being implemented as contracted. ESD developed additional tools to manage status. = Risk is the same = Risk is decreasing = Risk is increasing Assessment area is at LOW risk for impacting scope, schedule or budget Assessment area is at MODERATE risk for impacting scope, schedule or budget Assessment area is at HIGH risk for impacting scope, schedule or budget Page 2