AdvanTech , LLC Innovative Technology Management State of Hawaii Department of Taxation Tax System Modernization Program Independent Verification and Validation Year 2 Assessment 3 October 2017 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Executive Summary This document describes the findings and recommendations related to AdvanTech’s seventh assessment of the TSM Program, which represents Year 2 Assessment 3. This seventh assessment is focused primarily on the overall health of the program. The assessment also reviews Rollout 3 post-cutover status and issues, the planning and initiation for Rollout 4. any operational and support issues for TSM components currently in production, and any issues related to ongoing program execution. The process to perform this assessment included: • Interviewing key TSM Program team members and certain other stakeholders, some individually and others in focus group sessions. • Attending a sampling of regular program meetings. • Reviewing various program documents, the FCR database, and other artifacts. We have changed the format of the document from previous assessments to make it more concise, and to make even clearer issues and topics of concern for use by TSM management for tracking program progress in dealing with these issues. At the request of the new TSM management team (effective July 2017), we have also attempted to be more direct (and even blunt) in describing our findings. It is also important to note that the previous TSM management provided some verbal direction to AdvanTech regarding what topics should and should not be addressed in these reports, and also requested modifications to report content. Allowing these types of customerdriven changes to IV&V reports is not the norm in AdvanTech’s experience, and current TSM management has confirmed that this will not be the case going forward. First and foremost, the TSM Program has achieved some solid successes in spite of a number of issues discussed below. Rollout 3 was accomplished on schedule, with no major problems nor significant taxpayer complaints. Several taxes are now being administered using the TSM system, and the number of taxpayers enrolled in eServices continues to grow. There are still some issues with the system, and with users’ comfort with using the system, but none of them appear to be critical in nature. R4 definition activities are underway and appear to be proceeding generally as planned. There has been some discussion about reducing the scope of R4, or even extending the rollout date. If either of these alternatives is to be considered, they must be evaluated very soon by both TSM and DoTAX business units, taking into consideration DoTAX business cycles, any tie-ins between Individual Income Tax and the other scheduled R4 taxes, legacy system maintenance issues, and costs and impacts. 2 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management AdvanTech is concerned that the program has not matured at the rate we would expect based on our experience with other similar programs. This finding is based on an apparent lack of buy-in and expressions of negativity for the program by some managers, confusion over governance and responsibilities, and continued questioning of the vendor’s methodology. These items are affecting the program at a level normally seen at much earlier stages in the implementation. These types of issues are usually largely resolved well before R4 activities are underway. At present, the program is not operating in an optimal way. There continue to be a number of issues and risks related to program execution that, if not addressed and remediated immediately, may have a significant negative impact on the program’s ability to successfully deliver the upcoming critical Individual Income Tax release (R4) as planned. Many of these items have been discussed in several previous IV&V reports, adding to our concern about program maturity. The high priority issues and risks are listed here, and explained in more detail in the Findings section of the report. High Priority Issues 1. The TSM Program is subject to a high level of political sensitivity and is operating in a highly political environment, with various parties interpreting program progress, successes, and issues in different manners. 2. Buy-in for the program has been increasing in the business units, but there continues to be pockets of resistance and some passive/aggressive obstruction to the program. 3. The overall TSM governance structure, management support, and communications continue to evolve, but are still severely challenged and somewhat dysfunctional: a. The new governance structure, effective July 2017, and related lines of communication have not yet been clearly defined, with conflicting communications still being sent to stakeholders. This is particularly important as R4 is perhaps the most challenging rollout, and is the first full rollout being managed under the new structure. b. Management and direction for the program is being seriously affected by personality conflicts, particularly at the leadership level. c. There is still uncertainty regarding individual roles and responsibilities within the program, causing difficulty in making decisions, raising and resolving issues, and addressing program risks. d. There has been an ineffective use of the TSM PMO, and there is uncertainty about its future configuration and use. e. It is still not fully clear to all stakeholders that the program is businessdriven, not technology-driven. 3 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management f. There continues to be confusion regarding the decision-making process and responsibilities. g. At times the project has operated under a “veil of secrecy”. This over-tight control on discussions and communications has caused unnecessary work, anxiety, and non-productive negative adoption. 4. The TSM Program is operating to principles that in our experience are detrimental to a successful project: a. Overly strict adherence to the RFP. b. No set of key performance indicators aligned with an agency strategic plan to measure the overall benefit of the program. c. A lack of professional, dynamic, and rigorous management of the program plan, issues, and risks. d. An inability to come to mutual agreement and understanding of the project methodology, deliverables, and documentation. 5. There are system usability issues, real and perceived. High Priority Risks 1. The lack of funding approval jeopardizes successful R4 and overall program completion. 2. A significant increase in the number of taxpayers on the system with R4 could overload capacity to provide taxpayer support. 3. The DoTAX goal to be fully self-maintaining at the end of the warranty period is unrealistic. 4. Some progress has been made in the areas of business process reengineering and organizational change management, but there is much work still to be done, particularly with documenting business process changes. Key Recommendations 1. Clearly define and communicate roles and responsibilities within the new management structure, including Executive Sponsor, PMO Program Manager, TSM Project Manager, and other PMO members. Also ensure a common understanding of the changed roles for the DoTAX Director and DoTAX CIO with respect to the TSM Program. 2. Ensure that PMO Program Manager and TSM Project Manager are able to commit the necessary time (Ideally as close as possible to full time) to the program. 4 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management 3. Clearly define and communicate the process, roles, and responsibilities for decision making and escalation of issues. 4. Clearly define the communication channels between TSM management and DoTAX management, including how the DoTAX Deputy Director will keep the Director informed of TSM progress and issues. 5. Clearly define the communication channels and responsibilities between the TSM Program Office, DoTAX business units, ITSO, and ETS. 6. Review the charter and make-up of the TSM Stakeholders Group to ensure that all key user groups are represented, including DoTAX business units, ITSO, ETS, and Fast. The charter of the group should be clearly defined to specify its purpose and role in providing guidance and decision making for TSM. 7. Ensure that ITSO management and staff are fully supportive of the TSM Program, and committed to both necessary legacy data conversion and TSM knowledge transfer activities. 8. Work with users to identify and verify system usability issues (e.g. “too many clicks”) to determine whether they are training or learning curve issues, or actual system problems. 9. Evaluate the benefits, impacts, and risks of reducing the scope of R4 to solely Individual Income Tax, moving the other taxes planned for R4 into R5. 10. Perform a reasonableness review of the RFP requirements, and process change orders where appropriate. 11. Evaluate the current status of knowledge transfer plans and activities, and develop a back-up plan for performing post-warranty support services in the likely event DoTAX will be unable to fully maintain the system using its own personnel. These issues, risks, and recommendations, along with others of less priority, are discussed in further detail in the body of the report. The bottom line is that there are no technological reasons preventing the TSM program from being successful, with all taxes migrated to the system within the planned timeframe. By far the bulk of the issues center around governance, communications, resource commitment and buy-in, and expectation management. These issues are solvable, but will require the will and the concerted effort of DoTAX, ETS, and Fast Enterprises. 5 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Assessment 7 Process AdvanTech, LLC, has been contracted by the State of Hawaii, Department of Taxation (DoTAX) to perform Independent Verification and Validation (IV&V) services for the Tax System Modernization (TSM) Program. Under the terms of the contract, AdvanTech will perform periodic assessments of the TSM Program to help identify strengths, weaknesses, issues, and risks related to the implementation of the TSM Program, and to make recommendations for improving the implementation process. The AdvanTech team assigned to perform this assessment was comprised of individuals with significant experience with the planning, management, and assessment of tax processing and administration systems, including numerous projects employing the GenTax system similar to that being implemented by Fast Enterprises for DoTAX. AdvanTech’s approach to IV&V is to look at all aspects of program execution. We are not just focusing on vendor performance, but on all the pieces that comprise a complex program of the nature of TSM. As indicated above, the primary areas of focus for this assessment are the Rollout 3 post-cutover status and issues, and the planning and initiation for Rollout 4. The team also reviewed any operational and support issues for TSM components already in production, and any issues and risks related to ongoing program execution, leading to an assessment of the program’s overall health. AdvanTech employed various methodologies to gather and compile input related to project status, progress, and performance. The primary steps in the input gathering stage were interviewing key agency and vendor personnel, arranging focus group meetings to discuss various implementation activities, attending project meetings, and reviewing relevant project documents and artifacts. As part of our report, we have included an updated status for any recommendations made in the previous reports. In addition to our written report, we also provide the TSM leadership group with an oral debriefing at the end of our information gathering process. In this meeting, we provide an overview of our key findings and answer any questions the team may have. Interviews One important element in assessing the status and condition of the program is the gathering of input from various key stakeholders. To compile this input, the AdvanTech team held a number of onsite and telephone one-on-one and group interviews. We feel that we need to emphasize what has been stated in previous assessments, that participants were asked to be candid in their responses to the team’s questions, and were assured that their feedback will be reported in a manner so as not to identify the specific input as coming from specific individuals. This approach is a key component to obtaining meaningful input on program performance, issues, and risks. 6 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management The following represents the list of stakeholders AdvanTech met during the seventh assessment process: Assessment Scope Review • • • • TSM Executive Sponsor/State CIO – Todd Nacapuy DoTAX Deputy Director – Damien Elefante State CIO Office – Mark Choi State CIO Office – Rona Suzuki Preliminary Findings Exit Review • • • • TSM Executive Sponsor/State CIO – Todd Nacapuy DoTAX Deputy Director – Damien Elefante (via telephone) State CIO Office – Mark Choi State CIO Office – Rona Suzuki Governor’s Office Update • • Administrative Director – Ford Fuchigami Assistant Administrative Director – Lynette Marushige Individual Interviews • • • • • • • • • • • DoTAX Director – Maria Zielinski DoTAX IT Manager – Robert Su DoTAX Administrator, Compliance – Kevin Wakayama (via telephone) DoTAX Audit Program Coordinator – Ikaika Rawlins (via telephone) DoTAX Development and Conversion Lead – Helen Ng DoTAX Audit Manager – Lynn Lyckman DoTAX Audit Manager – Regina Yuen DoTAX Management Analyst, Tax Processing – Carlu Cuizon Fast Enterprises Project Director – Delena Bratton Fast Enterprises Solution Architect – James Doucette Fast Enterprises Training Lead – Katie Christianson Group Interviews • ETS – Mark Choi, Rona Suzuki • TSM PMO – Latrece Cotton, Valerie Iinuma, Stephen Wilson • DoTAX Taxpayer Services & Processing Management Team – Nicki Ann Thompson, Todd Kuromoto, Gayle McGee, Jennifer Oshiro 7 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management • DoTAX Compliance – Janessa Bonifacio (Collections Program Manager), Valentin Gabaon (Collections), Duquesne Hulihee (Hilo – District Manager), Kathleen Sorrells (Hilo – Office Audit), Wendy Yoshioka (Hilo – Field Audit), Erin Tsuda (Kauai – District Manager), Deanne Koshi (Kauai – Field Audit), Denise Cordeiro (Maui – Acting District Manager), Joyce Takahashi (Maui – Field Audit) • DoTAX Primary Business Analysts – Lola Miyashiro, Leslie Terasako • DoTAX Departmental Business Analysts – Susan Adamson, Bonnie Fujii, Young Kwak, Wendy Tokuda, Karen Xelasco, Wendy Yoshioku-Moore • Fast Enterprises Development & Conversion Leads o Returns Processing & E-Services – Troy Daniels, Joseph Felix o Payments, Financials and Revenue Accounting – Jacob Bovatsek o Audit & Collections – Curtis James o Conversion – Brian Rahne o Production Support – Isaac Savella • External Communications o Deborah Kwan – DoTAX Program Information Officer o Keith DeMello – ETS Senior Communications Manager Attend Meetings Another method used by the AdvanTech team to gain insight into project progress, issues and risks, is to attend some sample meetings. We also use this approach to determine a sense of team dynamics. The attendee(s) from AdvanTech are not typically active participants in the meeting, but are there to listen and observe. However, we will provide input related to experience on other projects if requested. Due to scheduling conflicts and the fact that there were not many regular meetings being held during the period of our visit, we only attended the following sessions for the seventh assessment: • TSM Stakeholder Meeting • TSM Production Support Meeting Document Review Another element of the assessment process is the review of key project documents and artifacts. The purpose of AdvanTech’s review is not to determine whether the documents meet contractual requirements. Our focus is to determine if they represent solid practices in each area, and provide a reasonable basis for executing and managing the project, including whether they provide a basis for monitoring various project activities in accordance with the plans for those activities. 8 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management For this seventh assessment, the AdvanTech team reviewed the following project documents and artifacts: • • • • • • • • • • • • • • • Detailed Schedule for Current Rollout (as represented in the FCR Workbench) Project Status Reports Project Issues Log Project Risk Management Log Joint DoTAX/Fast Status Log of IV&V Recommendations SQRs and SQR Statistics Pull List List of Change Requests Memos (9) from Compliance regarding TSM Concerns Knowledge Transfer Progress Update ITSO Requests for Knowledge Transfer Memo from ETS to DoTAX regarding Clarified Expectations for Knowledge Transfer DoTAX TSM Survey Results (Draft) TSM Executive Steering Committee (ESC) Charter Memos on TSM Management from Governor Ige, Deputy Director Elefante, and Director Zielinski 9 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Assessment 7 Findings AdvanTech is concerned that the TSM Program has not matured at the rate we would expect based on our experience with other similar programs. This finding is based on the following observations: • There is still an apparent lack of buy-in, expressions of negativity, and some passive/aggressive obstruction to the program by some managers. • There has been a continued pattern of confusion over program governance and responsibilities, and this has been further exacerbated by the change in program management structure and personnel. • There is a continued questioning of Fast’s methodology. • There does not appear to be an overall sense of partnership between DoTAX, ETS, and Fast. These items are affecting the program at a level normally seen at much earlier stages in the implementation, and are usually resolved well before R4 activities are underway. They are causing a certain amount of dysfunctionality in program execution. The TSM Program has historically operated with an inward focus and a project culture where there was a reluctance to consult and communicate with internal (DoTAX) and external stakeholders. The need for better communication and stakeholder consultation has been a consistent theme in each of our past reports. With the introduction of the new governance structure in July 2017, there are indicators of improvements in communications, staff empowerment, and program management functions. However, there remains confusion around roles and responsibilities, particularly with respect to the role of the DoTAX Director versus the Deputy Director, the role of ETS versus ITSO, and the reporting structure within the program office itself, including the PMO. High Priority Issues 1. The TSM Program is subject to a high level of political sensitivity and is operating in a highly political environment, with various parties interpreting program progress, successes, and issues in different manners. TSM is the largest technology initiative currently underway in the State, so the program has a very high level of visibility. It is also being closely monitored because of relatively recent failures of certain large programs that carried a high cost. 10 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management This sensitivity was heightened by the issues with the eServices registration component during the R2 cutover. These issues, which impacted a relatively small percentage of taxpayers and which have been corrected, still generated the perception by several key stakeholders, particularly in the legislature, that the program was struggling, and possibly even failing. This perception led to the legislature denying, at least for this fiscal year, the approval for the additional funding needed to complete the program, and to the change in leadership and governance structure for the program. The political component has now been further exacerbated by the filing of grievances with the union by a number of DoTAX employees claiming issues with the new TSM management (effective July 2017), and intervention by the union with the legislature, governor’s office, and the press. The overall impact of this politicization of the program is to create an image of failure, which could end up being self-fulfilling. This in turn is creating uncertainty about the future of the program, and potential disruption to the plans and activities of the program. 2. Buy-in for the program has been increasing in the business units, but there continues to be pockets of resistance and some passive/aggressive obstruction to the program. The biggest area of concern is ITSO management’s apparent lack of support for the TSM Program's success under the current leadership model. ITSO has historically been very supportive, dedicating significant resources to the effort, with the DoTAX IT Manager being a primary champion of the program. However, that level of support and involvement has diminished since the change in TSM leadership was put in place in July of this year. ITSO has continued to perform the necessary activities for data conversion, and is still partially engaged in working on SQRs for the GenTax system and other TSM components, but not at the same level as in the past. AdvanTech received input from at least three sources that, since the change in TSM leadership, key ITSO staff were being encouraged to discontinue their project involvement. Several ITSO personnel that were involved in TSM have filed grievances with the union concerning the change in leadership. AdvanTech has not seen the actual grievances, so we are unable to determine if they are simply a general protest against the change, or represent real complaints that could impact the success of the project. As mentioned in the first issue above, the union has intervened with the Governor’s office and the legislature, and the story has been picked up by the press. There are also rumors that the staff were being encouraged to file grievances with the union, but we could not substantiate those rumors. In any event, whether true or 11 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management not, just the perception that this is occurring can have a highly detrimental effect on the program. There is also a perception that ITSO has a lack of commitment to the TSM knowledge transfer program, as evidenced by the change in priority of several ITSO personnel from TSM activities to legacy maintenance shortly after the change in TSM leadership. AdvanTech received mixed input on the level of ITSO’s commitment to knowledge transfer, with the current belief by ETS that ITSO does not want to re-commence knowledge transfer until the warranty period (i.e. post implementation). This belief is based on discussions ETS had with DoTAX management, and for which ETS sent a memorandum to DoTAX to confirm this understanding. As of the time of AdvanTech’s assessment visit, DoTAX had not yet responded to the ETS memo. Further, ITSO resources no longer physically sit with the project team (a critical success factor for knowledge transfer). However, the DoTAX IT Manager indicated that ITSO is willing to participate in knowledge transfer, but is waiting to respond to the ETS memo and waiting to send people back to the program office until a number of personnel grievances submitted to the union have been resolved. In the meantime, ITSO has prepared a list of knowledge transfer needs by category which was provided to Fast. This conflict of understanding must be resolved immediately to determine what level of knowledge transfer can be achieved during the remaining implementation process and warranty period. DoTAX management should do whatever is necessary to ensure that ITSO management and staff are fully supportive of the TSM Program, and committed to both necessary legacy data conversion and TSM knowledge transfer activities. The Compliance Division continues to have areas where support for the program is less than optimal. This is due to some concerns about system usability, some of which are real, while others appear to be mostly learning curve, along with a negative attitude by some users because the new system does not do things the in exactly the same manner as the old. However, Compliance management has proactively compiled lists of issues, recommendations, and other feedback from each of its business areas, including the district offices, in an effort help facilitate the resolution of their concerns. The addition of new senior resources within Compliance, with a mandate to analyze and track resolution of TSM-related issues, is also expected to help address these concerns. The Compliance Division has also set up a TSM Update Committee to resolve outstanding issues and to integrate Compliance as fully as possible into the TSM R4 development. This committee is responsible for staffing TSM with the right persons, providing appropriate user input into definition meetings, elevating problems to Compliance leadership, and then to TSM leadership. 12 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 3. AdvanTech , LLC Innovative Technology Management The overall TSM governance structure, management support, and communications continue to evolve, but are still severely challenged and somewhat dysfunctional. The TSM Program governance structure and management responsibilities have gone through major changes, and are still not fully bedded down. The changes continue to have a significant effect on program operations. There still exists some feelings of fear, uncertainty and doubt amongst DoTAX and TSM staff regarding the change. If not resolved effectively, the program is at risk of entrenching factions who may work at cross-purposes. DoTAX has already experienced staff movements off of the project and grievances that purportedly relate to competing views of program management. AdvanTech did receive some encouraging comments that the general climate is improving and that there is confidence that as the changes solidify, further improvements will be realized. But this feeling is definitely not consistent across all DoTAX and TSM staff, with some staff still wondering why the change occurred considering that previous rollouts were delivered on schedule. In addition, R4 is the first full rollout under the new structure, so it is critical that management responsibilities and communication channels be solidified quickly to minimize potential disruption to the R4 implementation process. The most important issues to deal with related to the change are: a. The governance structure and lines of communication have not yet been clearly defined, with conflicting communications still being sent to stakeholders. Conflicting memos have been sent out trying to explain what the changed structure means to TSM operations, but have apparently created more confusion than clarity. The “memo wars” are symptomatic of the lack of clarity around roles and responsibilities, even at the most senior levels, which has a ripple effect into the organization. We are also concerned that the new governance structure is completely isolating the DoTAX Director from TSM involvement. Assigning the Deputy Director with the primary responsibility for setting program direction and coordinating involvement by the business units helps provide a business-driven focus to the effort, as long as his role is clearly defined and he is able to dedicate sufficient time to the program. However, the Director has overall responsibility for all DoTAX operations, and the TSM system will be the principle tool for executing those operations. The Director has also played a key role as a champion of the program and should be encouraged to continue in that role. We feel it is imperative that at a minimum, the Director be regularly kept up to date on program activities and key decisions. AdvanTech recommends that steps be taken to clearly define the communication channels between TSM management and DoTAX 13 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management management, and to clearly define the communication channels and responsibilities between the TSM Program Office, DoTAX business units, ITSO, and ETS. In addition, the Fast Project Director needs a clear primary point of contact, and points of contact and counterparts for the Fast team leads need to be confirmed. b. Management and direction for the program is being seriously affected by personality conflicts, particularly at the leadership level. There are personality conflicts at the TSM/DoTAX/ETS leadership level that are having an impact on the program management approach, decision making, buy-in and support, and sense of partnership. These conflicts are focused largely between the previous and current TSM management teams, with negative statements being made between them. We recommend that the parties come together in a face-to-face session, possibly with a third-party mediator, to resolve these differences. c. There is still uncertainty regarding individual roles and responsibilities within the program. As noted above, there have been conflicting communications concerning the changed roles of the DoTAX Director vs. Deputy Director and the ETS CIO vs. the DoTAX IT Manager. This must be clarified quickly. We also continue to get input that the members of the PMO, and the Primary, Secondary, and Departmental BAs still do not have full guidance regarding their responsibilities and authority. This is causing difficulty in making decisions, raising and resolving issues, and addressing program risks. TSM management should consider engaging in a formal responsibility exercise, such as RACI (Responsible, Accountable, Consulted, Informed), for the key TSM program participants, including participants from DoTAX, ETS, and Fast. This exercise should be used to surface and resolve areas where there is specific confusion about “who has responsibility” (e.g. resolve responsibility for end-to-end testing). The PMO should reiterate to the Departmental BAs that part of their role is to act as liaisons with their respective business units to communicate to and from the units and the TSM Program office. DoTAX and ETS should ensure that PMO Program Manager and TSM Project Manager are able to commit the necessary time (Ideally as close as possible to full time) to the program. d. There has been uncertainty about and ineffective use of the PMO. In the past, the focus of the PMO was split between “PMO” specific duties (managing scope, schedule, deliverables, issue/risk logs, etc.) and as focal points for configuration decision making, even though not all of the PMO members had a tax business background. This split in roles and lack of 14 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management background made it difficult for the PMO to function effectively. Since the change in TSM leadership in July, some of the original PMO members have left the program, and others do not feel they have clarity on their current roles, nor their longevity. TSM management should make final decisions on the configuration and staffing of the PMO, and clearly define the roles and responsibilities of the PMO as a body and of each individual member. e. It is still not fully clear to all stakeholders that the program is businessdriven, not technology-driven. There still is a perception among some DoTAX staff that TSM is a technology initiative being driven by IT (first ITSO, now ETS). Along with that perception are feelings that the business units are not always being engaged in key decisions regarding project direction and system configuration. Ineffective interaction between IT and the business is one of the key causes of failure in large programs of this nature. There must be effective collaboration between the program office, DoTAX business units, and ITSO, and between ETS and DoTAX management. Assigning the DoTAX Deputy Director to act as the coordinator of business unit interests as his primary responsibility may help to alleviate this perception and ensure better interaction between the business and TSM. However, we remain concerned that isolation of the DoTAX Director from the process, and the apparent partial disengagement by ITSO from the program, could offset any gains in this area. Meetings of the TSM Stakeholders Group have resumed on a regular schedule as R4 activities have commenced in earnest. We believe these meetings to be very valuable for the program, particularly if focused more on issues and key decisions, rather than status updates. The make-up of this group should be reviewed and adjusted as necessary to ensure that all key parties are represented, including DoTAX business units, ITSO, ETS, and Fast. The charter of the group should be clearly defined to specify its purpose and role in providing guidance and decision making for TSM. f. There continues to be confusion regarding the decision-making process and responsibilities. During the early stages of the program, responsibilities for making decisions on various program matters was not clearly defined. Early in the R3 cycle, the PMO developed a a document called Project Decision and Escalation Process, which was reviewed with and approved by the TSM Stakeholders Group. This document helped clarify which decisions were to be made at the development team level, and which needed to be escalated for further review. 15 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management As part of that process, all requirements definition and configuration decisions were being cycled through the affected business units before being finalized. This helped to ensure broad input. The Project Decision and Escalation Process document confirmed the balance of empowering program team members to make appropriate decisions, coupled with an effective process for escalating those decisions that have a significant impact on business processes or taxpayer interface. It is critical that the decision-making process be clearly defined and communicated. With the new TSM management structure, high priority must be given to reviewing, revising, and communicating the decision process and other related program governance procedures. g. At times the program has operated under a “veil of secrecy”. The project climate has sometimes been described as working within a "veil of secrecy" or even a climate of "paranoia". There was a continuing resistance by the previous TSM management to communicate broadly, both outside and to a lesser degree within DoTAX. There was also hesitancy to expand governance bodies to include other participants, or to consider any variance from the original project vision, scope, approach, timeline, or RFP. In discussions with the previous TSM Program Director, this reluctance was based on the belief that any variance from plans and/or RFP requirements, or problems with the TSM system or program operations, would be perceived as program "failure", with serious potential consequences in a highly politicized climate, including a possible political decision to deny funding. As part of this effort to control communications, the previous TSM management regularly provided verbal direction to AdvanTech regarding what topics should and should not be addressed in the IV&V reports, and also requested modifications to report content. Allowing these types of customer-driven changes to IV&V reports is not the norm in AdvanTech’s experience, and current TSM management has confirmed that this will not be the case going forward. As a result, stakeholders (including DoTAX users and legislators) sometimes suffered from a lack of fact-based information that undermined confidence as to whether the program was working effectively, achieving its goals, or meeting tax clients' needs as it proceeded. In a sense, the climate resulted in a selffulfilling prophecy, with funding approval being delayed and the governance structure completely changed. This over-tight control on discussions and communications has caused unnecessary work, anxiety, and non-productive negative adoption. There has been improvement in making communications with all stakeholders more open and transparent, but the process for communicating is not yet well defined. 16 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Priority should be given to ensuring communications regarding the program are open and straightforward, and that there are no artificial bottlenecks to effective communications. This will help set expectations and reduce the possibility for misunderstanding. As part of this process, the new TSM management is now posting the TSM IV&V reports online. 4. The TSM Program is operating to principles that in our experience are detrimental to a successful project. a. Overly strict adherence to the RFP. The contract with Fast Enterprises states that the RFP takes precedence over Fast’s proposal and BAFO. This contracting approach can be effective, but only if it is administered with a rational interpretation. There have been instances where members of the previous TSM PMO involved in the management of the contract have pushed for adherence to a particular RFP requirement even when it does not make any sense. Some of this attitude still appears to exist, particularly within ITSO. An example of this is the requirement to provide a kiosk where taxpayers can come to the DoTAX offices and use a computer to interact with the agency using the eServices function. Research has indicated that this is not an efficient way to provide service, as most people who would try to utilize the kiosk are either not computer literate, so would need help from a staff member anyway, or would not be inclined to bring to a kiosk all of the records needed to complete a transaction. Even though the Taxpayer Services & Processing has indicated that this requirement was added to the RFP as a wish list item without fully understanding the pros and cons, and now is no longer interested in implementing the function, there has still been push from ITSO to have Fast implement it solely because it is an RFP requirement. Another example relates to the plans for activating the IVR’s call back feature, allowing callers not to have to wait online, and to further refine the calling queue. Even though the programming work to enable these changes had been largely completed by ITSO staff, there was unwillingness by ITSO management to put the changes into production due to concerns about potential impact on the system warranty. These warranty concerns had not been communicated to TSM, Fast, or the affected business unit (Taxpayer Services & Processing) until TS&P asked about the status of the program upgrades in the Stakeholders meeting. The issue was finally resolved in that meeting with the intervention of the DoTAX Director and the TSM Project Manager, along with Fast’s indication that there would not be any warranty issues related to the upgrade in question. DoTAX and ETS should work with Fast to perform a reasonableness review of the RFP requirements, and process change orders where appropriate. 17 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management b. No set of key performance indicators aligned with an agency strategic plan to measure the overall benefit of the program. Integrated tax systems generally provide value in many ways. This value can be from increased revenues through enhanced compliance programs, increased work efficiencies, and reduced infrastructure costs. Where possible, these benefits should be measured, and a set of metrics developed to confirm the value of the program. DoTAX should work with Fast to develop appropriate performance metrics, and a method for monitoring and reporting against those metrics. The performance indicators should be benchmarked with other revenue/tax agencies that have implemented similar programs. Two members of the PMO have begun working on developing an initial KPI model. c. A lack of professional, dynamic and rigorous management of the program plan, issues and risks. There are issues and risks logs located in FCR, but they do not appear to be used often nor maintained on a timely basis. In order to compile a complete list of issues and risks being tracked by the program office requires going to multiple places within FCR. Issue and risk logs should be consolidated and kept current. Effective use of these logs will reduce project “churn” (i.e. concerns about whether an issue or risk is being addressed can be allayed knowing that it has been assigned to an individual or team for resolution, and progress is being monitored and tracked). This is an activity that should assigned to a member of the PMO. d. An inability to come to mutual agreement and understanding of the project methodology, deliverables, and documentation. Fast has developed a methodology for implementing GenTax that has been successfully utilized on many projects. However, that methodology could be considered a hybrid of different development life cycles (agile, waterfall, etc.), so was not familiar to DoTAX personnel, causing issues of understanding and acceptance. In addition, GenTax is considered a commercial-off-the-shelf (COTS) product, even though it still requires a significant process for defining how it will be configured to meet the needs of a particular client. As a COTS product, GenTax has an established architecture that is generally very flexible, but still requires that configuration is completed within the confines of that architecture. Because the legacy systems and previous failed projects were custom built systems, some DoTAX staff have had the expectation that GenTax could be configured either exactly as it is done in legacy, or otherwise customized to follow a preconceived notion of how a process should be done. 18 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management The discomfort with the methodology and the initial lack of understanding of how COTS works is common to GenTax projects, but it is normally resolved as program participants become more familiar with the product and the methodology. On TSM, there is a higher level of discomfort and continued desire for customization than we would expect at this stage of the program based on our experience with other GenTax projects. Another facet of Fast’s methodology is to focus on implementation over documentation. While this approach helps to ensure on time delivery of each rollout, there continues to be concern by both DoTAX and ETS about the extent and quality of the documentation in general, and more particularly, that the mechanism for documenting configuration decisions is inconsistent, not cohesive, and not easily traceable. Some of the documentation that does exist is in meeting minutes, making it difficult to locate and track as the program progresses With R4 definition activities well underway, the PMO should work with Fast development team leads and the Primary BAs to ensure a solid, consistent process is in place across all teams. This process should enable easy traceability of configuration decisions to as-built configuration. 5. There are system usability issues, real and perceived. There continue to be some complaints about system usability that are not necessarily tracked through the trouble ticketing and SQR processes. These are items such as “it takes me many more clicks in the new system to perform the same job.” Some of these may just be training issues, others may be situations where it does take longer to do a particular task, but in so doing it has improved other downstream activities. There is also a concern that some of the users making complaints do not have basic computer skills, so are not comfortable with the navigation and help tools incorporated in the system. Associated with the usability issues, there has been an element of “rumors” which should be addressed through fact-based analysis. For example, there has been a perception in the organization that front end “backlogs” are serious and preventing both Collections and Audit from being able to effectively perform their duties. Appropriate resources need to be deployed to analyze the facts behind these rumors to determine reality vs. perception, using specific examples and root cause analysis. As mentioned earlier, Compliance management has proactively compiled lists of issues, recommendations, and other feedback from each of its business areas, including the district offices. These lists provide identify some of the specific problems they are facing with system usability, training and documentation, and involvement in system development and testing. 19 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Fast and DoTAX should work together to reach out to some users to better determine the nature of usability issues discussed above, and find ways to resolve them (user bulletins, additional training, etc.). Some of this activity has occurred, particularly with the front-end processing area. However, this process should also occur in other areas, particularly where users are having more difficulty coming up the learning curve (e.g. Collections). The TSM team should also review the memos from Compliance and develop action plans accordingly. Users continue to be challenged with the need to work in two separate systems. This is a natural effect of a phased implementation of the new system, but it does add extra burden on the users. These challenges will be resolved naturally once the implementation of TSM is complete and the legacy systems are retired. High Priority Risks 1. The lack of funding approval jeopardizes successful R4 and overall program completion. As discussed in previous assessments, the Legislature denied the request for the second round of funding for the program for the current fiscal year. The legacy systems (both hardware and software) are in their end-of-life stage. DoTAX feels it is imperative to keep moving forward with TSM to minimize the possibility of a critical disruption of service. A similar denial of funding in the next session will almost certainly ensure that the program will not reach full completion, and could possibly interrupt R4 in mid-stream. This would require DoTAX to work in multiple systems (the legacy system and the new GenTax system), prevent DoTAX from realizing fully the anticipated benefits of TSM, and leave significant exposure to the failure of the remaining legacy systems. DoTAX has shifted some operating funds into the program for this fiscal year in an effort to allow continuation of work on the TSM Program until the request for the additional program funds is submitted and acted upon in the next legislative session. 2. A significant increase in the number of taxpayers on the system with R4 could overload capacity to provide taxpayer support. There will be a significant jump in taxpayers using the GenTax eServices functionality when Individual Income Tax is brought online, relatively few of which will have been previously registered for eServices. Actions are underway to activate the IVR’s call back feature, allowing callers not to have to wait online, and to further refine the calling queue. Fast and DoTAX are looking into a broader reprogramming of the IVR system as part of R4, but there is still a risk that this increase in new users may generate more activity than the call center can handle. 20 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management DoTAX is evaluating other options for handling the total call volume, including using operational staff to man phones during peak calling periods. Engaging sufficient support resources to address taxpayer and preparer concerns is absolutely critical to successfully support the roll out of IIT and ensure that there is not a public perception of “failure”. AdvanTech also recommends that extra care be given to the preparation of online help aids, such as the videos prepared for R3. In depth testing of the R4 eServices functionality should also be performed. For R3, in addition to the normal internal testing process, the Taxpayer Advocate’s office reached out to 200 tax practitioners and 100 taxpayers known to be vocal users to request them to review the changes being made to eServices with R3 cutover. Supplemental testing of this nature should be considered for R4. 3. The DoTAX goal to be fully self-maintaining at the end of the warranty period is unrealistic. DoTAX has consistently stated its goal to be "fully self-sustaining" in terms of system support by the end of the TSM Program warranty period. In each of our previous IV&V reports, we have advised that this is an overly ambitious goal. In fact, to our knowledge, being fully self-sustaining has not been achieved in any other jurisdiction which has implemented GenTax (with one possible minor exception). Any significant level of ongoing maintenance requires a great deal of advance planning, investments in DoTAX staff, and ongoing provision of information and documentation about changes to the core product code. To accomplish this, significant training must occur for the DoTAX technical IT staff, business analysts, and expert users in order for DoTAX to be able to support the TSM system going forward. "Self-sustaining" was always a "stretch goal" for the State of Hawaii, and unlikely to be achieved even under ideal circumstances. However, the current circumstances are far from ideal. The knowledge transfer process appears to have been thrown into complete confusion. As noted in High Priority Issue 2 above, ITSO management and staff appear to have stepped way back from their previous commitments to early and continued knowledge transfer efforts and assessments. ITSO’s level of commitment to and capacity for knowledge transfer must be resolved immediately to determine what level of KT can be achieved during the remaining implementation process and warranty period. Lack of knowledge transfer will not prevent the project from proceeding or succeeding. However, it will ensure that the stretch goal of full self-maintenance cannot possibly be achieved, and will even result in a lower level of selfmaintenance capability than most GenTax jurisdictions. Ultimately, it will also 21 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management result in a greater dependence upon the vendor post project, requiring a more costly maintenance package for the State. DoTAX, ETS, and Fast should work together to evaluate the current status of knowledge transfer plans and activities, and develop a back-up plan for performing post-warranty support services in the likely event DoTAX will be unable to fully maintain the system using its own personnel. This contingency plan should identify the possible split of post-Implementation responsibilities between DoTAX and Fast, and the related budgetary impacts. 4. Some progress has been made in the areas of business process reengineering (BPR) and organizational change management (OCM), but there is much work still to be done, particularly with documenting business process changes. In the early stages of the TSM Program, there were differing expectations between DoTAX and Fast regarding the methodology and responsibility for BPR and OCM. DoTAX felt that Fast should take the lead and use their experience from other GenTax implementations to guide DoTAX to best practices. DoTAX is still concerned about Fast’s reluctance to provide examples from other sites. Fast has felt that DoTAX needed to be more proactive in identifying areas where they believe they would benefit from change, and that the DoTAX business units were more inclined toward retaining current practices (“we have always done it that way”) than adopting best practices. For the past several months, Fast has assigned two qualified individuals to work part time with the TSM Program Office and DoTAX to develop plans for both these areas. They have performed surveys and developed workshops to engage DoTAX management in the BPR and OCM process. This process is continuing to move forward, with further meetings planned for November. There continues to be a risk that BPR and OCM activities will lag behind the implementation effort, resulting in the possibility that opportunities may be missed in some cases for adopting optimal business processes in conjunction with system configuration. High priority should be given to making progress with the BPR and OCM activities, proactively identifying those opportunities for change, and ensuring that any changes are documented and incorporated in user training. There are a number of other issues and risks related to program execution, many of which have been discussed in several previous IV&V reports. The ongoing nature of some of these items adds to our concern about program maturity. On the other hand, the program continues to proceed on schedule in spite of these issues. R3 cutover occurred as planned, with no major problems nor significant taxpayer complaints. There are still some issues with the system, and with users’ comfort with using the system, but none of them appear to be critical in nature. 22 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management R4 definition activities are underway and appear to be proceeding generally as planned. However, the program is not operating in an optimal way, which could have a negative impact on the upcoming critical individual income tax release. System Development and Operations The cutover for R3 went fairly smoothly, and system functionality is largely bedded down. The number and severity of fixes (SQRs) and change requests still in process falls within a range we would consider normal for a GenTax implementation at this stage based on our experience with many other GenTax projects. Rollout 4 is scheduled for November 12, 2018, and consists of Individual Income, Partnership, Fiduciary, and Estate & Transfer taxes, along with cashiering. R4 definition and initial configuration activities are underway. Some stakeholders have expressed concern that R4 is falling behind schedule, but we did not yet see any specific evidence of delay. There has been some preliminary discussion about reducing the scope of R4, or even extending the rollout date. If either of these alternatives is to be considered, they must be evaluated very soon by both TSM and DoTAX business units, taking into consideration DoTAX business cycles, legacy system maintenance issues, and costs and impacts. It is AdvanTech’s opinion that delaying the rollout of Individual Income Tax has significant risk, as there is limited flexibility in the business cycle for IIT. However, the move of other R4 taxes to R5 is worth considering, though there is often a tie-in between Individual Income and the other scheduled R4 taxes, so that should be taken into consideration in the evaluation. 23 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Users continue to be challenged with the need to work in two separate systems. This is a natural effect of a phased implementation of the new system, but it does add extra burden on the users. These challenges will be resolved naturally once the implementation of TSM is complete and the legacy systems are retired. Front End Processing: There are no major changes to this area since the previous report. The scanning process is working reasonably well, with the backlog under control, though there was a spike in late July with the semi-annual GET filings. The decision has been made that two scanners in Oahu are sufficient. Current throughput continues to indicate that the two scanners can handle the estimated volumes. There is still some concern that there is no redundancy in case one scanner goes down or is needed for testing programming changes, but DoTAX will manage those issues to the degree possible. DoTAX has also decided to procure to additional regional scanners. The Captiva data capture software continues to improve and “learn” as recognition errors are identified and corrected. Users continue to gain proficiency with the correction process. As part of R4 definition activities, continued emphasis should be given to getting forms fully designed (with input from TSM), reviewed, and approved sufficiently in advance of the scheduled configuration. Deposit 21 continues to work well for getting checks processed and deposited, but there continues to be challenges with the reconciliation of payments for tax types not yet moved to GenTax. This will be the case until TSM implementation is complete, though the process has continued to improve as staff gains comfort with Deposit 21 and with the development of additional reconciliation reports. TS&P is still concerned that the lock box arrangement is the biggest cause for payment processing delays. Delayed payment processing has serious potential impacts including issuance of “false” statements of account (i.e. not reflecting payments already made), “broken” payment plans (i.e. based on incorrect determinations that payment plan obligations have not been met), and wasted collections efforts for balances not truly owed. Thus, every effort must be made to ensure payments are processed and allocated to correct accounts as soon as possible after their receipt. Taxpayer Processing is commended for assigning resources to analyze and propose solutions for resolution for business process issues within their organization. DoTAX has decided not to utilize the mail opening equipment provided as part of TSM, as it would only handle approximately 25% of the incoming mail due to size constraints, 24 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management thus requiring additional pre-handling. DoTAX is in the process of adding features to the existing Pitney Bowes equipment to improve overall mail handling. eServices: The rollout of eServices components in R3, including the upgrade to Version 10 of the GenTax online functionality, was achieved with relatively few issues. The issues seen after R2 cutover with taxpayers having difficulty registering for eServices under the new system did not recur. As part of R3, Fast prepared a series of short videos that were made available online to assist the taxpayers with registration, filing, payment, and account access questions. The decision to use Modernized Electronic Filing (MeF) for Corporate Income Tax reduced the scope of the R3 online component, helping to reduce post-cutover support activity. However, with MeF scheduled as a separate mini-rollout in January 2018, care must be given to develop solid plans for managing and executing that effort. As noted above, R4 will see a dramatic increase in the number of online users, relatively few of which will have been previously registered for eServices. We recommend that extra care be given to the preparation of online help aids, such as the videos mentioned above, and with adjusting call center capacity to handle increased volumes, particularly during peak filing periods. Extensive, in depth testing of the R4 eServices functionality should also be performed. Supplemental testing using taxpayers and practitioners as was done in R3 should be considered for R4. System Configuration: R4 configuration is in the very early stages, so there are no major issues to report. There still are a number of complaints about system usability issues such as “it takes too many clicks”, “it’s not the way the old system worked”, “it is more difficult to find information”, etc. Fast and DoTAX should work together to reach out to users to better determine whether they are training or learning curve issues, or actual system problems, and to find ways to resolve them (user bulletins, additional training, etc.). Some of this activity has occurred, particularly with the front-end processing area. However, we still believe this process needs to occur in other areas, particularly where users are having more difficulty coming up the learning curve (e.g. Collections). The addition of new senior resources within Compliance (Audit and Collections), with a mandate to analyze and track resolution of TSM-related issues, is expected to help address these concerns. 25 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Data Conversion: The R3 data conversion proceeded as planned. Users are still learning how to perform actions in GenTax using the converted data, but that is normal for any new system implementation that includes converting legacy data. The R4 data conversion planning process is underway, with emphasis on identifying legacy data clean-up requirements. Fast has indicated that they would like to see more involvement by the business units in the conversion planning process. Discussions are continuing on the conversion of historical images. The TSM Program Office is analyzing the storage needs and related costs for various levels of historical image conversion. In addition, some of the historical images would need to go through a separate time consuming rendering process to be moved into TSM. Access Security: There still seems to be confusion about who is responsible for maintaining the access security functions in GenTax. We could not determine whether DoTAX (ITSO) has identified specific individuals to train and assume the system security role. There was an instance where a contract employee of ITSO requested access to the system, and the access was granted without going through normal approval processes. AdvanTech recommends that access security policies and procedures be fully documented and communicated, and that roles and responsibilities for maintaining the TSM access security model be defined, with individuals specified for those roles. As with previous rollouts, for a period of time after R3 cutover, there were a number of help requests regarding access security. Many were related to username and password resets, but others were situations where access rights had changed. The number of requests of this nature has continued to decline as users become more experienced with the system and related security controls. User and Production Support: The support process for internal users continues to work reasonably well. There is a help desk function to provide users a single point of contact for their system help needs, whether TSM or legacy. There is also a trouble ticketing system to capture all user issues, whether they may be system problems, training issues, problems with security settings, or enhancement requests. As previously reported, the ticketing system seems to be generally functioning as planned, but there does not appear to be a method of tracking calls that do not result in trouble tickets, reducing the ability to detect trends and identify areas needing extra attention. 26 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Approved tickets related to TSM are then moved into the SQR process for tracking and resolution/implementation. The SQR process is working effectively to record and categorize these issues, and then track them through development and testing. The number and severity of open SQRs is reasonable when compared to other GenTax implementations of similar size and scope. However, the process for prioritizing SQRs is not effective. Each business unit provides input on the priority of their specific SQRs, but there does not appear to be consistency on how the priority levels are determined, nor is there a method for setting overall priorities across business units. There also has been some confusion related to the classification of items as SQRs, change requests, or pull items. The program has set up a virtual production support team. Fast has assigned an individual to lead the production support effort, plus another individual to focus on issues with the non-GenTax components. In addition, each development team has a secondary BA whose primary responsibility is production support. As indicated in the section on Configuration, there are ongoing complaints about system usability that are not necessarily tracked through the trouble ticketing and SQR processes. These are items such as “it takes me many more clicks in the new system to perform the same job.” Some of these may just be training issues, others may be situations where it does take longer to do a particular task, but in so doing it has improved other downstream activities. As noted in previous assessments, the call center staffing has been increased and the automated call system has been refined to provide more selective routing to aid with the taxpayer support function. In addition, plans have been in place to activate the IVR’s call back feature, allowing callers not to have to wait online, and to further refine the calling queue. However, even though the programming work to enable these changes had been largely completed by ITSO staff, there was some hesitancy by ITSO to put the changes into production due to concerns about potential impact on the system warranty. These concerns were discussed and resolved during the stakeholder meeting, with the implementation to be completed shortly thereafter. There will be a significant jump in taxpayers using the GenTax functionality when Individual Income Tax is brought online. Fast and DoTAX are looking into a broader reprogramming of the system as part of R4, but there is still a risk that this increase in new users may generate more activity than the call center can handle. DoTAX is evaluating other options for handling the total call volume, including using operational staff to man phones during peak calling periods. Engaging sufficient support resources to address taxpayer and preparer concerns is absolutely critical to successfully support the roll out of IIT and ensure that there is not a public perception of “failure”. 27 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Infrastructure: The hardware and network components of TSM installed at the permanent data center continue to perform as planned. There is no facility in place for doing a completely seamless failover to another location in the event of a full shutdown at the current production site. Plans to move the DR site to the University of Hawaii data center have been put on hold. Fast and DoTAX are working together on a plan to upgrade facilities at the DoTAX building to allow for a fully secure and redundant DR site. Project Execution As discussed above, most of the critical issues and risks faced by the program relate to project execution as opposed to system capability or usability. There are elements of the TSM Program that we would consider to be dysfunctional. Program Management: The change in the management structure for the TSM Program is still not fully bedded down, and it is having a significant effect on program operations. As discussed above, there continues to exist some feelings of fear, uncertainty and doubt regarding the change. There were some encouraging comments that the general climate is improving and that there is confidence that as the changes solidify, further improvements will be realized, but this is not an across-the-board feeling within DoTAX. The most important issues to deal with related to the change are discussed in the High Priority Issues above. There continues to be some siloing of the work teams, which may result in configuration decisions being made in one team without input from other teams whose functions may be impacted. It is important to work on steps to reduce siloing with R4 definition activities already well underway. Meetings of the TSM Stakeholders group have resumed on a regular schedule as R4 activities have commenced in earnest. We believe these meetings to be very valuable for the program, particularly if focused more on issues and key decisions, rather than status updates. As noted above, some key technical staff continue to express a desire to limit their work on TSM, which has the potential to impact data extracts and conversion for R4, and to severely hamper the knowledge transfer process. This is an important issue for management to resolve as quickly as possible. 28 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Testing: R4 test planning has just gotten underway. While the R3 testing program was generally executed as planned, both DoTAX and ETS have continued to express concerns with the process. They still feel that it is somewhat rushed, both for scenario writing and execution, and that there is significant pressure to limit the complexity and push through the passing of test scenarios. On the other hand, the TSM testing team is concerned that DoTAX did not provide a sufficient number of testers on a timely basis that have a strong working knowledge of Hawaii tax policies and operations, and have familiarity with the elements of TSM already deployed. Resource balancing is always a challenge in a program of this magnitude, as personnel are needing to do their normal workloads while dealing with learning the new system, working in two separate systems, and being asked to participate in program activities (definition, testing, training) to varying degrees. However, adequate testing is critical to the success of the new system, and people participating in testing get additional awareness of the system, making it easier for them to work with it when it is moved into production. Moreover, the demand for quality testing resources only increases as the project progresses with the addition of regression testing, as well as the potential impacts of version upgrades. DoTAX feels that the testing process is made more difficult because the configuration is constantly changing at the same time scenarios are being developed and/or executed, leading to the need to often re-write scenarios. This issue is a natural, but challenging offshoot of Fast’s iterative development methodology. Combined with the resource issues, it has placed more burden on the BAs to both write scenarios and execute those scenarios, which may place some limitations on the overall effectiveness of the testing process. All of these issues have been identrified in previous IV&V reports, so it is imperative that Fast, DoTAX, and ETS work together to ensure understanding of Fast’s testing approach, with the objective of finalizing agreement on expectations for scenario development, testing execution, and documentation of test results. Training: As with testing, R4 training planning is just getting underway. As indicated in the previous report, the training team took several steps during R3 to address DoTAX concerns that training did not go into enough depth and/or breadth on some topics, that the pace of the classroom training was too fast to enable users to fully absorb the information, and that there was not enough cross functional training. These steps 29 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management included creating additional handouts, and posting training scenarios, curriculum documents, and PowerPoint presentations into the share drive. As with any complex system, training can only go so far to making users productive in the system. It will still take hands-on time working within the system to become fully comfortable. In general, we heard positive feedback regarding training. In terms of possible changes, staff are seeking more business process or “how does it work” type training (e.g. how does the interest calculator work within GenTax), as opposed to training strictly focused on how the system operates. Fast’s training program is typically based largely on a train-the-trainer approach, providing an Expert User Academy to a select group of users who are also expected to provide training to other users. For TSM, DoTAX decided that the expert users would not be required to perform classroom training, but could do so on a voluntary basis. Only a small percentage of the expert users opted to participate, so Fast had to perform a higher level of direct training than planned. This reduced participation by DoTAX should be taken into consideration in planning the training for R4 and R5. In addition, the expert user model does not seem to be taking hold as “Experts” do not feel like they are experts. Fast has requested that the individuals selected as Expert Users for R4 be made available for a longer, more intensive training program, but this commitment has not yet been confirmed by DoTAX. AdvanTech strongly recommends that DoTAX create a Training Coordinator role within their organization to help support ongoing business process and system training into the future, and to maintain the online help function post-implementation. Technical Knowledge Transfer: DoTAX has a very aggressive goal to be fully self-maintaining at the end of the warranty period. There is significant training that must occur for the DoTAX technical IT staff to be able to support the TSM system going forward. This includes training (both classroom and on-the-job) for system maintenance, production support, and operations processes for GenTax and the other hardware, software, database, and infrastructure related components of the TSM Program. As noted in the High Priority Risks above, AdvanTech remains concerned that this goal is unrealistic. Very few states, if any, that have implemented GenTax have achieved complete self-maintenance. This concern has been further magnified by the apparent decision by ITSO to reduce the involvement of ITSO staff in program activities geared to achieving the best level of knowledge transfer until the warranty period. Not only will that decision ensure that DoTAX will not be able to fully maintain the TSM system after the warranty period, but will make it necessary to have an even higher level of vendor involvement in ongoing maintenance and support activities. 30 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Up to the point where TSM leadership was changed, the knowledge transfer plan included a progress tracking model to assist with determining how far each technical person has progressed in gaining the necessary knowledge to perform their specific maintenance responsibilities post-implementation. Fast and ITSO were meeting regularly to discuss KT progress. Some of the ITSO development staff that had been assigned to TSM made impressive progress in gaining ability to perform GenTax production support, taking on a solid share of the fixes and enhancements identified in SQRs. Those activities have been put on hold. Training has been provided on some of the non-GenTax components of the overall TSM system. including the storage area network (SAN), scanning, data capture, and IVR hardware and software. However, even though ITSO staff have performed some upgrade work in these areas, ITSO management has been reluctant to allow that work to be put into production, expressing concerns that it could have a negative impact on system warranties. Fast has indicated that will not be the case. Documentation: DoTAX and Fast continue to work to resolve differences in expectations over documentation. A refined process for the development of documentation was instituted early in the R3 development cycle. The changes included defining new formats for recording business requirements, system configuration, and business process changes, with the DoTAX PMO assuming a larger role in coordinating the preparation of that documentation. However, there continues to be concern by both DoTAX and ETS that the documentation of configuration decisions is not always handled consistently (e.g. some are only documented in meeting minutes) and is not easily traceable. For some time, DoTAX has expressed a desire for a system overview document showing how all the components fit together, and for system operations documentation. One of the main components of documentation is the online help function within GenTax. While this help function is fairly robust and is context sensitive, DoTAX feels in some cases it does not provide enough depth related to the specific topic, nor does it provide an overlay of how the particular system action being described fits into the broader picture of the business function that that action falls within. On the other hand, both DoTAX and Fast agree that the online help function is difficult to maintain and keep current. Fast is concerned that DoTAX has not yet appointed someone to learn and assume responsibility for maintaining the online help function, particularly in light of the significant number of site-specific online help elements. 31 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Business Process Reengineering (BPR) and Organizational Change Management (OCM): During the early stages of the program, there were differing expectations between the DoTAX and Fast regarding the methodology and responsibility for business process reengineering. DoTAX felt that Fast should take the lead and use their experience from other GenTax implementations to guide DoTAX to best practices. Fast felt that DoTAX needed to be more proactive in identifying areas where they believe they would benefit from change, and that the DoTAX business units continue to be somewhat more inclined to retaining current practices (“we have always done it that way”). The parties have agreed that the TSM Program must have a plan and process for looking for opportunities to make business process changes proactively. As noted in the High Priority Risks above, Fast has assigned two qualified individuals to work part time with the TSM Program Office and DoTAX to develop plans for both these areas. They have performed surveys and developed workshops to engage DoTAX management in the BPR and OCM process. This process is continuing to move forward, with further meetings planned for November. As noted in the previous assessment, some documentation of business process changes that occurred with R2 and R3 are embedded in training materials and the online help function. However, we have not yet seen that detailed business process manuals have been fully developed, either by updating existing manuals within DoTAX business units for changes occurring due to TSM, nor by creating TSM specific manuals. DoTAX is continuing with its program of communications, both internally and externally. The internal communications have included interactive briefings with managers from all areas, including the neighboring islands, and “Town Hall” sessions to help set expectations and answer questions of all staff. The external communications include news releases, multiple mailings, website announcements (including FAQs), brochures for both tax preparers and legislators, and a web notification to existing eServices users about changes to eServices. Involvement in TSM by both the DoTAX and ETS public information officers has enhanced these efforts. Overall buy-in within DoTAX for the TSM Program is still not where it should be. Some business unit leaders and mid-level managers have taken a greater level of ownership of the program, and have done a better job of acting as champions of the effort by conveying their full support to their staff. However, this is not the case across the board. Some middle managers do not seem fully bought in, and are not using the system, not acting as champions, and are seemingly disconnected from the process. As indicated in the High Priority Issues above, there continues to be pockets of resistance, and even some obstruction to the program. 32 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Recommendations Throughout the TSM IV&V process, AdvanTech has been detailing specific recommendations and tracking their status. The following provides a compiled listing of all of those recommendations and their current status: Previous Recommendations 3.1 Front End Processing DoTAX, Fast and the subcontractors need to finalize the plan related to the use of zone reads and global anchors, then complete and test any programming changes required. It is appropriate to hold off until the year-end changes are completed. Previously Closed (see Assessment 5) 3.2 Front End Processing DoTAX should make it a high priority to release new forms providing for the use of the new 12-digit account numbers. Previously Closed (see Assessment 5) 3.3 Front End Processing DoTAX and Fast should work together to resolve the concerns over two vs. three scanners. DoTAX has decided not to pursue a third scanner in Oahu, but to purchase to additional regional scanners. This procurement is in process. This recommendation is being closed. 3.4 Front End Processing Data cubes, reports and/or other tools should be developed to aid with the reconciliation of daily deposits. Previously Closed (see Assessment 6) 3.5 Front End Processing DoTAX needs to decide which processing functions will be performed in the district offices, then establish the appropriate procedures and provide necessary training. Previously Closed (see Assessment 6) 33 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 3.6 AdvanTech , LLC Innovative Technology Management Front End Processing Complete the analysis of alternatives for resolving the mail opening issues, including review of other possible equipment solutions. As part of the analysis, provide statistics on the various sizes of incoming mail and their respective volumes. DoTAX has decided not to utilize the mail opening equipment provided as part of TSM, as it would only handle approximately 25% of the incoming mail due to size constraints, thus requiring additional pre-handling. DoTAX is in the process of adding features to the existing Pitney Bowes equipment to improve overall mail handling. This recommendation is being closed. 3.7 eServices Continue the process for analyzing and refining security controls for registering accounts on eServices. The issues with online registration have been resolved through improvements to the configuration of the online component. There were no significant issues in this area during the R3 cutover. However, R4 will see a dramatic increase in the number of online users, so this recommendation will remain open until then. 3.8 eServices Send a follow-up mailing to existing eHawaii.gov accounts as a reminder of the December 31 deadline for moving from eHawaii.gov to eServices. Previously Closed (see Assessment 6) 3.9 eServices Designate an individual within DoTAX to act as champion for eServices to work with the DoTAX organization and the TSM program office to help create an overall vision for eServices. As noted in the previous assessment. Rose Salvascione has assumed the role of eServices champion, but that it still is unclear whether that is an official designation, or an informal, de facto role based on her other job functions related to supporting online processing. We continue to recommend that DoTAX clarify who has primary responsibility for guiding the continued development of online services and the efforts to push for greater taxpayer utilization of those services. 34 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 3.10 AdvanTech , LLC Innovative Technology Management System Configuration Revisit Department Change #12 to ascertain if the option selected for audit payment offsets is the most optimal overall, or if there is another alternative that resolves the issue for the audits with previous refunds. Previously Closed (see Assessment 5) 3.11 System Configuration DoTAX should work with Fast to determine the best practices solution for the 13th period issue. DoTAX should be willing to evaluate possible changes to policy or procedure. For example, some states with annual reconciliation returns have eliminated those returns by changing the nature of their monthly or quarterly return processes. Previously Closed (see Assessment 6) 3.12 Data Conversion In future rollouts, ensure that the business units fully understand the level of data cleansing and the nature of any data re-formatting that will occur as part of the conversion process. Also ensure that the training program includes sufficient focus on how to use converted data in GenTax. For R3, the conversion team prepared more extensive documentation on how data elements are being converted, and the training team provided users with more in-depth information on status and use of converted data as part of Tier 3 training. We were not informed of any issues related to the R3 converted data or its use post-cutover. This recommendation is being closed, b ut the conversion and training teams should ensure that the same emphasis should be placed on communicating conversion plans and issues for R4. 3.13 Access Security Provide better communications to users regarding their security assignments, including a clear picture of what access rights they have been granted. Previously Closed (see Assessment 5) 3.14 Access Security Ensure that the security model and assignment of security roles is clearly documented in preparation for the upcoming system security audit. Previously Closed (see Assessment 5) 35 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 3.15 AdvanTech , LLC Innovative Technology Management User and Production Support Finalize the plans for and implementation of the ticketing system, and complete the transfer of TSM help line functions to the DoTAX help desk, with Fast continuing to provide support to the help desk on a schedule that coincides with the DoTAX work schedule. Previously Closed (see Assessment 5) 3.16 User and Production Support Consider releasing outgoing correspondence, particularly the SOFTs, in a phased manner to better control call volumes. DoTAX is issuing approximately 18,000 SOFTs to R2 and R3 taxpayers on a monthly basis with no apparent issues. SOFTs were not issued in August to allow the R3 cutover to get bedded down. This recommendation is being closed, though the method and timing for issuing SOFTs to the very large number of R4 taxpayers should be carefully considered as part of R4 cutover planning. 3.17 User and Production Support Adjust the configuration in the IVR system to enable the planned level of callers on hold, and analyze whether there needs to be an increase in call center staffing, either temporarily or permanently. As noted in previous assessments, the call center staffing has been increased and the automated call system has been refined to provide more selective routing to aid with the taxpayer support function. Actions are underway to activate the IVR’s call back feature, allowing callers not to have to wait online, and to further refine the calling queue. Discussions are being held with Fast about the potential benefits and timing of a broader re-programming of the system as part of R4 to deal with the significant jump in taxpayers using the GenTax functionality when Individual Income Tax is brought online, but there is still a risk that this increase in new users may generate more activity than the call center can handle. DoTAX continues to investigate options to expand call center capacity during peak calling periods through outsourcing, or using other DoTAX staff, temps and/or interns. 3.18 User and Production Support The call center should provide feedback to the TSM support function regarding the types and volumes of help requests received. 36 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management As noted in in the previous report, the call center is providing more feedback with an upgraded set of wrap-up codes to provide more meaningful data on call natures. This recommendation is being closed. 3.19 User and Production Support All SQRs should be given the appropriate priority designation, even if they are being delayed. This will ensure that the proper level of attention is still given to high priority items. As indicated in the body of the report, the process for prioritizing SQRs does not appear to be functioning as effectively as it should. appears to be working reasonably well for the small number of outstanding items. Individual business units generally prioritize their own SQRs, but there is no process for an overall, cross-functional prioritization. A consistent process for prioritizing SQRs on an enterprise-wide basis should be developed, instituted, and clearly communicated. 3.20 User and Production Support Consider having a “Top 15” or “Top 20” list of the most critical SQRs to further enhance the tracking and attention given to those items, and to help manage user expectations regarding priority setting and issue resolution. Previously Closed (see Assessment 5) 3.21 User and Production Support Program management should consider establishing a designated production support team, separate from the other work teams whose primary responsibility is to handle issues with components of TSM currently in production. This team can start small and tap into the other work teams to help resolve issues, then grow into a full-fledged production support function over time. Previously Closed (see Assessment 5) 3.22 User and Production Support DoTAX should consider doing an ongoing user readiness survey. This process entails meeting with business unit managers periodically (typically monthly) to ask a set of questions regarding each of their unit’s level of readiness based on where the program is at that given point in time. AdvanTech has provided an example of this type of survey utilized on another GenTax implementation. It is too late to implement for R2, but should be considered for future rollouts. During R3, business unit managers were engaged more regularly on production support and requirements definition activities, including approval of business requirements/configuration decisions. This appears to be continuing for R4. In 37 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management addition, the TSM PMO recently conducted a written survey sent to more than 400 DoTAX staff (264 responses were received), requesting input on nine questions related to the TSM system and program activities. However, we still do not see any specific process, other than training satisfaction surveys, for ascertaining user progress on gaining understanding and improving productivity with system elements currently in production, nor with the users’ readiness for R4 and future cutovers. We continue to recommend that a process be formalized as part of overall organizational change management. 3.23 Infrastructure DoTAX and Fast should work together to prepare an overall business interruption and disaster recovery plan, to include the installation of back-up infrastructure at a separate disaster recovery site to allow for failover capability. This back-up infrastructure could also potentially be used to handle certain production functions, such as the reporting database, to enhance performance of the main production environment. The DoTAX business interruption and disaster recovery plans remain a work-inprogress. DoTAX is using its existing hardware at DoTAX as a manual failover back-up and recovery site. Plans to move the DR site to the University of Hawaii data center have been put on hold. Fast and DoTAX are working together on a plan to upgrade facilities at the DoTAX building to allow for a fully secure and redundant DR site. 3.24 Infrastructure If not already done, Fast should clearly define the monitoring capabilities, both automated and manual, for all components of the system, including those components provided by subcontractors. Previously Closed (see Assessment 6) 3.25 Program Management We suggest that at a minimum the DoTAX BAs hold periodic joint meetings to discuss cross-team activities and issues. Fast team leads are meeting weekly to discuss cross-team questions and issues. The primary DoTAX BAs meet informally on a fairly regular basis. According to the Departmental BAs, they are not meeting as a group to discuss cross-team activities. AdvanTech continues to recommend that the Departmental BAs also schedule a regular meeting. 38 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 3.26 AdvanTech , LLC Innovative Technology Management Program Management DoTAX should consider refining the decision-making process to clearly indicate what decisions program team members are empowered to make, and which must be passed to the business units. As per the discussion on program governance in the body of the report, it is critical that the decision-making process be clearly defined and communicated. With the new TSM management structure, high priority must be given to reviewing, revising, and communicating the decision process and other related program governance procedures. 3.27 Program Management Fast should make a priority effort to complete revisions to the draft SOW so that it can be completed as part of the R3 planning and kick-off effort. Previously Closed (see Assessment 5) 3.28 Program Management Finalize the lessons learned document and provide it to key program team members (PMO, Fast Management and Team Leads, DoTAX BAs) and business unit management to be used as an R3 planning tool. Previously Closed (see Assessment 5) 3.29 Testing The Fast and DoTAX testing leads should sit down with the PMO and BAs to discuss the definition, purpose, composition, and process of the various testing pieces. The goal of this session should be to obtain DoTAX agreement that the testing program is comprehensive enough to meet DoTAX needs. As noted in the previous assessment, the DoTAX Sr. Quality Assurance Analyst reviewed this information with the PMO. The Fast testing lead reviewed the information with the BAs, SMEs, and testers during tester training. The process of having users develop test scenarios also helps ensure that the testing program is comprehensive. ETS has recently expressed some concern about the thoroughness of the testing process. AdvanTech recommends that the PMO Sr. Quality Assurance Analyst and Fast Testing Lead review the R3 test plans, processes, scenarios, and results to ensure that there is a clear understanding. 3.30 Testing DoTAX should ensure that as many R2 testers as possible be engaged in R3 scenario writing and testing. 39 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management For R3, there were significant challenges in getting resources from the business units to provide a sufficient number of testers on a timely basis that had a strong working knowledge of Hawaii tax policies and operations, and had familiarity with the elements of TSM already deployed. We are reconfirming our recommendation that the DoTAX business units provide qualified testers in sufficient numbers for R4 that have participated in the testing process in previous rollouts. 3.31 Training The program team should make sure the users know that the recorded training sessions and webinars available online. Previously Closed (see Assessment 5) 3.32 Training DoTAX and Fast should work together to ensure that DoTAX expert users are fully aware and committed to their responsibility to provide classroom training, and that the expert user training program includes instruction on how to provide this training. Fast’s training program is typically based largely on a train-the-trainer approach, providing an Expert User Academy to a select group of users who are also expected to provide training to other users. For TSM, DoTAX decided that the expert users would not be required to perform classroom training, but could do so on a voluntary basis. Only a small percentage of the expert users opted to participate, so Fast had to perform a higher level of direct training than planned. This reduced participation by DoTAX should be taken into consideration in planning the training for R4 and R5. In addition, the expert user model does not seem to be taking hold as “Experts” do not feel like they are experts. Fast has requested that the individuals selected as Expert Users for R4 be made available for a longer, more intensive training program, but this commitment has not yet been confirmed by DoTAX. This recommendation will remain open until the next assessment to determine if there are any remaining concerns about the training process. 3.33 Technical Knowledge Transfer Fast should work with DoTAX to refine the knowledge transfer plan to fully incorporate non-GenTax components and DBA training. Some technical training has been provided for the non-GenTax pieces, and DoTAX technical staff have gained good experience by working system issues related to those components. ITSO has shown some hesitancy to assume responsibility for some of this maintenance as they are concerned it may impact 40 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management the warranty related to those system components. This hesitancy has unnecessarily delayed the implementation of some system improvements, so AdvanTech recommends Fast work with DoTAX (ITSO) and ETS clarify any warranty concerns. 3.34 Documentation DoTAX should re-emphasize to affected staff the importance of timely review of program documents. Previously Closed (see Assessment 5) 3.35 Documentation Fast should continue to work with DoTAX to refine the composition of the online help. They should consider including links to such elements as functional guides and process flow documents to enable users to access the broader context of each GenTax function. The online help now includes over 500 site specific elements, but linking to process flows and functional guides has not yet been completed. There is a process for continuing to upgrade and update the function, and tie it to specific DoTAX business processes where feasible. However, adding more site-specific content will make it even more difficult to maintain. DoTAX has not yet identified any specific individuals to work with the Fast training team to learn the process for maintaining online help. There is a significant amount of knowledge transfer necessary to gain sufficient experience to maintain the online help. 3.36 BPR & OCM DoTAX and Fast management teams should meet to resolve the differences in expectations related to responsibilities for business process change prior to commencement of full R3 implementation activities. The parties are working on agreement on responsibilities for BPR and OCM activities. For the past several months, Fast has assigned two qualified individuals to work part time with the TSM Program Office and DoTAX to develop plans for both these areas. They have performed surveys and developed workshops to engage DoTAX management in the BPR and OCM process. This process is continuing to move forward, with further meetings planned for November, but will take time to effect any key changes. 41 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 3.37 AdvanTech , LLC Innovative Technology Management BPR & OCM DoTAX should work with Fast to finalize the approach for performing the organizational readiness assessment. The results of that assessment should be used to prepare an organizational change management plan and program. The OCM planning process did not specifically include a process for confirming user readiness for R3. We recommend that the OCM plan incorporate an approach for confirming that readiness for R4 and R5. AdvanTech has provided DoTAX with a sample template for performing ongoing user readiness assessments. 3.38 BPR & OCM Business unit leaders and mid-level managers should be strongly encouraged to become champions of the TSM Program, recognizing that it is the primary tool for the future of DoTAX, and that they are the principal drivers of acceptance of the system by their users. This area still seems to have critical deficiencies. Middle management does not seem fully bought in, with some managers not using the system, not acting as champions, and seemingly disconnected from the process. Some of these managers seem to be waiting for things to happen for them (or maybe to them), rather than being proactive drivers. Engaging them in the BPR program appears to have stimulated their engagement somewhat. However, continued uncertainty about the overall governance and leadership structure has had a detrimental effect in some areas. It is vitally important that the managers be made fully aware of their roles and responsibilities in the decision making, leadership, and communication processes related to TSM. Any pockets of resistance should be identified and dealt with through training, encouragement, etc. 4.1 Front End Processing Fast and ICS should work with DoTAX to develop an end-to-end process flow and reconciliation step diagram for Taxpayer Services and the scanning operations. These diagrams have not yet been completed, so this recommendation remains open. 4.2 Front End Processing Provide additional training to the Maui office in the scanning, check processing, and correction of rejected and orphan batches. Consider sending Brad Conradson from the Hilo office to provide the training. Previously Closed (see Assessment 5) 42 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 4.3 AdvanTech , LLC Innovative Technology Management eServices Ensure that eServices functionality is delivered earlier for R3 and future rollouts to allow for comprehensive testing in order to minimize problems. As mentioned in the last assessment, Fast took several steps to ensure that the online experience was smoother for the taxpayers. Additional demonstration sessions were held with DoTAX to review progress during the R3 development process. In addition to the normal internal testing process, 200 tax practitioners and 100 taxpayers known to be vocal users were requested to “play” in the new system, using their own data to try to register, file, and pay. R4 will see a dramatic increase in the number of online users, so this recommendation will remain open until then. The registration process for Individual Income taxpayers should be thoroughly tested before R4 cutover, sufficient online help aides should be provided, and call center capacity should be adjusted as necessary to deal with the increase in online use. TSM should also review the feedback from the outside testing for R3, and should consider repeating that outside testing for R4, but try to have it done a bit earlier in the process to allow more time to respond to the feedback. 4.4 eServices Priority should be given to making the decision regarding the use of eServices or MeF for Corporate Income Tax as soon as possible so that Fast has clear direction on how to proceed with R3 configuration. The decision should take into consideration the impact on taxpayers of each alternative, with ample development time given to ensure that taxpayers see minimal problems upon cutover. Previously Closed (see Assessment 5) 4.5 eServices Designate an individual within DoTAX to act as champion for MeF to work with the DoTAX organization and the TSM program office to develop an effective plan for implementing MeF for Corporate and Individual Income Taxes, regardless of whether eServices is implemented on an interim basis. The roles of eServices champion and MeF champion may have been combined under one individual (Rose Salvascione), but as indicated above, it was unclear if this was officially the case. We recommend that DoTAX clarify who has primary responsibility for guiding the development of the MeF program for Corporate Income Tax, especially since it will be rolled out separately in January 2018. 43 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 4.6 AdvanTech , LLC Innovative Technology Management Access Security DoTAX should appoint a security coordinator to work with Fast’s security lead to learn how to maintain all elements of the access security function. Previously Closed (see Assessment 6) 4.7 User and Production Support Fast and DoTAX need to make sure cutover plans, particularly those related to eServices, take into consideration the impact on call center volumes. Call center operations were adjusted to some degree based on lessons learned from R2 and cutover planning for R3. As noted in Recommendation 3.17 above, DoTAX continues to investigate options to expand call center capacity during peak calling periods through outsourcing, or using other DoTAX staff, temps and/or interns. R4 will see a dramatic increase in the number of online users, so R4 cutover planning should dovetail the planning for further call center refinements. 4.8 User and Production Support DoTAX should consider using interns to help man special eServices hotlines and help walk-in customers get registered for eServices, especially during peak periods such as the annual filing deadline. As indicated in 3.17 above, DoTAX continues to analyze call center operations, including the configuration of the IVR system and call flows, to determine what additional adjustments (system, staffing and training) may be required. 4.9 User and Production Support The production support team should consider additional methods for communicating the status of SQRs, such as periodic bulletins informing users of items put into production, combined, deferred, etc. The production support team continues to conduct regular meetings with the business units to discuss SQR status, with the frequency of those meetings based on the level of SQR (more often for a period following a cutover). AdvanTech still recommends that the team push out SQR status information to all users, particularly when fixes are put into production. 4.10 User and Production Support Fast and DoTAX should work together to reach out to some users to better determine the nature of usability issues discussed above, and find ways to resolve them (user bulletins, additional training, etc.). 44 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 AdvanTech , LLC Innovative Technology Management Some of this activity has occurred, particularly with the front-end processing area. However, we still believe this process needs to occur in other areas, particularly where users are having more difficulty coming up the learning curve (e.g. Collections). We recommend that this activity be given a high priority to help ease some of the existing tensions related to TSM and foster additional buy-in for the program. This activity would also help achieve a higher level of user readiness for R4 (see 3.22 above). 4.10 Training DoTAX business unit managers should work with their staffs to make sure users complete the Tier 1 training program. Previously Closed (see Assessment 5) 4.11 Training DoTAX should appoint training coordinator to work hand-in-hand with Fast’s training lead. Joshua Lee previously acted as the DoTAX training coordinator as one element of his PMO role. He is no longer working on TSM, and DoTAX has not yet identified a replacement to act in the role for R4 nor created a full-time trainer position as recommended by Fast. There is a significant amount of knowledge transfer necessary to gain sufficient experience to coordinate ongoing training and maintain the online help. 4.12 Technical Knowledge Transfer Fast should provide a systems overview document and system operations manuals/documentation. These documents have not yet been provided by Fast. Fast and DoTAX need to discuss further to reach agreement on the nature of these documents. In the meantime, Fast has prepared a set of functional guides for each component of GenTax, which will be living documents that are updated to reflect new or changed configuration with each rollout. ETS has recently expressed concerns that the various forms of documentation of system requirements and configuration are not consistent and cohesive. Fast, DoTAX, and ETS should work together to finalize agreement on what will constitute acceptable documentation of system design, configuration, and operations. This should be given high priority as concerns about documentation have been an ongoing theme throughout the life of the program. 45 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 4.13 AdvanTech , LLC Innovative Technology Management Documentation Fast team leads and DoTAX PMO members should ensure that all configuration decisions are being fully documented with the new process. There is still concern that the mechanism for documenting configuration decisions is inconsistent and not cohesive. With R4 definition activities well underway, the PMO should work with Fast development team leads and the Primary BAs to ensure a solid, consistent process is in place across all teams. This process should enable easy traceability of configuration decisions to asbuilt configuration. This activity should be given a high priority. 5.1 System Configuration DoTAX should work with Fast to complete the analysis related to merging Sellers Collection into GET, including whether the DoTAX developers could take the lead in the effort. This analysis should be given high priority if there is a desire to try to accomplish the change as part of R3. It may be more effective and more reasonable to plan the change for the beginning of the calendar year to better align with filing cycles, and to allow for sufficient time to plan and execute the change. Previously Closed (see Assessment 6) 5.2 User and Production Support The production support team should review the process for classifying an SQR as rejected to ensure that it is using the proper terminology in a given case. For SQRs that are rejected, the team should provide a clear indication of the reason(s) for rejection and communicate those to the originator of the issue. As indicated in the previous assessment, the support team is no longer classifying items as rejected, but is working with originators to make adjustments to the SQR to ensure they clearly indicate the nature of the issue. Several SQRs have also been now categorized as change requests, reflecting that the item is a desired change, not a system fix. This recommendation is being closed. 5.3 Infrastructure Fast should finalize a plan and get commitment from Dell for providing a resolution to the problem with the network switching equipment. If Dell does not provide a solid fix or replacement, then Fast should consider replacing the Dell equipment with another alternative. Previously Closed (see Assessment 6) 46 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 5.4 AdvanTech , LLC Innovative Technology Management Program Management The PMO should reiterate to the Departmental BAs that part of their role is to act as liaisons with their respective business units to communicate to and from the units and the TSM Program office. While we believe that the Departmental BAs have a general understanding that the liaison function is part of their role, we are still concerned that they do not feel empowered in that regard. As a group, they did appear somewhat more willing to speak freely about issues than during the previous assessment. They still appear to have a sense of uncertainty related to the change in management structure and program ownership. 6.1 eServices DoTAX should work with Fast prepare a detailed plan and schedule for the MeF implementation, which should be managed in a manner similar to any other rollout. AdvanTech saw elements of a plan for MeF within the overall project plan, but did not find a detailed plan specifically for the MeF implementation. 6.2 Program Management An Executive Steering Committee should be formed with key representatives from DoTAX, ETS, and TSM. DoTAX and ETS have prepared a TSM Executive Steering Committee (ESC) Charter which defines the membership and responsibilities of the ESC. The ESC will begin functioning as a body in the near future. This recommendation is being closed. New Recommendations 7.1 Program Management Ensure that PMO Program Manager and TSM Project Manager are able to commit the necessary time (Ideally as close as possible to full time) to the program. 7.2 Program Management Make final decisions on the configuration and staffing of the PMO, and clearly define the roles and responsibilities of the PMO as a body and of individual member. 47 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 7.3 AdvanTech , LLC Innovative Technology Management Program Management Clearly define and communicate the process, roles, and responsibilities for decision making and escalation of issues. If not resolved effectively, the program is at risk of entrenching factions who may work at cross-purposes. DoTAX has already experienced staff movements off of the project and grievances that purportedly relate to competing views of program management. 7.4 Program Management Clearly define the communication channels between TSM management and DoTAX management, including how the DoTAX Deputy Director will keep the Director informed of TSM progress and issues.. 7.5 Program Management Clearly define the communication channels and responsibilities between the TSM Program Office, DoTAX business units, ITSO, and ETS. 7.6 Program Management Review the charter and make-up of the TSM Stakeholders Group to ensure that all key user groups are represented, including DoTAX business units, ITSO, ETS, and Fast. The charter of the group should be clearly defined to specify its purpose and role in providing guidance and decision making for TSM. 7.7 Program Management Ensure that ITSO management and staff are fully supportive of the TSM Program, and committed to both necessary legacy data conversion and TSM knowledge transfer activities. 7.8 Program Management Conduct a meeting between key DoTAX and ETS managers, possibly utilizing a third-party mediator, to resolve apparent personality conflicts and develop a sense on internal partnership. 7.9 Program Management Evaluate the benefits, impacts, and risks of reducing the scope of R4 to solely Individual Income Tax, moving the other taxes planned for R4 into R5. 7.10 Program Management Perform a reasonableness review of the RFP requirements, and process change orders where appropriate. 48 Hawaii DoTAX TSM Program – IV&V Year 2 Assessment 3 7.11 AdvanTech , LLC Innovative Technology Management Testing Fast, DoTAX, and ETS should work together to ensure understanding of Fast’s testing approach, with the objective of finalizing agreement on expectations for scenario development, testing execution, and documentation of test results. 7.12 Technical Knowledge Transfer Evaluate the current status of knowledge transfer plans and activities, and develop a back-up plan for performing post-warranty support services in the likely event DoTAX will be unable to fully maintain the system using its own personnel. 49