TRANSPORT

AGENCY
WAKAKOTAHI

To
From
Date
Subject:

~

n

•

S9(2)(a)

19

82

26June 2018
NIEMS Production Deployment

AC

T

Purpose

M
AT

IO

N

This paper documents the production deployment for NIEMS on Friday 8 June 2018 and its
subsequent roll back. It summarises the actions that were completed and identifies lessons learned.

R

Background

IN

FO

It is a 17/18 SPE milestone to launch the National Incident and Event Management System in
Wellington and Christchurch.

L

Project Progress 17/18

E

O

FF
I

C
IA

Following a project reset, a high level roadmap out to 30 June 2018 was developed with the TOCs
and agreed by the NIEMS PSG in Nov 2017 then endorsed by the TGG in Dec 2017. See Appendix 1.
This roadmap was designed to deliver a first iteration towards the provision of consistent processes,
capability and tools to manage incidents and planned events nationally and is considered the first
feature set to be delivered under the transport Operating System.

ER

TH

A more detailed roadmap was refined early April 2018 in conjunction with the TOCs and shared with
the NIEMS PSG. This was further refined in May 2018. See Appendix 2 and 3.

U
N

D

Progress against the roadmap was reported on through the monthly Status Report to the NIEMS PSG
and the OS portfolio report to TGG as well as in quarterly SLT reporting.

AS
E

D

The project has been tracking to deliver against the 17/18 SPE milestone up until May 2018 where
recurring delays through the change process increased the risk of delivery timelines being impacted .
Production Deployment Planning

R

EL
E

As part of preparing to move ILS production to the cloud, a security assurance report on the Google
cloud platform was completed and reviewed by the NIEMS PSG in Feb 2018.
Go live dates were discussed and agreed with WTOC Manager. This date was delayed several times
due to a combination of Fujitsu readiness for servers and extreme weather. The programme was
initially targeting a go live date of 10 May 2018 and finally firmed up as 8 June 2018 for deployment.
The TOC Operational Checklist Acceptance Criteria template was completed to prepare for
deployment of production into the cloud. This includes sign off for relevant functions as follows:
•

Design

 AC

T

19

82

• Security
• Privacy
• Test Planning and Reporting
• Early Life Support
• Support Operate Phase
Release Notes
•
•
Disaster Recovery
•
Business Sign Off
• Project Management
Training was not required as this release did not change the operator functionality. It was agreed for
the Government Chief Digital Office (GCDO) cloud assessment process for the Google Cloud Platform
to be progressed in parallel to this process.

IO

N

The NZTA Fujitsu Change Management Request for Change Form was submitted to Fujitsu from
which a Fujitsu work request was generated and included in their weekly change plan.

FO

R

M
AT

The approval to migrate the ILS database and enable ILS production to be deployed in the cloud was
given as an exception to the historical change process but supported by the completion of these
standard change forms.

IN

Deployment

C
IA

L

Friday 8 Ju ne 1-5 AM

FF
I

The Project Manager, 2 Developers and the sh ift Team Leader completed the change in conjunction
with the Fujitsu DBA on shift.

O

An impleme ntation test exit report was completed documenting the process. See Appendix 4.

ER

TH

E

The process was completed at 4:30 AM and ema ii notification advising the business of the successful
data migration was sent at 4.42 AM

D

Post Deployment

D

U
N

Four service desk requests were logged mid-late afternoon on 8 June after the deployment. These
related to user login issues and duplicate or triplicate log entries.

R

EL
E

AS
E

The user login issues related to the ITS active directory and were all resolved by 4:30 PM 8 June.
Duplicate or triplicate log entries were occurring intermittent ly when populated from TRIES into
NIEMS. The duplication issue was confirmed resolved by 7:00 PM 8 June.

Further intermittent problems presented as saving issues over the course of the weekend and a new
support ticket was raised Sunday 19:30 PM 10 June. The ticket was mis-categorised by Fujitsu as a
P3, i.e. fix in business hours. This was not escalated to a P2 until Monday at 8:00 AM 11 June after a
discussion with Fujitsu.
Five more issues were raised on 11 June for problems presenting as:
• Details not saving or saving in the wrong event
• Event initiator unable to reopen event in TREIS
• Overall system slowness

 These have been categorised as symptoms of the same underlying problem.
Early analysis suggested that the problems may have been due to a database configuration setting.
As these are adjustable, some changes were made to try to resolve the issue. The problems were
intermittent in their nature and configuration changes did not resolve the underlying problem.

19

82

As part of the problem investigation the developer tried reducing the number of servers but this
caused the SCATS team to lose access to ILS for a time. With a severe weather warning in place for
the next 24 hours, alternative options were considered to resolve the issues.

AC

T

The decision was taken at 3:00 PM, in conjunction with WTOC to implement a temporary roll back
and allow time to find a solution to the problem, whilst providing a stable solution for the operators.
Open tickets were manually re-entered back into the old ILS.

M
AT

IO

N

Problem Resolution

R

The team completed an investigation into the user problems to identify and resolve the underlying
cause of poor response times, page submit errors and system performance.

C
IA

L

IN

FO

The cause was determined to be from the original ILS code using a database access library
(Hibernate) at one version and the newer NIEMS release using a later version of the library which
had some subtle changes in the management of connections to the database. As a result when
usage reached a certain volume level connections would start to become unavailable leading to the
intermittent fau Its.

E

O

FF
I

On Monday 11 June, the team were able to reproduce the errors, first on the staging server and
later on local development systems. Code changes were made to resolve the problem. Performance
load and stress tests were run and passed. Performance tests for the cloud deployment were also
reproduced locally at WTOC and passed.

TH

The following changes have been made to NIEMS:

ER

Chan e

AS
E

D

U
N

D

Update the ILS code to con-ectly use the database
libr~ry
Improve the efficiency of the old ILS code by
optimising access to the database to reduce the
n~ber of re<!undantc~ll~ in.!lonniil.usage
9P!imise ~o.~e of ~h~.11ev-. ~E.MS c~e.
Increase database instance size

R

EL
E

Improve ILS notification mechanism so that they
are regional based
I Create ii.feter.pedo~nance test

Resolve the issue of database connection pools
being exhausted quickly
Reduces overall stress on the system and will
improve overall response times and server load
R_educe query times
. ..
.
Increase the maximum number of connections to
database
Events updated in Auckland do not impact on the
server for WeUington users
I St;·ess
;eino~ ~hat the upper limit of
the system's perfonnance is__ _

test.so.that

Retrospect ive
A lessons learned meeting was undertaken with the NIEMS project team on Tuesday 12 June 2018 to
understand what happened, why and identify opportunities to improve. The following observations
and recommendations were documented:

 Observation

I Recommendation

Application was tested manually by users at the
time of release and through automated testing
but did not undergo sufficient load testing with
___

..... J Done

A detailed Implementation Plan was don~
severa l weeks in advance but the Fujitsu staff
implementing the plan were unfamiliar with it
at the time of executing the plan

82

"Full Dress Rehearsa l" with all parties involved
prior to D-Day

' ,, ...... -... ,. __ ... ,~ .... _., ___ ,.,, .......... ,______ ,_,,_, ___,_,.,,,.... ··-···········--

T

-+---··· ..

19

mu ltiple users

Automated parallel and high vo lume load
testing with prior qual ity test coverage is
implemented and added to the Definition of

Recommend clear lines of communication,
escalation and accountability be published

Interruptions to staff working on incident
response

Dedicated staff working on the problem should
not be interrupted to give regu lar updates.
Setup an incident management group chat to
manage communications wit hin the team and
nominate a spokesperson to interact with other
stakeholders
No changes be done in Production without
going through stage testing

R

L

IN

Changes directly in production affected live
users

FO

I

M
AT

IO

N

AC

Multiple cross comm unication due to key staff
away sick

The Fujitsu support desk was unclear on the
escalation rules and sign offs for the process
were unclear

FF
I

C
IA

Clear suppo rt process be published and signed
off by all parties prior to D-Day

I Designated staff be on hand with the
I customers for the initial few days - especia lly if
this is over a weekend.

D

ER

TH

E

O

~Anew re lease of software with fundamental
changes to underlying components was not
bedded down with users for a long enough
period

U
N

Issue Resolution and Performance Testing

AS
E

D

The underlying problem that manifested as poor response times, page submit erro rs and other
problems was investigated, identified and corrected.

R

EL
E

The problem was caused by the origina l ILS code using a database access library (H ibernate) at one
version and the newer NIEMS release using a later version of the library which had some subtle
changes in t he management of connections to the database. As a result when usage reached a
certain volume leve l connections would start to become unava ilable leading to the interm ittent
fau lts.
The errors were reproduced, first on the staging server and later on local development systems first
through manual load test and later throug h an automated test.

 A JMeter performance test was created in order to stress test the system. This test emulates a
number of users constantly using the NI EMS UI and creating incidents. The numbers are increased
until the system starts to show an error or increasing response times.
I -

-

II .

I e e 8

a •

Link to internal NZT A folder

• a I

•

I •

-

I - - I

e . :i.

-1 - I

•

I

I -

&

~

•

•

I

•

•

U

I •

7

•

19

82

Redep1oyment

AC

T

Detailed Planning Update

M
AT

IO

N

A detailed plan to redeploy ILS to the cloud was revised and agreed with the business on Thursday
21 June 2018 with a new target date of Saturday 23rd June 2018 subject to weather, road works and
no major incidents. See 0618.3.0 NIEMS Releases Detailed Plan.
The plan included a higher level of on-site support for the first days following the release.

IN

FO

R

Actions against the revised detailed plan were completed on Friday 22 June, all t ickets were signed
off and a final go-live was agreed to at 6pm 22nd.

L

Redeployment

C
IA

Saturday 23 June 7-11 PM

O

FF
I

The Manager Transport Operating System and Business Analyst were on site in Wellington for the
deployment. Along with the Project Manager, 2 Developers and the shift Team Leader they
completed the change in conjunction with the Fujitsu OBA on shift.

ER

TH

E

An implementation test exit report was completed documenting the process. See Appendix 5. The
process was completed at 11 PM and email notification advising of the successful data migration
was sent.

R

EL
E

AS
E

D

U
N

D

On-site support from the project team was maintained for 48 hours as per the rotating schedule and
observations documented in the Post Release After Care Notes.
No new problems were identified and operator's feedback was that they experienced increased
performance from the new system.

 Appendix 1
Approved High Level NIEMS Roadmap - November 2018

Project roadrnap

National Incident and Event Management System (NIEMS)

Draft NIEMS Road Map
A Cl~H.$

PrlOrity Data

anoapprova,s

,,...,,.le

Comms/

14 .. 0ec
NIEMS

PIOOwtY.ltSflOP

tng•g,• ment/

UMtduf.gn

Change pf•ns

& tffdng - - - - -

tbactmapto
TIGG

Revised

22"' Doc

A.grM 'go liv•'
crheriawlth

•iwl~

JMC.""'"'.9""

SOP's sloc:ktel:e

Pfo,ecl

BaCktn<l
integration
with Ul

19

16"NOII

.1•.:a,e llusln..~

lntegtate dm
teeattol:S8

Secur~tor

fl,source~

Sc,tlold WO<t PIM

Go llwMVP

~ eed

Agr.td rot.. 6
r0$p01'1Sibililllll$

arehftectute

LOckin

&!fl'Ait~

design
fN turM

R(Jalla.s~i.

oennt Toe

perfo,rnanc:e

UI Vtndorln

pi.c.

IOend:tycrttlcar

targ•u(KPl's)

MVP

R

Concept

AS
E

D

U
N

D

ER

TH

E

O

FF
I

C
IA

L

IN

framtw0rk

FO

Otvelop i ne1de0:1

m1inagemen1

EL
E

Flnaltffting

& Sign oft
completed

process

data fOfMYP

R

lmpltlffnt a
c:ont!O\K>VS

ifflp(O'W!fflent

e:se Pfootot

lmpltmon"'d

framewo,1.
Ag....

IO

UI wo r1(&hop

Woritsh0p

M
AT

ls«vice Oe:sign
1

Cumnt:ttm

N

AC

torecasts

T

OraJIJMC l!E

82

1P Nov

Compl«e

Incident

manage:mtnl
plans
~ltment

prbritySOP's

 Appendix 2
Updated NIEMS Roadmap - 6 April 2018

Project roadmap

Nati onal Incident and Event Management System (NIEMS)

,

Draft NIEMS Road Map

.~ -.------------------------------------------------~

Completed

I\J

l

• ,;

19

~ifY"t-1)

•woec,..TOC::1

In Progress
MOU and

croc

llj..t:11.c·llC

ln!egr.al.:
TREIS

.IC'CffllOILS

T

1_('1

Ntf.Mb.f'lh u1
.."O.,.

~croc
-"'"'

AC

•

82

<-

R

M
AT

IO

N

To Do

"'1(1 e
SC.Al I

J~Olt
f t

~

t1Qr1,t11

Jni-eQf'lf.t

VIAZi:.

Vyd~i

Goolllt<MO

ltlUOW,e

M~'Of~S~

•

I

lnl4"91.1•c•

D\l'H

L

IN

l!'!ltQ'fllt
TIM

FO

PostJune

C
IA

Appendix 3

O

FF
I

Updated NIEMS Roadmap - April 2018
National Incident and Event Management System (NIEMS)

E

<->
0

TH

i:

ER

1Cloudl

Jn Progress

......

lli~;J\erw:_.,

I

U

OUa110,,,_,.--,

R

EL
E

AS
E

D

1e91.f• R S

IC2
'1l1"",)

I
,,...,.,

N

• • al

To Do

N!EMS:JI

D

,,,. ,.

1•Rfl'S
-•··\e

i

N IEIII UI OnlQ"'

 Appendix 4

Implementation
Steps:

Details

lmplementer(s)

Date/Time

1

Create full database backup from /LS
SQL Server Database PROD

Fujitsu OBA

08/06/2018
01:01

2

Zi p the backup and encry_P,t e.d.wth <!.. .
password ;The password must be at
ioc ti~racters long and should
\
contains symbols and alphanumeric.

Fujitsu OBA

T

Implementation Test Exit Report - 8 June 2018

Task

~·----~

19

AC

---....-"

IO

,..,.-

Fujitsu OBA

08/06/2018
02:10

FO

R

M
AT

3

08/06/2018
01:15

N

·7

(1easf

82

#

IN

4

5

C
IA

L

Convert SQL Server dump to Postgres
SQL dump

6

(NZTA)

(NZTA)

08/06/2018
02:50

TH

E

08/06/2018
02:55

Apply Postgres dump to NIEMS
production database

08/06/2018
03:00

9

Release/Start NIEMS application prod

08/06/2018
03:05

Task

Details

D

ER

8

AS
E

D

U
N

08/06/2018
02:40

S9(2)(a)

Upload Postgres dump to GCP

7

08/06/2018
02:15

S9(2)(a)

O

FF
I

Make Postgres SQL dump to be
compatible w it h GCP Postgres

Fujitsu OBA

R

EL
E

PostTest /
Verification Plan:

H

lmplementer(s)

Date/Time

 S9(2)(a)

1

08/06/2018
03:05

Application Checks:
Check application is up
and r unning

b)

Check all pages load
successfully

c)

Compare what shown
on the page to what's
in the database

d)

Create a test
event/incident to
ensure it works.

!

•

e)

Run reports

f)

Check Travel Times (to
ensure connectivity to

T

19

82

a)

AC

(High level test
steps to qualify
the change
outcome with
supporting doc
emailed to
Change team if
required; Include
Technical and
User Testing.

TIM)
Login to Corporate

N

g)

I

R

I

FO

correctly.

2

08/06/2018
03:10

IN

Database checks:

Check NIEMS Postgres
DB table have exactly
same number of rows
as ILS SQL Server

Check sequences are
correctly set

FF
I

b)

C
IA

L

a)

M
AT

p

IO

Link to internal N"ZTA folder

d)

ER

TH

E

O

c)

Check Geometry
datatype columns are
correctly populated

Business Test:
WTOC to test NIEMS application as
per the embedded test p lan.

WTOC (NZTA)

08/06/2018
03:10

R

EL
E

AS
E

D

U
N

D

3

Check'TEXT'datatype
columns are correctly
populated

4

Go/No-Go decision

08/06/2018
04:25

5

If GO, notifies W TOC to use NIEMS
application

08/06/2018
04:30

 Appendix 5
Implementation Test Ex it Report - 23 June 2018

Fail

Comments

82

Pass

19

Test
Log in - Chrome, time taken to load, same
or better as current ILS

X

Same timing

2

Log in - Firefox, time taken to load, same or
better as current ILS

X

Same timing

3

Log in - IE, time taken to load, same or
better as current ILS

X

4

Historical data available from February
2018 in all tabs and able to search in
current, recent and history

5

Historical data available from July 2017 in
all t abs and able to search in current, recent
and history

6

TREIS roadworks displayed

7

TREIS roadworks data - description,
comment details, location, TREIS ref#
displayed

M
AT
R

Now works in IE

108 extra events in
the system from
previous rollo ut

X

X

X

Checked 5 events,
all showing correct
informat ion

X

Checked 5 events,
all showing correct
informat ion

ER

TH

E

O

FF
I

C
IA

L

IN

FO

X

IO

N

AC

T

1

TRE IS incidents displayed

9

TREIS incidents data - description, location,
source, start time, notification time,
incident level, TREIS ref# displayed

X

10

Open current fault from the fault tab,
search current fault in this tab

X

11

Open recent fault from the fau lt t ab, search
recent fault in this tab

X

R

EL
E

AS
E

D

U
N

D

8

12

Open fault history from the fault tab,
search w ithin fault history

X

3 extra events in the
previous system
from previous
rollout

 13

Pop up menu works in Fault tab

14

Faults: Dropdown fields are populated and
work, able to se lect all options, able to
change/clear selection

X

15

Open current SCATS from the SCATS tab,
and search for current SCATs event

X

16

Open recent SCATS from the SCATS tab, and
search for recent SCATs event

X

X

AC

T

19

82

-

Open SCATS history from the SCATS tab,,
and search for SCATs event from history

18

Open trave l times tab and display
information

19

Overall speed (faster than current ILS), able
to move between tabs and events with
ease (no delay)

20

Create new closure, able to comp lete all
fields

21

Progress Closure with "Sign Out" button

22

"Sign out" added to timeline timestamp
accurate, timestamp edited

IO

N

17

33 extra events in
the previous system
from previous
rollout

R

Journey times not
displaying but the
signs are off

IN

FO

X

M
AT

X

X

Slight delay in
creating, less than 5
seconds

X

X

"Sign out" timeline timestamp editable

X

Changes to times
not captured in
timeline - same as
ILS testing

24

Progress Closure with "Closure In" button

X

25

"Cl osure In " added to timeline timestamp
accurate

X

R

EL
E

AS
E

D

23

U
N

D

ER

TH

E

O

FF
I

C
IA

L

X

26

Closure time line timestamp editable

X

27

Progress Closure with "Police Advised"
button

X

Changes to times
not captured in
timeline - same as
ILS testing

 "Police Advised" added to timeline
timestamp accurate

X

X

30

Progress Closure with "Reopened" button

X

31

"Reopened" added to timeline timestamp
accurate

X

82

"Police Advised" timeline timestamp edited

19

29

Changes to times
not captured in
timeline - same as
ILS testing

AC

T

28

X

33

Progress Closure with "Off Network" button

X

34

"Off Network" added to t imeline timestamp
accurate

IN

X

36

Create a new incident, able to complete all
fields

X

All sections tested,
map tested, timeline
checked, all buttons
checked all ok - Only
once did the field
self-clear

37

Progress incident using "All Lanes Open"
button

X

"All lanes open" added to timeline
timestamp accurate

X

C
IA

L
X

Changes to t imes
not ca ptured in
timeline - same as
ILS testing

35

D

U
N

D

ER

TH

E

O

FF
I

"Off Network" t imeline timestamp editable

AS
E

38

X

Changes to times
not captured in
timeline - same as
ILS testing

X

Tim ings much better
than previous
version, picked up in
3 seconds

"All lanes open" timeline timestamp
39

editable

R

EL
E

·- -

FO

-

M
AT

"Reopened" timeline timestamp editable

R

32

IO

N

Changes to t imes
not captured in
timeline - same as
ILS testing

40

Progress incident using "Traffic Normal"
button

 "Traffic Normal" added to timeline,
timestamp accurate

X

43

Progress incident using "Close" button

X

44

"Close" added to timeline, timestamp
accurate

X

45

"Close" timeline timestamp editable

X

82

X

AC

T

19

42

"Traffic Normal" t imeline timestamp
editable

Changes to times
not captured in
time line • same as
ILS testing

IO

N

41

FO

R

M
AT

STMS w ill not
popu late but th is
section not used by
Wellington. ATOC
has a list that
popu lates for them
in this

46

Create new roadwork, able to complete all
fie lds

47

Progress roadwork using "Roadworks in "
button

X

48

"Roadworks in" added to timeline
timestamp accurate

X

TH

E

O

FF
I

C
IA

L

IN

X

so

Progress incident using "Off network"
button

D

U
N

"Off network" added to time li ne,
timestamp accurate

X

X

R

EL
E

AS
E

D

51

X

ER

49

"Roadworks in" timeline timestamp
editable

Changes to times
not captured in
time line - same as
ILS testing

52

"Off network" t imeline timesta mp ed itable

X

Changes to times
not captured in
time line - same as
ILS testing

53

Open Detail Report (All Incidents) from the
reports tab

X

Timeline showing
out of sequence

X

Time captures are
showing the time
down to the
millisecond

54

Open Detail Report (Leve l 3 and Above)
from the reports tab

 Open Summary Report (All Incidents) from
the reports tab

X

56

Open Monthly Summary Report (Events)
from the reports tab

X

57

Open Duration Report (Incidents) from the
reports tab

X

58

Open Roadworks Summary from the
reports tab

X

59

Open Closure Summary from the reports
tab

X

60

Open Timings Report (Incidents) from the
reports tab

X

61

Open Incident Duration Report Enhanced
from the reports tab

X

62

Open Summary Report (level 3 and above)
from the reports tab

63

Open Summary Report (Level 2 and above)
from the reports tab

X

64

Open Incident Report (SCATS) from the
reports tab

C
IA

X

65

Open Incident Duration Report Contractor
from the reports tab

66

Open Incident Duration Report New from
the reports tab

67

Open Month Summary Report (Events) By
Region from the reports tab

FO

R

M
AT

IO

N

AC

T

19

82

55

X

X

U
N

D

ER

X

Open General Summary Report from the
reports tab

X

D

68

TH

E

O

FF
I

L

IN

X

Open Fault Summary Report from the
reports tab

X

70

Open Twitter Summary Report from the
reports tab

X

71

Open TAR Summary Report from the
reports tab

X

72

Generate a Detail Report (All Incidents)
using the date range fields

X

R

EL
E

AS
E
69

Order changed but
not badly

 73

Generate a Detail Report (Level 3 and
Above) using the date range fields

X

74

Generate a Summary Report (All Incidents)
using the date range fields

X

75

Generate a Monthly Summary Report
(Events) using the date range fields

X

76

Generate a Duration Report (Incidents)
using the date range fields

X

77

Generate a Roadworks Summary using the
date range fields

X

78

Generate a Closure Summary using the date
range fields

X

79

Generate a Timings Report (Incidents) using
the date range fields

X

80

Generate a Incident Duration Report
Enhanced using the date range fields

81

Generate a Summary Report (Level 3 and
above) using the date range fields

82

Generate a Summary Report (Level 2 and
above) using the date range fields

83

Generate a Incident Report (SCATS) using
the date range fields

84

Generate a Incident Duration Report
Enhanced using the date range fields

85

Generate a Incident Duration Report New
using the date range fields

IO

N

AC

T
X

O

D

ER

TH

E

X

X

X

U
N

Generate a Month Summary Report
(Events) By Region using the date range
fields

AS
E
EL
E
R

M
AT
R

FO
X

FF
I

C
IA

L

IN

X

D

86

19

82

- -•...-;::.:.._

X

87

Generate a General Summary Report using
the date range fields

X

88

Generate a Fault Summary Report using the
date range fields

X

89

Generate a Twitter Summary Report using
the date range fields

X

90

Generate a TAR Summary Report using the
date range fields

X

 92

Manually plot event on map

X

93

Search map using hot key (ALT +L) and plot
on the map, able to change location by
researching using ALT + L

X

94

Change locations using the search function

X

Search now working
in new system

82

X

19

Map opens when I click on map tab

T

91

Map speed

AC

95

Minor lag,
comparable to ILS

N

X

Speed of buttons updating fields in details

X

X

Checked multiple
tabs and multiple
(5+) events and all
populating as it
should

X

Added notes to
three separate
events - working
fine

X

TREIS event created
at 22:57, populated
in ILS at 22:59. no
duplicate reported

X

Same as above.

C
IA

L

97

IN

Clicking on the header in
incidents/roadworks/fau lts brings up the
details in section down below

FO

R

96

M
AT

IO

Much better than
ILS was prior to the
upgrade

FF
I

Make sure the notes when added to the
event do not populate on another event

TH

E

O

98

No duplicate TREIS events populating in ILS

ER

99

Dummy event entered into TREIS and
populates into ILS system

R

EL
E

AS
E

D

U
N

D

100