Google/NearForm/HSE 1)Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 18:32 Sent Items To: Yul Kwon Ronald Ho Michael Stillwell Colm Harte Ben Cloney; You replied on 08/08/2020 18:53. Hi all Major issues being reported as COVID Tracker eating battery, we've had no app updates. From what we've seen reported it looks like it's Google Play Services possibly on Samsung only but we haven't validated this. See below thread on Reddit. https://www.reddit.com/r/ireland/comments/i5wo9v/covid app acting strange/?utm me dium=android app&utm source=share Slán Gar 1 2)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 18:53 Sent Items To: Yul Kwon Ronald Ho Michael Stillwell Colm Harte Cc: Cian O'Maidin Ben Cloney; Fran Thompson; see below reports of COVID Tracker/Google Play Services battery consumption issues. People are having to uninstall app, we need to get an update on this ASAP slán Gar 2 3)From: Gar T. MacCriosta Sent: 08 August 2020 18:32 To: Yul Kwon; Ronald Ho; Michael Stillwell; Colm Harte; Ben Cloney; damian.beresford@nearform.com Subject: Major Issue: Google Play Services eating battering Hi all Major issues being reported as COVID Tracker eating battery, we've had no app updates. From what we've seen reported it looks like it's Google Play Services possibly on Samsung only but we haven't validated this. See below thread on Reddit. https://www.reddit.com/r/ireland/comments/i5wo9v/covid app acting strange/?utm me dium=android app&utm source=share https://twitter.com/ismiseleenie/status/1292072609039998981?s=20 Slán Gar 3 4)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 19:20 Sent Items Show all 9 recipients To: Ronald Ho Cc: Yul Kwon Michael Stillwell Colm Harte Cian O'Maidin Ben Cloney; ... ; Hi Ronald This is all reported online we don't have access to bug reports, from what I've found so far it's • • • • Slán Gar 4 OnePlus s10 a7 on android 10 a20 5)From: Ronald Ho Sent: 08 August 2020 19:11 To: Gar T. MacCriosta Cc: Yul Kwon; Michael Stillwell; Colm Harte; ; damian.beresford Cian O'Maidin; Ben Cloney; Fran Thompson Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. hi Gar, thanks for reaching out - we're looking into it. do you have sample devices where this is happening? the reddit thread didn't have any devices for us to try and draw a correlation. any other bug reports you can get from these devices to help with debugging? - Ronald 5 6)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 19:23 Sent Items To: Fran Thompson; we might need to get a bug report off one of them, is that a possibility? 6 7)From: Fran Thompson Sent: 08 August 2020 19:14 To: Gar T. MacCriosta Subject: RE: Major Issue: Google Play Services eating battering Gar Samsung s10 and a20 to name two... Regards Fran Regards Fran The HSE’s Office of the Chief Information Officer, Delivering #eHealth4All www.ehealthireland.ie Fran Thompson Interim Chief Information Officer fran.thompson@hse.ie +353 1 635 2711 HSE Dr. Steevens’ Hospital Dublin 8, Eircode D08 W2A8 Oifig an Phríomhoifigeach Faisnéise, FSS Ospidéal Dr Steevens Baile Átha Cliath 8 -------- Original message ------From: "Gar T. MacCriosta" Date: 08/08/2020 18:53 (GMT+00:00) To: Yul Kwon , Ronald Ho Colm Harte Cc: damian.beresford , Cian O'Maidin Cloney , Fran Thompson Subject: Re: Major Issue: Google Play Services eating battering + ade 7 , Michael Stillwell , ade , Ben see below reports of COVID Tracker/Google Play Services battery consumption issues. People are having to uninstall app, we need to get an update on this ASAP slán Gar 8 8)From: Gar T. MacCriosta Sent: 08 August 2020 18:32 To: Yul Kwon; Ronald Ho; Michael Stillwell; Colm Harte; Ben Cloney; damian.beresford Subject: Major Issue: Google Play Services eating battering Hi all Major issues being reported as COVID Tracker eating battery, we've had no app updates. From what we've seen reported it looks like it's Google Play Services possibly on Samsung only but we haven't validated this. See below thread on Reddit. https://www.reddit.com/r/ireland/comments/i5wo9v/covid app acting strange/?utm me dium=android app&utm source=share https://twitter.com/ismiseleenie/status/1292072609039998981?s=20 Slán Gar 9 9)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 19:34 Sent Items Show all 9 recipients To: Ronald Ho Cc: Yul Kwon Michael Stillwell Colm Harte ; ade damian.beresford Cian O'Maidin Ben Cloney; ... Hi Ronald We have an s10 (Frans) what do you need to do with it. Let us know what you need? Fran just tried clearing cache, clearing data on an a20 (GPS Version 20.26.14.12040632000851) will let you know. Slán Gar 10 10)From: Ronald Ho > Sent: 08 August 2020 19:11 To: Gar T. MacCriosta Cc: Yul Kwon; Michael Stillwell; Colm Harte; ade ; damian.beresford Cian O'Maidin; Ben Cloney; Fran Thompson Subject: Re: Major Issue: Google Play Services eating battering ; CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. hi Gar, thanks for reaching out - we're looking into it. do you have sample devices where this is happening? the reddit thread didn't have any devices for us to try and draw a correlation. any other bug reports you can get from these devices to help with debugging? - Ronald Reporting frequent wake-ups; can you clarify if there are recent Google PLay Services updates that would impact this? 11 11)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 19:41 Sent Items To: Ronald Ho < >; fran doesn't have dev tools, colm is on the way home and will generate a bug report 12 12)From: Ronald Ho Sent: 08 August 2020 19:37 To: Gar T. MacCriosta Cc: Yul Kwon; Michael Stillwell; Colm Harte; ade ; damian.beresford Cian O'Maidin; Ben Cloney; Fran Thompson Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. let's get a bug report before data is cleared. - Ronald 13 13)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 19:43 Sent Items Show all 9 recipients To: Ronald Ho Cc: Yul Kwon Michael Stillwell Colm Harte ade damian.beresford Cian O'Maidin Ben Cloney; ; ; ; ... Hi all Colm Harte will be at his 'puter in 10 mins and will generate a bug report. He is experiencing the same issue? He has restarted the device and this issue seems to persist. Slán Gar 14 14)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 19:45 Sent Items Show all 10 recipients To: Ronald Ho Cc: Cian O'Maidin Yul Kwon Michael Stillwell Colm Harte < ade damian.beresford Ben Cloney; ; ; ... nothing Ronald the app hasn't change since last release >2 weeks ago Re: Major Issue: Google Play Services eating battering 15 15)REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 19:52 Sent Items Show all 10 recipients To: Ronald Ho Cc: Cian O'Maidin Yul Kwon Michael Stillwell Colm Harte ; ; >; ade damian.beresford Ben Cloney; ... no we haven't asked users to adjust anything with relation to battery 16 16)From: Ronald Ho < Sent: 08 August 2020 19:50 To: Gar T. MacCriosta Cc: Cian O'Maidin; Yul Kwon; Michael Stillwell; Colm Harte; ade ; damian.beresford ; Ben Cloney; Fran Thompson; Dave Burke Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. ok got it. no app client or server changes. have you asked users to disable battery optimizations for your app? - Ronald 17 17)From: Ronald Ho Sent: 08 August 2020 19:42 To: Gar T. MacCriosta Cc: Cian O'Maidin; Yul Kwon; Michael Stillwell; Colm Harte; ade damian.beresford ; Ben Cloney; Fran Thompson; Dave Burke Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. thanks Gar, we're not seeing a related push on our end on this (we're still looking), also quick searching on twitter and reddit didn't see other countries reporting this. a bug report from any affected device would help tremendously. You mentioned no recent app changes, but have there been recent updates to the app on the serverside (it's reactive native right?) - Ronald On Sat, Aug 8, 2020 at 11:37 AM Gar T. MacCriosta > wrote: Reporting frequent wake-ups; can you clarify if there are recent Google PLay Services updates that would impact this? 18 18)From: Cian O'Maidin Sent: 08 August 2020 19:18 To: Gar T. MacCriosta Cc: Yul Kwon; Ronald Ho; Michael Stillwell; Colm Harte; ade ; damian.beresford ; Ben Cloney; Fran Thompson Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. People reporting from other android devices too 19 19)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 23:38 Sent Items Show all 12 recipients To: Jake Klinker Cc: Colm Harte Nandini Kappiah Adewale Oshineye Ben Cloney; Cian O'Maidin < Damian Beresford < Dave Burke < ; >; >; >; ; >; ... Thanks Jake. It’s 23:40 ish here, I’ll be checking in but if you need anything urgently please call Sent from my iPhone On 8 Aug 2020, at 23:35, Jake Klinker < > wrote: Gar - we're still trying to determine the right mitigation strategy, but we think there are a few options to fix without user intervention. We'll keep you updated as we decide on a path forward. On Sat, Aug 8, 2020 at 3:29 PM Colm Harte Thanks Jake, > wrote: Will try this. We have found the key that is the issue. This has now been cleared out of our server side data so will prevent this problem going forward. Regards, Colm On Sat, 8 Aug 2020 at 23:15, Jake Klinker 20 > wrote: If you go to Settings > Google Exposure Notifications > Exposure checks and in the top right corner choose the menu option and delete the checks, that may clear out the pending matching run after a reboot. Please let me know if it does not. I think I've tracked this down to keys which include a report_type, which is failing to parse properly, potentially due to a difference between your proto definition and ours. I'm still trying to work out the details. On Sat, Aug 8, 2020 at 2:25 PM Colm Harte Thanks Jake, wrote: I've tried clearing play services data cache, force rebooting the phone but error still reappears when phone restarts, any advice on how to stop ENS being stuck in this loop and draining battery? Thanks, Colm On Sat 8 Aug 2020, 22:16 Jake Klinker, wrote: Hi Colm, thanks for the files, I'm trying to go through them now to see if I can find anything. As for your question about the error code being returned - we run key processing on a background thread that kicks off after the API returns the successful response, so I'd expect all calls to still return successful here even if something was wrong in one of them. I'll update you once I have more data. On Sat, Aug 8, 2020 at 2:08 PM Nandini Kappiah wrote: We are trying to run the keys through our tests to see if we can reproduce what is failing. We will get back! On Sat, Aug 8, 2020 at 2:03 PM Colm Harte Hi Dave, > wrote: Attached file (1596909611816.zip) contains all keys we have processed over the last 14 days. If it's an invalid key then it should be in this file. It's also possible we somehow ended up with a corrupted file, I attached the files we have generated over the last couple of days for reference (BundledFiles.zip). Regards, Colm On Sat, 8 Aug 2020 at 21:54, Colm Harte Hi Dave, > wrote: Yes I have an archive of all the key files we have generated, have been going through them and passing them in individually into ENS but have so far received a 21 success response each time. Am I correct in assuming that an invalid key should return a non success response? With the app uninstalled and ENS now showing as off in device settings and after rebooting the phone the error is still appearing. I assume ENS is internally processing a cached file? Any suggestion on how best to force ENS to stop processing, clear Play Services cache data? Colm On Sat, 8 Aug 2020 at 21:47, Dave Burke +nandinik, jklinker wrote: Right, it seems the thread in GMS Core is spinning trying to parse a bad key. Obviously the code in the future needs to be more defensive against a bad key, but it seems there was a bad key passed in to trigger the issue? Is there an archive of keys / protos over the last day or two that could be checked to root cause? On Sat, Aug 8, 2020, 1:40 PM Colm Harte Hi Dave, > wrote: This seems to be ENS itself that is stuck in a loop and not our app repeatedly calling the service. I've uninstalled the app and can still see that same message constantly repeated. Via the app I can correctly process exposure files and so far have been unable to find any that have failed. Regards, Colm On Sat, 8 Aug 2020 at 20:57, Colm Harte Hi Dave, > wrote: Some sample files attached. Colm On Sat, 8 Aug 2020 at 20:40, Dave Burke > wrote: If you can share an example proto being sent down we can check if it's malformed etc. On Sat, Aug 8, 2020, 12:36 PM Colm Harte Hi Dave, Checking that now, will let you know what I find. 22 wrote: Colm On Sat, 8 Aug 2020 at 20:35, Dave Burke The real engineers are looking at this. > wrote: But in the meantime, Colm, I do think the issue is with that log line. I can see the CPU is constantly spinning, GMS Core persistent process is constantly running ExposureMatchingService, grabbing/releasing a wakelock (indicating a loop). The phone cannot sleep, hence battery drain. Can you check your diagnosis server for bad data (corrupt exposure key data)? That would explain why this is specific to the Irish app and not correlated with a change from Google or Nearform on the phone side. 23 20)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 07:34 Sent Items Show all 12 recipients To: Michael Stillwell Cc: Jake Klinker Colm Harte Nandini Kappiah Adewale Oshineye Ben Cloney; Cian O'Maidin Damian Beresford < ; ; >; ... Will get you details. Person with devices not online yet, waiting on response. Thanks again for the work on this guys. Slán Gar Sent from my iPhone 24 On 9 Aug 2020, at 07:32, Michael Stillwell > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. They will need to be in allowlist. Can you provide some accounts. (Probably hold some back, so you can also test a production release.) On Sun, 9 Aug 2020, 07:28 Gar T. MacCriosta, > wrote: Thanks Jake. I’m waiting on Colm to get back online. We have some devices in this state but not all are associated with allowed list accounts. Can we use these or add them to allowed list?? G On 9 Aug 2020, at 07:07, Jake Klinker wrote: Hi Gar, Colm, We've identified a fix on our side and are planning on pushing a new build to your allowlisted early access accounts soon. This'll be version 203302002 when your devices receive it. I think Colm already discovered this, but we found a key that was 18 bytes instead of 16 in one of the files you shared with us, and that key was the one causing the problem so we should make sure that it is not served to any users anymore and that the right guardrails are in place on the server to prevent similar keys from being served. Once the build is pushed to your devices, it'll cause the matching job to restart again, which will then properly error out due to the bad key and remove the job so that it isn't run again. We'd like your help to verify this as well if you still have test devices in this state, but we should be able to locally verify with the files you've provided if you don't have any devices experiencing this anymore. After we have verified that the fix gets devices back into a good state, we'll be able to push it to all Irish users over the course of a few days. Please let me know if you have any questions or concerns, Jake On Sat, Aug 8, 2020 at 4:44 PM Jake Klinker > wrote: It looks like reboot should not be required if you uninstall and reinstall. On Sat, Aug 8, 2020 at 4:42 PM Colm Harte Thanks Jake, 25 > wrote: So the immediate fix for users should be to uninstall and reinstall, is a reboot required? Reagrds, Colm On Sun 9 Aug 2020, 00:24 Jake Klinker, wrote: Thanks Colm, sorry about that, I double checked the code and it only deletes finished matching requests, not in process ones it looks like. Wiping random IDs will solve it because we skip processing the files if there aren't any scan results stored, however once you get scan results in the database again it'll probably trigger/break it again. Our plan is looking like we'll push out a hotfix, at least to Irish users, which can properly handle report types as per our public documentation. That update will cause devices to restart the matching process and then properly parse and process the files that are broken now. Until this is ready, uninstalling and reinstalling the app will trigger us to clear all of it's data and get rid of the stuck task. On Sat, Aug 8, 2020 at 4:15 PM Colm Harte I cleared my Random IDs and rebooted and the error is now gone. > wrote: Colm On Sun, 9 Aug 2020 at 00:09, Colm Harte Hi Jake, wrote: Clearing the ENS checks log a=has not resolved the issue for me. I'm still seeing the same error in the logs. Colm On Sat, 8 Aug 2020 at 23:38, Gar T. MacCriosta wrote: Thanks Jake. It’s 23:40 ish here, I’ll be checking in but if you need anything urgently please call Sent from my iPhone On 8 Aug 2020, at 23:35, Jake Klinker wrote: Gar - we're still trying to determine the right mitigation strategy, but we think there are a few options to fix without user intervention. We'll keep you updated as we decide on a path forward. On Sat, Aug 8, 2020 at 3:29 PM Colm Harte 26 wrote: Thanks Jake, Will try this. We have found the key that is the issue. This has now been cleared out of our server side data so will prevent this problem going forward. Regards, Colm On Sat, 8 Aug 2020 at 23:15, Jake Klinker wrote: If you go to Settings > Google Exposure Notifications > Exposure checks and in the top right corner choose the menu option and delete the checks, that may clear out the pending matching run after a reboot. Please let me know if it does not. I think I've tracked this down to keys which include a report_type, which is failing to parse properly, potentially due to a difference between your proto definition and ours. I'm still trying to work out the details. On Sat, Aug 8, 2020 at 2:25 PM Colm Harte Thanks Jake, wrote: I've tried clearing play services data cache, force rebooting the phone but error still reappears when phone restarts, any advice on how to stop ENS being stuck in this loop and draining battery? Thanks, Colm On Sat 8 Aug 2020, 22:16 Jake Klinker, wrote: Hi Colm, thanks for the files, I'm trying to go through them now to see if I can find anything. As for your question about the error code being returned - we run key processing on a background thread that kicks off after the API returns the successful response, so I'd expect all calls to still return successful here even if something was wrong in one of them. I'll update you once I have more data. On Sat, Aug 8, 2020 at 2:08 PM Nandini Kappiah wrote: We are trying to run the keys through our tests to see if we can reproduce what is failing. We will get back! On Sat, Aug 8, 2020 at 2:03 PM Colm Harte Hi Dave, wrote: Attached file (1596909611816.zip) contains all keys we have processed over the last 14 days. If it's an invalid key then it should be in this file. It's also possible we somehow 27 ended up with a corrupted file, I attached the files we have generated over the last couple of days for reference (BundledFiles.zip). Regards, Colm On Sat, 8 Aug 2020 at 21:54, Colm Harte Hi Dave, wrote: Yes I have an archive of all the key files we have generated, have been going through them and passing them in individually into ENS but have so far received a success response each time. Am I correct in assuming that an invalid key should return a non success response? With the app uninstalled and ENS now showing as off in device settings and after rebooting the phone the error is still appearing. I assume ENS is internally processing a cached file? Any suggestion on how best to force ENS to stop processing, clear Play Services cache data? Colm On Sat, 8 Aug 2020 at 21:47, Dave Burke < +nandinik, jklinker wrote: Right, it seems the thread in GMS Core is spinning trying to parse a bad key. Obviously the code in the future needs to be more defensive against a bad key, but it seems there was a bad key passed in to trigger the issue? Is there an archive of keys / protos over the last day or two that could be checked to root cause? On Sat, Aug 8, 2020, 1:40 PM Colm Harte Hi Dave, wrote: This seems to be ENS itself that is stuck in a loop and not our app repeatedly calling the service. I've uninstalled the app and can still see that same message constantly repeated. Via the app I can correctly process exposure files and so far have been unable to find any that have failed. Regards, Colm On Sat, 8 Aug 2020 at 20:57, Colm Harte Hi Dave, 28 wrote: Some sample files attached. Colm On Sat, 8 Aug 2020 at 20:40, Dave Burke > wrote: If you can share an example proto being sent down we can check if it's malformed etc. On Sat, Aug 8, 2020, 12:36 PM Colm Harte Hi Dave, wrote: Checking that now, will let you know what I find. Colm On Sat, 8 Aug 2020 at 20:35, Dave Burke The real engineers are looking at this. wrote: But in the meantime, Colm, I do think the issue is with that log line. I can see the CPU is constantly spinning, GMS Core persistent process is constantly running ExposureMatchingService, grabbing/releasing a wakelock (indicating a loop). The phone cannot sleep, hence battery drain. Can you check your diagnosis server for bad data (corrupt exposure key data)? That would explain why this is specific to the Irish app and not correlated with a change from Google or Nearform on the phone side. 29 battery 1 of 6 21)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 08:48 Sent Items Show all 12 recipients To: Colm Harte Cc: Jake Klinker < Nandini Kappiah Adewale Oshineye Ben Cloney; Cian O'Maidin < Damian Beresford < Dave Burke ; >; >; >; >; ... When are we in a position to test this hotfix?? Sent from my iPhone On 9 Aug 2020, at 08:07, Colm Harte wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. 30 Thanks Jake, The key causing the issue was removed from our server and export files last night. We will be adding server side checks to ensure a non 16 byte key cannot appear in export files. Regards, Colm On Sun 9 Aug 2020, 07:28 Gar T. MacCriosta, > wrote: Thanks Jake. I’m waiting on Colm to get back online. We have some devices in this state but not all are associated with allowed list accounts. Can we use these or add them to allowed list?? G 31 22)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 09:39 Sent Items Show all 13 recipients To: Michael Stillwell Cc: Colm Harte Jake Klinker Nandini Kappiah Adewale Oshineye Ben Cloney; Cian O'Maidin Damian Beresford ; ; ; ; >; ; ... Hi Michael +Terry Halpin I think Colm is best placed to test this right now. Slán Gar 32 The key causing the issue was removed from our server and export files last night. We will be adding server side checks to ensure a non 16 byte key cannot appear in export files. Regards, Colm On Sun 9 Aug 2020, 07:28 Gar T. MacCriosta, > wrote: Thanks Jake. I’m waiting on Colm to get back online. We have some devices in this state but not all are associated with allowed list accounts. Can we use these or add them to allowed list?? G 34 24)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 11:29 Sent Items To: Colm Harte Got it. I didn’t get the previous one. Sent from my iPhone On 9 Aug 2020, at 11:28, Colm Harte > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Did you get this? ---------- Forwarded message --------From: Colm Harte < Date: Sun, 9 Aug 2020 at 10:45 Subject: Re: Major Issue: Google Play Services eating battering To: Michael Stillwell Cc: Gar T. MacCriosta < Jake Klinker Nandini Kappiah , Adewale Oshineye Cloney < , Cian O'Maidin Beresford , Dave Burke Fran Thompson Ronald Ho Kwon , Ben , Damian >, Yul Hi Michael, The update fixes the issue, on v203302003 I now see the below lines logged 2020-08-09 10:42:36.002 11070-15052/? E/ExposureNotificationJni: Failed to decode exposure key 2020-08-09 10:42:36.002 11070-15052/? E/ExposureNotificationJni: Found an invalid key, skipping entire batch I'm still not able to identify an exact sequence to clear the issue on phones on v203302001. An uninstall and reinstall did not correct the issue. 35 Regards, Colm 36 25)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 11:30 Sent Items Show all 12 recipients To: Michael Stillwell < Cc: Colm Harte < Jake Klinker Nandini Kappiah >; ; Adewale Oshineye Ben Cloney; Cian O'Maidin Damian Beresford ... Hi What’s the next step? looks like from Colm’s email that fix seems to eliminate the issue. Slán Far Sent from my iPhone 37 26)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 13:08 Sent Items Show all 12 recipients To: Michael Stillwell Cc: Fran Thompson; Colm Harte Jake Klinker Nandini Kappiah Adewale Oshineye Ben Cloney; Cian O'Maidin ; ; ; ... Thanks Michael keep us posted on timeline please. We want to get an update out to media it’s being reported on lunchtime news. Can we say we have a fix?? 1. Is there any other validation we need to do? 2. What’s the timeline from when we release 90% of devices will be updated? 3. Is there any way for users to force the update? 4. Is this the long term fix or will there be a subsequent update? With relation to the allowed list users.Fran is on the allowed list but hasn’t seen this patch come through? Is there a way to force?? Slán Far Sent from my iPhone On 9 Aug 2020, at 12:26, Michael Stillwell wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Thanks for the confirmation that the update seems to have fixed things, that's good news. 38 There were a few options for deploying this change to production. The people on this thread from the West Coast will need to decide which one we go with when they wake up over the next few hours. But to set expectations, I don't see the update being widely deployed before Sunday evening. Michael On Sun, Aug 9, 2020 at 11:57 AM Fran Thompson > wrote: Hi The quicker this gets resolved the better, the speed we react and fix a problem is very important. Regards Fran 39 27)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 13:59 Sent Items Show all 15 recipients To: Michael Stillwell Cc: Fran Thompson; Colm Harte Jake Klinker Nandini Kappiah Adewale Oshineye Ben Cloney; Cian O'Maidin ; ; ; ; ... @colm can you provide details to Michael?? How are you responding to queries? Let’s align the message @ben @ryan @lorcan Slán Far Sent from my iPhone On 9 Aug 2020, at 13:56, Michael Stillwell > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. +Julie Dilger +Lorcan O’Flaherty +Ryan Meade since we're getting questions about this too. Hi Gar, Re validation, ideally we would like to verify that a device runs the matching process more or less continuously before getting the update, but then runs normally after the update. The bugreport you provided (bugreport-ELE-L09EEA-HUAWEIELE-L09-2020-08-08-2001-22.zip) shows the first part well. If you can generate something similar again after that device gets version 203302003 that would be helpful in validating the fix. 40 Perhaps increase the size of the logfile buffer (via developer options) on the devices currently in a bad state to try to make sure all relevant info is captured. (See screenshot.) Note that we are confident that the fix does resolve the particular problem you're having. We do need to do some more QA but the main issue is how to roll it out. I can't comment on the timelines, but someone will follow up on this in a few hours. Michael On Sun, Aug 9, 2020 at 1:08 PM Gar T. MacCriosta wrote: Thanks Michael keep us posted on timeline please. We want to get an update out to media it’s being reported on lunchtime news. Can we say we have a fix?? 1. Is there any other validation we need to do? 2. What’s the timeline from when we release 90% of devices will be updated? 3. Is there any way for users to force the update? 4. Is this the long term fix or will there be a subsequent update? With relation to the allowed list users.Fran is on the allowed list but hasn’t seen this patch come through? Is there a way to force?? Slán Far Sent from my iPhone On 9 Aug 2020, at 12:26, Michael Stillwell wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Thanks for the confirmation that the update seems to have fixed things, that's good news. There were a few options for deploying this change to production. The people on this thread from the West Coast will need to decide which one we go with when they wake up over the next few hours. But to set expectations, I don't see the update being widely deployed before Sunday evening. 41 Michael On Sun, Aug 9, 2020 at 11:57 AM Fran Thompson > wrote: Hi The quicker this gets resolved the better, the speed we react and fix a problem is very important. Regards Fran The HSE’s Office of the Chief Information Officer, Delivering Fran Thompson Interim Chief Information Officer f HSE #eHealth4All Dr. Steevens’ Hospital www.ehealthireland.ie Dublin 8, Eircode D08 W2A8 Oifig an Phríomhoifigeach Faisnéise, FSS Ospidéal Dr Steevens Baile Átha Cliath 8 -------- Original message -------- 42 28)From: "Gar T. MacCriosta" > Date: 09/08/2020 11:31 (GMT+00:00) To: Michael Stillwell > Cc: Colm Harte Jake Klinker Nandini Kappiah Adewale Oshineye Cloney , Cian O'Maidin Beresford , Dave Burke Fran Thompson >, Ronald Ho Kwon Subject: Re: Major Issue: Google Play Services eating battering , Ben , Damian Yul Hi What’s the next step? looks like from Colm’s email that fix seems to eliminate the issue. Slán Far Sent from my iPhone On 9 Aug 2020, at 09:30, Michael Stillwell > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi Gar, The test process: 1. Find a device that currently has the problem that is also on the allowlist. 2. Wait until that device receives version 203302002 of the ENS module. Check for the version either via: - Settings > Google > COVID-19, scroll to bottom. Last 9 digits of Version should be 203302002. See screenshot, which shows 203302001 (this device doesn't have the update yet). - adb command: adb shell dumpsys activity provider com.google.android.gms.chimera.container.GmsModuleProvider grep 'nearby_en'. 3. Problem is expected to go away once this version is on the device. - I'm trying to figure out how to verify that the problem has gone away (other than reduced battery usage over time). Not totally sure but I think you will see "Found an invalid key, skipping entire batch" in the logs. We're adding to the allowlist but it will be hours before this device gets the update. Existing allowlisted accounts/devices should get the update much faster. 43 Michael On Sun, Aug 9, 2020 at 8:49 AM Gar T. MacCriosta When are we in a position to test this hotfix?? wrote: Sent from my iPhone On 9 Aug 2020, at 08:07, Colm Harte wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Thanks Jake, The key causing the issue was removed from our server and export files last night. We will be adding server side checks to ensure a non 16 byte key cannot appear in export files. Regards, Colm 44 29)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 14:34 Sent Items Show all 15 recipients To: Michael Stillwell Cc: Fran Thompson; Colm Harte < Jake Klinker Nandini Kappiah Adewale Oshineye Ben Cloney; Cian O'Maidin >; >; ; ; >; ... Hi all Just to reiterate the time sensitivity of this. We’re back into a work week tomorrow and people will not be able to survive without a phone, the risk is that we lose public confidence and start seeing hiGh rates of deletion it’s over 12k since this started. We need to get the fix out ASAP. Slán Gar Sent from my iPhone On 9 Aug 2020, at 13:56, Michael Stillwell < > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. +Julie Dilger +Lorcan O’Flaherty +Ryan Meade since we're getting questions about this too. Hi Gar, Re validation, ideally we would like to verify that a device runs the matching process more or less continuously before getting the update, but then runs normally after the update. 45 The bugreport you provided (bugreport-ELE-L09EEA-HUAWEIELE-L09-2020-08-08-2001-22.zip) shows the first part well. If you can generate something similar again after that device gets version 203302003 that would be helpful in validating the fix. Perhaps increase the size of the logfile buffer (via developer options) on the devices currently in a bad state to try to make sure all relevant info is captured. (See screenshot.) Note that we are confident that the fix does resolve the particular problem you're having. We do need to do some more QA but the main issue is how to roll it out. I can't comment on the timelines, but someone will follow up on this in a few hours. Michael 46 30)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 16:07 Sent Items Show all 15 recipients To: Dave Burke Cc: Michael Stillwell Fran Thompson; Colm Harte Jake Klinker Nandini Kappiah Adewale Oshineye Ben Cloney; >; >; >; ; ; ; ... Hi Dave Understood. The file in question didn’t hit servers until Friday PM (Colm should be able to give more precise info) pretty much everyone will have downloaded and the issue is blocking subsequent checks against teks since then. Once this clears we will be able to get all users caught up. Any idea on time it will take for a fix to propagate across user devices?? Slán Gar Sent from my iPhone On 9 Aug 2020, at 15:55, Dave Burke > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Understood, Gar. We are treating with highest priority. Looking at fastest (and safest) way to get a point fix into the EN module, tested, with expedited roll out. More as we have it... Quick question - how long was the bad file on server? Do we assume all devices downloaded it? 47 31)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 18:39 Sent Items Show all 20 recipients To: Yul Kwon Cc: Colm Harte Bradley Nelson < Garry Boyer < Lydia Fitzpatrick Alex McPhillips < Pete Voss < Michael Stillwell < >; >; >; >; >; ... Thanks Yul Sent from my iPhone On 9 Aug 2020, at 18:17, Yul Kwon > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Adding a few more folks from our eng, comms, and policy teams Gar/Fran/Colm - Just to add to Dave's comment, this issue is our highest priority -- we've been sprinting on a fix that we can push out as fast as possible without creating risk of inadvertently making things worse. We should have a concrete plan soon, and would also be happy to support you on external messaging. Thanks. Yul On Sun, Aug 9, 2020 at 9:43 AM Colm Harte < > wrote: One more bug report this time after processing a file with no invalid keys. Taken on the same device as last 2. I've only been able to verify the patch on 1 device, my other 2 test devices have so far not updated their version of google play services to the latest. 48 Regards, Colm 49 32)Re: Tracking App REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 18:44 Sent Items To: Ben Cloney; Cc: Fran Thompson; There seems to be no consistency in the solutions 1. clearing cache reported as sometimes working but Also doesn’t work 2. Clearing data as a above 3. Deleting the app and reinstalling as above There doesn’t seem to be a consistent set of steps that will work for everyone. I’m reluctant to advise clearing data from google play in case there are knock on impacts, cache would be fine but looks like it’s not sufficient. Deleting the app will resolve the battery issue but i think we should ask people to wait for the update. Open to suggestions on this. Slán Far Sent from my iPhone On 9 Aug 2020, at 18:17, Ben Cloney wrote: Hi Gar Can you explain the complicated fix to me again, clearing the cache, does it mean the random IDs that have been shared with you disappear. Some people are also saying switching the phone off and on again works! Ben Ben Cloney Acting Head of Digital Communications Division, HSE 10 / 11 Cornmarket, Dublin, D08, P520 Mobile hse.ie/communications @hselive Read our blog https://www.hse.ie/digital/blog 50 33)From: Brian Malone Date: Sunday 9 August 2020 at 18:12 To: Ben Cloney Fran Thompson Cc: PRESS Subject: Fw: Tracking App Hi Ben and Fran, Have never heard of Goosed.ie - will I just go back with the response issued earlier today or do you want to address this cache related query? Response sent to previous queries here: We were notified yesterday by some people with Android phones that their batteries were running down quicker than usual. We have identified the issue and are working with our developers and Google to fix it. There was a problem relating to the Exposure Notification System getting stuck in a loop when processing the Random IDs. Google are releasing an update to Play services in the next 24 hrs that will resolve the issue for people that are experiencing this problems. Additional controls are being introduced to ensure this does not happen again. Thanks Brian 51 34)From: Martin Meany Sent: Sunday, August 9, 2020 5:45:04 PM (UTC+00:00) Dublin, Edinburgh, Lisbon, London To: PRESS Subject: Tracking App CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hey guys Sorry I'm late to this. I'm looking into the tracking app battery issues which seem to be caused by Google Play I've seen a solution to fix the issue online which looks at clearing the cache. Does this also wipe out contact tracing already carried out or does the app store this elsewhere other than the cache? Any additional info you've sent out to outlets woud be appreciated. _________________ Marty Meany Editor @ Goosed.ie 52 35)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 18:47 Sent Items To: Yul Kwon >; Thanks for the support Yul. Not what anyone needed we’ll see the damage tomorrow in app deletions my guess is that a good amount will come back as the numbers of cases tripled yesterday and 3 counties got locked down again. We will lose some but I think we may also pick up some people via the media attention. Some journos are lining up to attack but so be it. Slán Gar Sent from my iPhone 53 36)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 19:05 Sent Items To: Yul Kwon >; Cheers Yul we’ll get this sorted 💪 thank Jake from me that’s a legendary sacrifice Sent from my iPhone On 9 Aug 2020, at 18:53, Yul Kwon > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Our team is meeting in ten minutes to finalize the plan. Yeah, I really hope this doesn't lead to large uninstalls, and I love how you look at the silver lining ("we may also pick up some people via the media attention") :) I don't know if it's coming through on the email thread, but our team really has been sprinting on this around the clock since you first flagged it to us. Jake Klinker (one of our lead engineers) was in the middle of an Ironman when we pinged him, and he's been on this nonstop since along with the rest of our eng team. We'll try to finalize things in the meeting and get back to you with a plan after. Thanks. Yul 54 37)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 20:11 Sent Items Show all 20 recipients To: Colm Harte Cc: Dave Burke Yul Kwon Bradley Nelson Garry Boyer Lydia Fitzpatrick Alex McPhillips Pete Voss ; ; >; ; ... Thanks Dave and team that’s great news. noted re server data ingestion we will have nature it validates individual teks as well as files. We will monitor here for progress let us know if we can help. Ben/Fran we can coordinate on updates Slán and thanks again Gar Sent from my iPhone On 9 Aug 2020, at 20:06, Colm Harte > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Thanks Dave, We will start adding server validation logic tomorrow to catch invalid keys. Will review the the Google reference server code to ensure we have matching checks. Regards, Colm On Sun 9 Aug 2020, 19:39 Dave Burke, wrote: Update from our side: we have started rolling out an improvement to the EN module so that (a) it's resilient to malformed input, (b) phones in a bad state will recover. It's at 10% and we're increasing to 25% as we speak. We will hold there for 8 hrs to verify telemetry 55 looks good (and in parallel validate with our internal testing). Then push to 100%. Phones check in every 4 hrs so should be resolved for most everyone as people start their work day on Ireland in be morning. We also took a look at the Nearform server code and would recommend adding validation on data ingestion. We're happy to share what we've done for our servers. Finally, we should coordinate on PR so our messaging is aligned, also that we don't give out too much information until the rollout is complete. We can definitely say that a fix has started rolling out now. Julie and Yul from our side are following up. Thanks, Dave 56 38)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 20:34 Sent Items Show all 20 recipients To: Julie Dilger Cc: Colm Harte ; >; >; >; >; ; ... Hi Julie ref Bens earlier email. Ben’s copy from earlier for press queries We were notified yesterday by some people with Android phones that their batteries were running down quicker than usual. We have identified the issue and are working with our developers and Google to fix it. There was a problem relating to the Exposure Notification System getting stuck in a loop when processing the Random IDs. Google are releasing an update that will resolve the issue for people that are experiencing this problem. Additional controls are being introduced to ensure this does not happen again. Julie copy A bug was introduced due to a formatting error, which led to increased battery usage in a number of devices with the COVID Tracker app. The issue has now been resolved and a fix has been pushed out. HSE, NearForm, and Google have also implemented additional measures to improve the resilience of the Exposure Notifications System and ensure that these types of issues will not arise going forward." Seems to be broadly inline. I think we provided a little more info. I think the improve resilience is the line we will take in future queries Over to Ben and Fran for comment Slán Gar 57 Sent from my iPhone On 9 Aug 2020, at 20:24, Julie Dilger < > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi Gar, Ben, Fran, We've worked on some holding language (below) that we can all use until the fix is fully deployed. As Dave mentioned we don't want to provide too much detail at this point. It would be great if we could all work from the same messaging. Let me know if you have any questions. Julie "A bug was introduced due to a formatting error, which led to increased battery usage in a number of devices with the COVID Tracker app. The issue has now been resolved and a fix has been pushed out. HSE, NearForm, and Google have also implemented additional measures to improve the resilience of the Exposure Notifications System and ensure that these types of issues will not arise going forward." 58 39)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 20:35 Sent Items Show all 20 recipients To: Julie Dilger Cc: Colm Harte Dave Burke < Yul Kwon Bradley Nelson Garry Boyer Lydia Fitzpatrick Alex McPhillips ; >; >; >; ; ... Hi Julie ref Bens earlier email. Ben’s copy from earlier for press queries We were notified yesterday by some people with Android phones that their batteries were running down quicker than usual. We have identified the issue and are working with our developers and Google to fix it. There was a problem relating to the Exposure Notification System getting stuck in a loop when processing the Random IDs. Google are releasing an update that will resolve the issue for people that are experiencing this problem. Additional controls are being introduced to ensure this does not happen again. Julie copy A bug was introduced due to a formatting error, which led to increased battery usage in a number of devices with the COVID Tracker app. The issue has now been resolved and a fix has been pushed out. HSE, NearForm, and Google have also implemented additional measures to improve the resilience of the Exposure Notifications System and ensure that these types of issues will not arise going forward." Seems to be broadly inline. I think we provided a little more info. I think the improve resilience is the line we will take in future queries Over to Ben and Fran for comment Slán Gar 59 40)Re: Update for morning ireland REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 20:50 Sent Items To: Ben Cloney; Cc: Fran Thompson; I think it’s fine, we have a little more info earlier about random ids but it’s fine Sent from my iPhone On 9 Aug 2020, at 20:43, Ben Cloney < > wrote: Are you ok with sending google’s wording to Morning Ireland Ben Hi Brian Can you send this update to morning Ireland A bug was introduced due to a formatting error, which led to increased battery usage in a number of Android devices with the COVID Tracker app installed. The issue has now been resolved and a fix has been pushed out to Android devices by Google. HSE, NearForm, and Google have also implemented additional measures to improve the resilience of the Exposure Notifications System and ensure that these types of issues will not arise again. Ben Cloney Head of Digital HSE Communications 60 41)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 21:30 Sent Items Show all 20 recipients To: Julie Dilger < Cc: Colm Harte < Dave Burke Yul Kwon Bradley Nelson < Garry Boyer Lydia Fitzpatrick ; ; ; >; >; >; >; ... Hi all Are there any compatibility issues that we should be aware of i.e. sdk v1.6 updates. We are using are using std ens config and interfaces? Slán Far Sent from my iPhone On 9 Aug 2020, at 20:50, Julie Dilger > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Thanks Gar, the messages seem relatively in line. I'll check back in the morning to make sure everything is landing smoothly. Thanks Julie 61 42)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 21:52 Sent Items Show all 20 recipients To: Michael Stillwell < Cc: Julie Dilger Colm Harte Dave Burke Yul Kwon < Bradley Nelson Garry Boyer < Lydia Fitzpatrick >; ; m>; >; >; m>; ... Thanks Michael. Fran just reported getting the update keys are being processed and phone seems to have ‘calmed down’ 👍 we will monitor at this end Sent from my iPhone On 9 Aug 2020, at 21:49, Michael Stillwell > wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Gar re compat, we don't think you will be affected (there should be no behavioral changes) but please test. We would've preferred a more targeted fix but this wasn't possible. Note: features such as the “Shorten rolling period” change, which we did originally want to enable as part of the 1.6 release (as explained in the "Early TEK release support implementation and testing" email) are NOT enabled (this and other features can be toggled server-side). Michael On Sun, Aug 9, 2020 at 9:30 PM Gar T. MacCriosta > wrote: Hi all Are there any compatibility issues that we should be aware of i.e. sdk v1.6 updates. We are using are using std ens config and interfaces? 62 Slán Far Sent from my iPhone 63 43)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 06:40 Sent Items Show all 20 recipients To: Dave Burke Cc: Yul Kwon Michael Stillwell Julie Dilger Colm Harte ; ; >; ... Hi all Thanks again for the help and the huge effort you guys put in to resolve. We will be asking people to come back; the sentiment on Twitter suggested that people are open to returning once the issues were fixed. Any thoughts on this are welcome? Ben, Fran and I will discuss this morning and put something together. @julie/@ben can we add something to the press release to say that it’s “safe to get back in the water”?? Slán Gar Sent from my iPhone 64 44)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 08:16 Sent Items Show all 20 recipients To: Yul Kwon Cc: Dave Burke < Michael Stillwell Julie Dilger Colm Harte < Bradley Nelson < Garry Boyer Lydia Fitzpatrick m>; >; ; >; ... thanks Yul, we will keep you posted 65 45)From: Yul Kwon Sent: 10 August 2020 08:15 To: Gar T. MacCriosta Cc: Dave Burke; Michael Stillwell; Julie Dilger; Colm Harte; Bradley Nelson; Garry Boyer; Lydia Fitzpatrick; Alex McPhillips; Pete Voss; Fran Thompson; Jake Klinker; Nandini Kappiah; Adewale Oshineye; Ben Cloney; Cian O'Maidin; Damian Beresford; Ronald Ho; Lorcan O’Flaherty; Ryan Meade Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi Gar, Yes, it should be safe to encourage users who've uninstalled the app to reinstall it. As for users who haven't uninstalled the app and are still experiencing the battery drain issue, that should be resolved as soon as their phones have downloaded the new module with the fix (we think most phones in Ireland should do this today). Good luck! Yul On Sun, Aug 9, 2020 at 10:40 PM Gar T. MacCriosta < wrote: Hi all Thanks again for the help and the huge effort you guys put in to resolve. We will be asking people to come back; the sentiment on Twitter suggested that people are open to returning once the issues were fixed. Any thoughts on this are welcome? Ben, Fran and I will discuss this morning and put something together. @julie/@ben can we add something to the press release to say that it’s “safe to get back in the water”?? Slán Gar Sent from my iPhone On 10 Aug 2020, at 01:28, Dave Burke wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Metrics look good, ramping to 100% now. It takes on average 12hrs for devices to download the new module and it's possible there are a few stragglers (e.g. no internet connection etc). We've seen ~60,000 uninstalls and ~10,000 reinstalls. Are you considering any messaging to encourage reinstall? Dave 66 46)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 08:20 Sent Items To: Yul Kwon thanks again Yul to you and all the team; pass on my apologies to Jake about the Iron man (was he in the middle of it?) Slán Gar 67 47)From: Yul Kwon Sent: 10 August 2020 08:15 To: Gar T. MacCriosta Cc: Dave Burke; Michael Stillwell; Julie Dilger; Colm Harte; Bradley Nelson; Garry Boyer; Lydia Fitzpatrick; Alex McPhillips; Pete Voss; Fran Thompson; Jake Klinker; Nandini Kappiah; Adewale Oshineye; Ben Cloney; Cian O'Maidin; Damian Beresford; Ronald Ho; Lorcan O’Flaherty; Ryan Meade Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi Gar, Yes, it should be safe to encourage users who've uninstalled the app to reinstall it. As for users who haven't uninstalled the app and are still experiencing the battery drain issue, that should be resolved as soon as their phones have downloaded the new module with the fix (we think most phones in Ireland should do this today). Good luck! Yul 68 48)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 08:36 Sent Items To: Yul Kwon 😴😴😴 we're reviewing lots of stuff today 69 49)From: Yul Kwon Sent: 10 August 2020 08:32 To: Gar T. MacCriosta Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Of course, your success is our success, we're all in this together. We were surprised that NearForm's key server didn't build in a check for malformed keys, since the reference server design we had published accounts for this possibility. In any case, it's a good though painful learning on all sides, and the fix we're pushing globally should prevent this for other key servers that don't incorporate this check. I'm going to bed now but will be reading emails as soon as I wake up. Keeping my fingers crossed to hear good news from you when I do. Good luck! Yul 70 50)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 10:08 Sent Items Show all 9 recipients To: Julie Dilger Cc: Colm Harte Alex McPhillips Pete Voss ; Fran Thompson; Ben Cloney; Cian O'Maidin Damian Beresford ... thanks Julie, I'll defer to Ben on this one, all press queries & response go through him. 71 51)From: Julie Dilger < Sent: 10 August 2020 09:54 To: Gar T. MacCriosta Cc: Colm Harte; Alex McPhillips; Pete Voss; Fran Thompson; Ben Cloney; Cian O'Maidin; Damian Beresford; Ryan Meade Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. - to a smaller group Hi Gar, i just wanted to check what you're confirming to press this morning as I've had a number of calls and wanted to ensure we're aligned. Julie 72 52)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 14:43 Sent Items Show all 20 recipients To: Yul Kwon Cc: Dave Burke Michael Stillwell Julie Dilger Colm Harte Bradley Nelson < Garry Boyer < Lydia Fitzpatrick ; >; ; ; ... Hi all Thanks again for the help over the weekend. Based on the timeline from last night we expect this rollout should be complete about now. We have some reports of phones that haven't updated yet. Can you help with the questions below... • • • • • • • Can you give us an update on the % of users who will have received an update? Is this a number we can use in media queries? Does the update obey the device settings regarding downloading over WiFi only? Is there a way for users to aggressively pull down the update via play store? Is there a way for a user to know they have the update? Will restarting the device trigger an update? Is there any other guidance we can provide to app users? Thanks again Slán Gar 73 53)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 17:03 Sent Items Show all 20 recipients To: Yul Kwon Cc: Dave Burke Michael Stillwell Julie Dilger Colm Harte Bradley Nelson Garry Boyer Lydia Fitzpatrick ; ; ; ; >; ... Thanks Yul, keep us posted and we will keep tracking at this end. Slán Gar 74 54)From: Yul Kwon > Sent: 10 August 2020 16:33 To: Gar T. MacCriosta Cc: Dave Burke; Michael Stillwell; Julie Dilger; Colm Harte; Bradley Nelson; Garry Boyer; Lydia Fitzpatrick; Alex McPhillips; Pete Voss; Fran Thompson; Jake Klinker; Nandini Kappiah; Adewale Oshineye; Ben Cloney; Cian O'Maidin; Damian Beresford; Ronald Ho; Lorcan O’Flaherty; Ryan Meade Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi Gar, The update has already been downloaded by the majority of users in Ireland (feel free to share this with press). We expect there to be a long-tail of devices that will eventually download the update over the next day or so, although some devices could take up to a few days. There's unfortunately no way for a user to control or observe the state of the download given that it's being delivered as a module. So users won't be able to trigger the download by restarting the device or pulling it down from the Play Store. As for whether the update will obey the device settings regarding downloading over Wifi only, the answer is no, but the size of the download is only about 5 megabytes. We can share another update with more concrete numbers in a couple of hours. Thanks. Yul 75 55)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 17:22 Sent Items Show all 20 recipients To: Yul Kwon Cc: Dave Burke < Michael Stillwell Julie Dilger Colm Harte Bradley Nelson Garry Boyer Lydia Fitzpatrick >; >; ; ... Hi Yul Is there a % we can use with the media in terms of coverage? Slán Gar 76 56)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Tue 11/08/2020 17:46 Sent Items Show all 20 recipients To: Yul Kwon Cc: Dave Burke < Michael Stillwell Julie Dilger < Colm Harte < Bradley Nelson < Garry Boyer < Lydia Fitzpatrick ... >; ; >; >; >; ; Hi Yul Sorry about the slow response, thanks for the update on this. The global rollout is excellent news. We'll keep you posted on progress and any other issues. Slán Gar 77 57)From: Yul Kwon Sent: 10 August 2020 22:04 To: Gar T. MacCriosta Cc: Dave Burke; Michael Stillwell; Julie Dilger; Colm Harte; Bradley Nelson; Garry Boyer; Lydia Fitzpatrick; Alex McPhillips; Pete Voss; Fran Thompson; Jake Klinker; Nandini Kappiah; Adewale Oshineye; Ben Cloney; Cian O'Maidin; Damian Beresford; Ronald Ho; Lorcan O’Flaherty; Ryan Meade Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi Gar, Here's the latest from our end. We estimate that about 70% of users in Ireland have now downloaded the update. Although the number of successful updates is continuing to rise each hour, as a precautionary measure we're pushing out a global update that will ensure that nearly all users worldwide receive the update this week (again, there's always a long tail, but this should pretty much catch any Irish users who may have been missed in our Irish-specific push). Let us know if you have any other questions. Have a good night! Yul 78 58)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Tue 11/08/2020 20:28 Sent Items Show all 20 recipients To: Yul Kwon Cc: Dave Burke Michael Stillwell Julie Dilger Colm Harte Bradley Nelson Garry Boyer Lydia Fitzpatrick ; ; ... Thanks all for the efforts on this we really appreciate all of your efforts to make this happen. 💪 Slán Far Sent from my iPhone On 11 Aug 2020, at 20:26, Yul Kwon wrote: CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Gar and friends, The update should now be installed on >95% of devices in Ireland, and we expect to get close to 99% in a few hours. If you hear of any new complaints about this, please let us know. Otherwise, we hope this matter has now been resolved. Thanks! Yul 79 59)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Tue 18/08/2020 15:11 Sent Items To: Ryan Meade Ben Cloney; Cc: Fran Thompson; ; thanks Ryan, Ben is on leave this week. Let me come back to you (I may need to wait for ben on the website). If you could share information from that survey it would be really useful. Slán Gar Gar Mac Críosta Contracted to work on behalf of the Office of the Chief Information Officer OCIO - COVID Tracker Lead 80 60)From: Ryan Meade Sent: 18 August 2020 14:46 To: Ben Cloney Cc: Gar T. MacCriosta; Fran Thompson Subject: Re: Major Issue: Google Play Services eating battering CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi Ben, Just coming back to you on this - Julie and I have been talking to our marketing colleagues and there are a few things we can suggest. Let me know if any of the below are of interest and I can set up a call if necessary: • • • We are currently running a survey on adoption and uninstallation in a number of countries, including Ireland. I'm told we can share some insights from this, which should be available later this week or early next week. We could initiate a push notification from the Play Store - this won't specifically target people who have uninstalled but could help drive re-adoption. If you want to run a message about the battery issue being resolved on your site we could potentially run a home-page promotion (blue link under search box) on google.ie to coincide with this. Let me know if you want me to set up a call. My colleague was also asking if you're considering any case studies on where the app has proved helpful? Kind regards, Ryan On Thu, 13 Aug 2020 at 16:19, Julie Dilger > wrote: - Moving a bunch of Google engineering folks to bcc to save their inboxes Hey Ben, Let me sync with our marketing team and see what we can do. Will come back to you asap. Julie On Thu, Aug 13, 2020 at 10:36 AM Ben Cloney Hi Julie 81 wrote: Thanks again for everyone’s support on sorting out the issues over the last few days. We would like to start on a win-back campaign. I think it would be sensible to look at something similar to the play-store promotion that we did in the first week. Here are the numbers from the play store. We have a net loss of well over 83,000 users from our Android user base. Date Aug 7, 2020 Aug 8, 2020 Aug 9, 2020 Aug 10, 2020 August 11, 2020 Augustt 12, 2020 Returning Users Total All uninstalls Net Loss 597 2,525 5,343 3,415 1,246 14,074 4,978 -7,850 8,391 72,010 5,632 -57,987 9,025 45,679 10,416 -26,238 6,396 11,115 7,833 3,114 3,394 7,253 6,043 2,184 29,049 152,656 40,245 -83,362 Any thoughts or ideas would be most welcome. Thanks Ben Ben Cloney Acting Head of Digital Communications Division, HSE 10 / 11 Cornmarket, Dublin, D08, P520 hse.ie/communications @hselive Read our blog https://www.hse.ie/digital/blog 82 New Devices Battery Issues 61)REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Fri 21/08/2020 12:24 Sent Items To: Ben Cloney; Fran Thompson; 1 attachment Hi See attached, I've included Oonagh's suggested updates on the impact on numbers and the section from the release note on impact on reporting metrics. Slán Gar Gar Mac Críosta Contracted to work on behalf of the Office of the Chief Information Officer OCIO - COVID Tracker Lead 83 Other Comms 62)REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sat 08/08/2020 18:43 Sent Items To: Ben Cloney; Fran Thompson; Cc: Colm Harte < Cian O'Maidin damian.beresford Hi Ben Update for social media. Reports of COVID Tracker impacting battery life. HSE are investigating the issue and have escalated this with Google. The issue appears to be related to a recent Google Play Services update and is not related to the COVID Tracker app. We will update on this issue shortly. PS I've raised this with Google and will keep you posted Slán Gar 84 Battery Draining Summary of Issues 63)REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 09:58 Sent Items To: Ben Cloney; Fran Thompson; Cian O'Maidin ; Colm Harte damian.beresford@ You replied on 10/08/2020 10:34. Hi all Here's a summary of events; I've stuck to the main events Reported Symptoms: battery draining and phones heating up First Report: Saturday 09/08 - 13:18 Twitter https://twitter.com/ismiseleenie/status/1292072609039998981?s=2 0 & https://www.reddit.com/r/ireland/comments/i5wo9v/covid app acting strange/?utm medium=android app&utm source=share Device Analysis: android reporting battery usage coming from COVID Tracker and Google Play Services Current Status: Awaiting hotfix to allowed list devices Root Cause: invalid TEK; an 18 byte TEK was passed in a file into ENS. TEKs are 16 bytes. The key in question was received from the Federation server and is a single key Fix: • • • The invalid key has been removed from IE Key Registry in HSE The connection to the federation server has been disabled until Federation Server issues are resolved Hotfix from Google that will handle invalid keys; this will flush the offending key and resume service Timeline: • • • • • • • DPER (Owen Harrison) raised with GMC at 15:47 GMC - escalated to FT & BC & NearForm 18:00 Escalation: Escalated to Google: Saturday 09/08 - 18:30 Google team responded 19:34 Analysis overnight Fix identified: Sunday AM Testing Fix Sunday ~09:30 pending receipt of update to allowed list device Problem Resolution: 85 • • Validate individual keys are correctly formed before publishing Validate Federation Server behaviour and resolve invalid key issue Other Info: This appears to be an IE only issue; we have no reports from elsewhere Follow-on Actions: • • • Slán Gar 86 HSELive update on Social Media informing users that a fix is on the way Check Play Store for app deletions Analyse CCN numbers for Fri/Sat/Sun 64)Re: Battery Draining Summary of Issues REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 13:39 Sent Items To: Ben Cloney; Fran Thompson; Cian O'Maidin Colm Harte damian.beresford Side effect: any device experiencing this issue is not checking keys And will not check until it’s resolved Sent from my iPhone 87 65)Re: Tracking App REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Sun 09/08/2020 19:30 Sent Items To: Ben Cloney; Cc: Fran Thompson; Google team meeting right now. We should have a detailed plan from them on what next shortly. G Sent from my iPhone On 9 Aug 2020, at 18:51, Ben Cloney wrote: Agree we don’t want to get into half arsed fixes, just want to understand so we can respond to query. Ben Cloney Head of Digital HSE Communications 88 66)Gar T. MacCriosta Mon 10/08/2020 08:19 Sent Items To: Ben Cloney; Fran Thompson; Hi all Think we should do a blog post and clarify the issue in more detail, ben let's discuss later. See below. Slán Gar Title: COVID Tracker Battery Issues Explained First off apologies to everyone who was impacted by this issue. The issue has been resolved by an update released by Google last night (Sunday 9th August), for anyone experiencing the issue once the update rolls out your battery consumption will return to normal. We've been working hard to make COVID Tracker robust and reliable. In this case we didn't hit the mark. On Friday (7th of August) night COVID Tracker started to exhibit unusual behaviour on Android devices. Devices were reporting heavy battery consumption and some devices were heating up. While we can't guarantee no bugs will ever occur again we are now working to review COVID Tracker to identify other areas where issues like this might occur. For anyone interested in the underlying causes. A bug was introduced due to a malformed TEK (Temporary Exposure Key). TEKs are the daily Random IDs generated by the Exposure Notification System on your phone. When an app user has a positive result from a COVID19 test they are asked to upload these Random IDs, these IDs are stored in a registry and other phones running COVID Tracker download these IDs and compare them to IDs that they have collected on their device, if their's a match you are notified otherwise life goes on as usual. TEKs are 16 byte identifiers (the length of a the TEK). In this case one of these TEKs gained an extra 2 bytes which meant it was malformed. There is a complicated process of aggregating TEKs into files which are downloaded in batches onto devices. While we had validation on the file structure we weren't validating individual TEKs. Fix 1: Validate each key is well-formed. The file with the offending malformed TEK was generated on Friday 7th at 23:00 and subsequently downloaded by the app which kick-started the problem. COVID Tracker passes the file to ENS (Exposure Notification System) and receives back any matches. This process is all running in the background even on a users phone. In this case ENS accepted the file as valid and started to process matching against each TEK. The malformed TEK was accepted and a thread started spinning; repeatedly trying to process the malformed key. This showed to users as an increase in battery consumption from COVID Tracker and Google Play Services. Because the issue manifested in Google Play Services we couldn't fix via a COVID Tracker app update. Google, HSE and NearForm have worked around the clock to resolve this; the issue has now been resolved and a fix has been pushed out to Android devices by Google. HSE, NearForm, and Google have also implemented additional measures to improve the resilience of the Exposure Notifications System and ensure that these types of issues will not arise again. 89 67)Re: Major Issue: Google Play Services eating battering REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 08:36 Sent Items To: Ben Cloney; Fran Thompson; agreed 90 68)From: Ben Cloney Sent: 10 August 2020 08:24 To: Gar T. MacCriosta; Fran Thompson Subject: Re: Major Issue: Google Play Services eating battering Morning We should wait until later on today to start suggesting people reinstall the app so we know that all is good. Ben Ben Cloney Acting Head of Digital Communications Division, HSE 10 / 11 Cornmarket, Dublin, D08, P520 hse.ie/communications @hselive 91 69)Re: Battery Draining Summary of Issues REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 10:34 Sent Items To: Ben Cloney; Fran Thompson; Cian O'Maidin Colm Harte < damian.beresford Hi all Here's a summary of events updated; Reported Symptoms: battery draining and phones heating up First Report: Saturday 09/08 - 13:18 Twitter https://twitter.com/ismiseleenie/status/1292072609039998981?s=2 0 & https://www.reddit.com/r/ireland/comments/i5wo9v/covid app acting strange/?utm medium=android app&utm source=share Device Analysis: android reporting battery usage coming from COVID Tracker and Google Play Services Current Status: Awaiting hotfix to allowed list devices Root Cause: invalid TEK; an 18 byte TEK was passed in a file into ENS. TEKs are 16 bytes. The key in question was received from the Federation server and is a single key. The key was injected in as part of a test on Friday as NI had no keys to share. The key was in the correct format (visually correct) but was malformed containing 18 bytes. This test key wasn't removed after the test and was packaged in a download file and distributed after 23:00 Friday. Fix: • • • • • The invalid key has been removed from IE Key Registry in HSE The connection to the federation server has been disabled until Federation Server issues are resolved Hotfix developed by google to will handle invalid keys; this will flush the offending key and resume service; it also guards against subsequent invalid Hotfix received from Google and validated on allowed list devices Sunday PM ~12:30 to initial devices; NF confirmed that the hotfix worked and the affected phone recovered fromt the previous state Hotfix distributed by Google Sunday evening; o Google distributed to 25% by 01:40 AM and validated telemetry o Google subsequently pushed to 100% this will take up to ~12 hours Timeline: 92 • • • • • • • • • • • • • • Saturday DPER (Owen Harrison) raised with GMC at 15:47 GMC - escalated to FT & BC & NearForm 18:00 Escalation: Escalated to Google: Saturday 09/08 - 18:30 Google team responded 19:34 Analysis overnight Sunday Fix identified: Sunday AM Testing Fix Sunday ~09:30 pending receipt of update to allowed list device Hotfix updated on NF test device ~12:30 Hotfix pushed to targetted Irish devices ~19:00 Monday Google confirmed telemetry and initial distribution complete at 01:48 Problem Resolution: • • • Validate individual keys are correctly formed before publishing Validate Federation Server behaviour and resolve invalid key issues ENS will not try to reprocess invalid keys Other Info: • This appears to be an IE only issue; we have no reports from elsewhere Impacts: • • • 60K app deletion (10k new downloads which is a mix of net new people and people who experienced the issue trying to resolve it themselves. App Reported registrations number will be correct but will double count the users who deleted and redownloaded Loss of hair and sleep Media Coverage: • • • https://www.independent.ie/business/technology/covid-tracker-app-battery-drainproblem-to-be-fixed-hse-promises-39436486.html https://www.irishtimes.com/business/technology/google-to-roll-out-software-fix-toremedy-issue-with-covid-app-1.4325981 https://www.irishexaminer.com/news/arid-40029647.html Follow-on Actions: • • • 93 After action review including root cause of issue - BC/CH/GMC - Complete Press release agreed with google and distributed across channels - BC & Google Backend changes to validate TEKs (multiple checks CH to clarify) - CH • • • Slán Gar 94 Review Expleo interop testing report - NF/HSE Evaluate if there is a way to get the app to kill background processing on ENS - kill switch on processing that would allow app to eliminate the issue - GMC/CH to discuss and raise with google Alerting & monitoring for user related issues - BC to add GMC to soc media monitoring alerting 70)irish Times Clarification REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 10/08/2020 16:11 Sent Items To: Ben Cloney; Hi Ben, We need to clarify a statement from this article https://www.irishtimes.com/business/technology/fix-for-covid-tracker-approlled-out-to-all-android-users-google-says-1.4326646 Deleting the app, however, would also remove the list of bluetooth IDs from phones, meaning users who have done so and subsequently reinstalled it in an attempt to solve the battery issue will not be able to be contacted should a close contact from the previous 14 days test positive for Covid-19. Deleting the app does not remove the IDs, the app data is removed the ENS data can not be removed by deleting the app. Slán Gar 95 71)From: Ben Cloney Sent: 10 August 2020 15:54 To: Gar T. MacCriosta Subject: FW: App query Hi Gar Does having a match mean this person should have received a close contact alert? Ben Ben Cloney Acting Head of Digital Communications Division, HSE 10 / 11 Cornmarket, Dublin, D08, P520 Mobile hse.ie/communications @hselive Read our blog https://www.hse.ie/digital/blog 96 72)From: Sarah Murphy Date: 12 August 2020 at 14:02:46 IST To: "Gar T. MacCriosta" Cc: Niall Sinnott Subject: FW: FOR ACTION: To nominate an expert to the eHN Technical Work Group - ddl 07/08/2020 CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Gar, I’m just of a call with the eHealth Network and they were looking for some learnings on the battery issue with Android for the app to share with other MS on their confluence page. They mentioned it was discussed briefly at the technical group meeting today. Would you be able to provide me with a few paragraphs detailing the issue and how it was rectified. Kind regards, Sarah Sarah Murphy Information, eHealth & ICT External _ An Roinn Sláinte Department of Health Bloc 1, Plaza Míseach, 50 - 58 Sráid Bhagóid Íochtarach, Baile Átha Cliath, D02, XW14 Block 1, Miesian Plaza, 50 - 58 Lower Baggot Street, Dublin D02 XW14 __ 97 73)From: Niall Sinnott Sent: Monday 13 July 2020 15:56 To: eHEALTH-NETWORK@ec.europa.eu Cc: Gar T. MacCriosta Sarah Murphy Subject: RE: FOR ACTION: To nominate an expert to the eHN Technical Work Group - ddl 07/08/2020 I am nominating Mr. Gar McCriosta (copied) as the expert to this technical working group. He is our product expert in Ireland and can link in our tech development partners as required. I will continue to support and participate as required. Niall Niall Sinnott Head of eHealth & Information Policy, R&D and Health Analytics Division. An Roinn Sláinte Department of Health email: niall sinnott@health.gov.ie 98 74)Re: Background Processing Issues Briefing Note Briefing note.docx REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Mon 24/08/2020 15:49 Sent Items To: Ben Cloney; Fran Thompson; Hi Ben See section in Green, I think we should make it clear that we have idenitifed an issue, and that we've resolved it. I think the text doesn't call that out clearly enough; I would stick with the original explanation. Slán Gar The Covid Tracker App utilises the Google/Apple Exposure Notification System via an API. This briefing note describes the background processing issues experienced by Exposure Notification System (ENS) based apps and outlines our approach to resolving these issues. Several issues have been identified with background processing in ENS based apps by Apple and Google, the German Contact tracing app team and through our own investigations. Background processing enables a contact tracing app and the exposure notification system to work in the background when a contact tracing app is not open or the phone is not in use. A regular app is subject to the rules of the phone it is installed on. If the battery is low, background processing may be terminated by the operating system (iOS or Android), its ability to start and continuously run is limited by whatever permissions the operating system and the user has allowed. Examples of these types of issues include when specific power saving rules are in place or after restarting the phone before the app is opened may mean that either the app does not download the latest keys until the app is opened or that the phone does not share keys until the phone is sufficiently charged. To fix these issues Google and Apple have released updates that grant ENS based apps special permissions and ensure it runs more reliably in the background. We have also updated our app to ensure that the app will work at all times in the background. Due to the high level of privacy built into the app we do not have the same level of metrics and diagnostics that would normally be used in other apps. Instead, we rely on a global network of health services and developers to identify issues so we can continue to improve the effectiveness of the Covid Tracker App as the technology develops and resolve issues as quickly as possible. We have recorded that over 600 people have received close contact notifications through the app and unfortunately a number of these have gone on to test positive for coronavirus. This demonstrates that the app is working and we are encouraging anyone who has downloaded the Covid Tracker App to update their phone to the latest version of their phone’s operating system and download the updated version of the Covid Tracker App when it is released later on this week. Gar Mac Críosta Contracted to work on behalf of the Office of the Chief Information Officer OCIO - COVID Tracker Lead 99 75)Re: Draft Response to DOH re note attached REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Thu 27/08/2020 09:54 Sent Items To: Fran Thompson; Ben Cloney; Hi Some additions The number of registrations we report in the app is the total number of people who have completed the onboarding process. Since the launch this has been used as it is a better guide of activation of a user. Users who download an app and never complete the onboarding process are not protected as they haven't enabled the exposure notification system on their phone. The metric is a guide however as a result of the privacy preserving nature of the app this metric can't tell if you are new or returning, therefore it includes people who have downloaded, deleted and reinstalled the app. This metric worked as a proxy in the beginning but post the battery issue it led to a large number of people deleting the app and reinstalling or in some cases not reinstalling. We are moving to a new set of metrics based specifically on the number of devices that have the exposure notification system activated as a % of population. This gives us an effective coverage metric; we plan to release this metric in the next version of the app. Please note it will take some weeks before this metric stabilises as people will need to upgrade to the latest version of the app for us to collect this number. -- breakdown of numbers from below People delete apps for any number of reasons, space & storage issues, accidentally, cleaning up the phone, poor experience, trust or privacy concerns etc. We are surveying users to understand what they like, what they want and what they dislike, this input shapes the roadmap for COVID Tracker. Deletion and reinstallation is a common cycle for many apps. The deletions resulting from the battery issues on the weekend 7th August are the result of that specific event which regrettably impacted users. We are working to win those users back... Gar Mac Críosta Contracted to work on behalf of the Office of the Chief Information Officer OCIO - COVID Tracker Lead 100 76)From: Fran Thompson Sent: 27 August 2020 09:13 To: Gar T. MacCriosta; Ben Cloney Subject: Draft Response to DOH re note attached The functionality of the HSE’s COVID Tracker app puts user’s privacy and security first and foremost therefore many of the metrics the other apps utilise are not available to the HSE App. We have to rely on extrapolation from the number of users that choose to share metrics with the HSE and with Apple and Google. The figures provided below are a best estimated based on available data from the app store and play store. Google provides data on all installations, Apple only provides data on app installations for users that opt to share metrics. 27% of users choose to share metrics with Apple. The app has been installed 744k times on iphones and 909k times on Android devices. A total of 1.653 million. It has been uninstalled 157k times on iphones and 291k times on Android devices. A total of 376k. A simple figure is that the app is running on 1.277, however this does not and cannot take into account the number of installs or removals by a single device. As part of our latest version we will have one additional metric which will allow us to better understand the actual number of daily users the metric will measure the number of times keys are downloaded to devices. Once the upgrade is installed on most devices this will give us a clear picture but we estimate this picture will be about 2 weeks away from fully emerging due to the delays in people upgrading. 101 77)Re: Press release REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Thu 27/08/2020 20:21 Sent Items To: Ben Cloney; Cc: Fran Thompson; Looks fine suggest we run past Julie and Yul From google as a courtesy to keep them sweet. Sent from my iPhone On 27 Aug 2020, at 19:26, Ben Cloney wrote: Hi Guys Any comments? Ben Stay safe, protect each other, download the latest version of the COVID Tracker App. Today we have released an update of the Covid Tracker App based on feedback from people that are using it. From research carried out by the HSE with people who use the app we found that people want to see more detailed information about cases in their county focusing the current situation. The update also includes improvements required by the latest version of the Exposure Notification System (ENS) from Apple and Google. We are encouraging everyone to download the latest version of the COVID Tracker App and update the software on their phones to the latest versions from Apple or Google. The app has already notified nearly 800 people who have been in close contact with someone with coronavirus. This means they get the information they need quickly to reduce the spread of the virus and our Contact Tracing Team has been able to organise testing for them. Unfortunately some of these close contacts have gone on to test positive for coronavirus. The app is helping to keep the people that use it safe and protect everyone else. The more people that download the app the more effective it will be at helping us to stop the spread. We encourage everyone to download the app to help keep everyone stay safe, protect each other and stop the spread of the virus. 102 Note to editors: Apple and Google are continuously working to improve how the Exposure Notification System works. The latest version improves background processing for ENS based apps. Background processing enables a contact tracing app and the exposure notification system to work in the background when a contact tracing app is not open or the phone is not in use. A regular app is subject to the rules of the phone it is installed on. If the battery is low, background processing may be terminated by the operating system (iOS or Android), its ability to start and continuously run is limited by whatever permissions the operating system and the user has allowed. Examples of these types of issues include when specific power saving rules are in place or after restarting the phone before the app is opened and may mean that either the app does not download the latest Random IDs until the app is opened or that the phone does not share keys until the phone is sufficiently charged. To improve ENS performance Google and Apple have released updates that allow ENS to bypass these restrictions and ensure it runs more reliably in the background. We have also updated our app to ensure that the app will work at all times in the background. Ben Cloney Acting Head of Digital Communications Division, HSE 10 / 11 Cornmarket, Dublin, D08, P520 Mobile hse.ie/communications @hselive Read our blog https://www.hse.ie/digital/blog 103 78)Re: Mass deletion of the Covid 19 app REPLYREPLY ALLFORWARD Mark as unread Gar T. MacCriosta Fri 28/08/2020 17:59 Sent Items To: Ben Cloney; Cc: PRESS; Fran Thompson; Perfect Ben Sent from my iPhone On 28 Aug 2020, at 17:38, Ben Cloney wrote: Hi Oonagh Here is the response to Weckler’s query. The COVID Tracker App is currently installed on just over 1.2 million devices. As with any app some people will delete it and reinstall it and some people will choose not to. Understandably, when we had the battery issue on Android phones earlier in August a significant number of people who experienced the problem uninstalled the app. Those issues were fixed quickly and many of these users have now returned. We are now seeing a steady net daily growth of about 1500 people per day. With the exception of the days around the 10th August the data available from the app store and play store indicates that the number of users choosing to uninstall the app is in-line with other similar apps. We released a new update to the app yesterday based on feedback we have received from users which has already been installed on over 120,000 phones. We are encouraging everyone that has downloaded the app to update it to the latest version and also to ensure that their iOS or Android software is up-to-date. The app is working well and we have now recorded over 800 close contact alerts and unfortunately some of these people have gone on to be diagnosed with COVID-19. Ben Cloney Acting Head of Digital Communications Division, HSE 10 / 11 Cornmarket, Dublin, D08, P520 Mobile hse.ie/communications @hselive 104 Read our blog https://www.hse.ie/digital/blog 79)From: PRESS Date: Friday 28 August 2020 at 15:12 To: Ben Cloney Subject: FW: Mass deletion of the Covid 19 app Hi Ben Please see query below. Can you advise on a response? I’ve included the CEO’s comments from yesterday’s briefing. Briefly on the COVID app, you’ll have seen that there were some issues in terms of the app which we we’ll have addressed as part of – with Google and indeed Apple. We’ve a total of 1.65 million downloads on the app. There were of course some which were deleted and we are working through that. Probably about 1.2 million active users now but we are seeing people reload, etc. One of the difficult changes with us in terms of the measurement of data is that we have gone for a devolved suite of processes on our app so it’s not held centrally by ourselves, so extracting data, we can only extract what we have access to. The number of people, just to give you however the app is working. The number of people who’ve uploaded anonymous keys, so they’ve been contacted and uploaded their keys once they’ve been contacted as a positive case, is 308 people. The number of people who’ve received close contact notifications has been 663. In fact, it’s almost at 700 today. Daily checks on symptoms is roughly about a quarter of a million people per day. And we’ve also recorded at 600 people as mentioned – over 600 people have received the notification. And there have been a proportion of those who have actually been tested as positive. So, the app is working. It is an element of our key tool in terms of contact testing and tracing Thanks Oonagh Oonagh O’Mahony, Press Officer Press and Media Office ǀ Communications Division ǀ HSE ǀ 10/11 Cornmarket ǀ Dublin 8 ǀ D08X8C ǀ Email: press@hse.ie ǀ Tel: www.hse.ie @hselive youtube.com/HSEIreland ǀ 105 80)From: Adrian Weckler Sent: Friday 28 August 2020 15:03 To: PRESS Subject: Mass deletion of the Covid 19 app CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi there, I have a questions about the Covid 19 app. As per Paul Reid’s figures yesterday, does the difference between the downloads (1.65m) and “active users” (1.2m) mean 450,000 people downloaded the app? Thanks, Adrian Adrian Weckler Technology Editor, Irish Independent/Sunday Independent/Independent.ie 27-32 Talbot St, Dublin 1, D01 X2E1 Please think of the Environment before printing this email. Private Confidential & Privileged This email and any files and attachments transmitted with it are confidential and/or privileged. They are intended solely for the use of the intended recipient. Any views and opinions expressed are those of the individual author/sender and are not necessarily shared or endorsed by Independent Newspapers (Ireland) Limited or any associated or related company. The content of this email and any file or attachment transmitted with it may have been changed or altered without the consent of the author. If you are not the intended recipient, please note that any review, dissemination, disclosure, alteration, printing, circulation or transmission of this email and/or any file or attachment transmitted with it, is prohibited and may be unlawful. If you have received this email or any file attachment transmitted with it in error, please notify Independent Newspapers (Ireland) Limited, by emailing postmaster@independent.ie or contact : The Systems Manager Independent Newspapers (Ireland) Limited, 27 - 32 Talbot Street, Dublin 1, Ireland. Switchboard +353 (0)1 7055999 This footnote also confirms that this email message has been swept by 106 Microsoft Exchange Online Protection for the presence of computer viruses. Registered in Ireland No: 153066 107