NATIONAL TRANSPORTATION SAFETY BOARD Vehicle Recorder Division Washington, DC 20594 November 4, 2019 Onboard Image & Data Recorder Group Chairman’s Factual Report By Sean Payne 1. EVENT Location: Date: Aircraft: Operator: NTSB Number: Tempe, Arizona March 18, 2018 2017 Volvo XC90, Uber Technologies, Inc. HWY18FH010 2. GROUP A group was convened on May 4th, 2018. The group’s work resulted in a series of notes that contributed to the work in section 4 but was performed as a separate activity. The group consisted of the following members: Group Chairman: Sean Payne Mechanical Engineer National Transportation Safety Board (NTSB) Member: Jane Foster Electrical Engineer NTSB Member: Jennifer Morrison Highway Investigator NTSB Member: David Rayburn Highway Investigator NTSB Member: Rafael Marshall Human Performance Investigator NTSB HWY18MH010 Onboard Image Recorder Factual Report Page 1 Member: Noah Zych Head of System Safety Uber Advanced Technologies Member: Mikael Ljung Aust Technical Specialist Volvo Cars Sweden Member: Vincent D’Auria Manger of Regulatory Affairs Volvo Cars USA Member: Kevin Biesty Deputy Director Arizona Department of Transportation 3. SUMMARY For a description of this event, refer to Crash Summary, which can be found in the public docket for this accident. 4. DETAILS OF INVESTIGATION On March 30, 2018, the National Transportation Safety Board (NTSB) Vehicle Recorder Division received the following data files associated with an aftermarket dashboard camera system, as well as a system propriety to Uber vehicles: Recorder Manufacturer/Model: Recorder Serial Number: Janus V3 – Janus Cam N/A Recorder Manufacturer/Model: Recorder Serial Number: Uber Proprietary System N/A 4.1. Recorder Description Janus V3 – Janus Cam The Janus V3 is a vehicle mounted data and video recorder marketed toward fleet management applications. The device is capable of recording GPS, acceleration and video data. The device is capable of recording an inward and outward facing video input at 1280 x 720 pixels in resolution as well as a third video stream (typically rear facing) at 720 x 480 pixels in resolution. The manufacturer provides software to playback the video data in combination with the accelerometer and GPS data. HWY18MH010 Onboard Image Recorder Factual Report Page 2 Uber Proprietary Video Capture System The camera system consisted of ten cameras that gave a 360-degree view of the surrounding environment. The system included two cameras with narrow fields of view for long-range forward stereo imaging, one single-lens camera with a wide field of view for medium-range forward imaging, two single-lens cameras with wide fields of view for medium-range imaging of lateral areas, two single-lens cameras with wide fields of view for imaging the area behind the vehicle, and four surround-view cameras positioned for close-range imaging. The range at which an object could be detected depended on its size and visibility. The ADS processed the optical data from all cameras. The primary use of the imaging data from the forward camera was in the detection of obstacles and the classification of detected objects. The forward camera was also used to recognize traffic signs and the status of traffic lights. In addition to monitoring and real-time analysis of perceived objects, the cameras continually recorded the driving environment. 4.2. Video Files Dash Camera Uber Technologies provided the Tempe Police Department with the memory card from the after-market dashboard camera system. A copy of the dashboard recordings were later provided to the NTSB. The device recorded multiple video files in a proprietary format. The files were recorded at a rate of 30 frames per second and did not contain an audio track. The video files were recorded from three cameras, one facing forward of the vehicle, one facing inward that captured the vehicle operator (VO) and one facing rearward. Software associated with the dashcam system was obtained that allowed playback of the video files in conjunction with log files that recorded limited GPS location derived vehicle parameters at a rate of 2 Hz. Uber System Uber Technologies provided the Tempe Police Department with files from the proprietary camera system. Copies of the files were later provided to the NTSB. The files were congruent with the descriptions provided in section 4.1 of this report. 4.3. Timing and Correlation The times used in this report are expressed as Local Time of the accident mountain standard time, (MST). Timing of the transcript was established by correlating the video events to the impact of the vehicle with the pedestrian as seen on the vehicle’s log files. HWY18MH010 Onboard Image Recorder Factual Report Page 3 Specifically, the moment the driver impacted the pedestrian was synchronized with acceleration data which indicated the vehicle had a rapid change in acceleration. The accelerometer data was recorded at 2 Hz which likely did not capture peak acceleration data at impact and therefore has a resolution of +/- 0.5 seconds that correlates to the vehicle’s impact with the pedestrian. 4.4. Summary of Recording Contents Video events below are discussed in conjunction with vehicle log data. A definition of recorded vehicle log data parameters is presented in Appendix A. The VO’s gaze was classified throughout the inward facing video recording. Video transcription software was utilized to mark the times in which the VO glanced either upward or downward. The VO’s eye’s were clearly visible in the recording and the position of the eyes, not the VO’s head, was utilized in determining if the VO’s gaze was generally upward or generally downward. No effort was made to classify exactly what the VO was looking at other than noting the VO’s gaze was generally upward or downward. Speed and direction data are derived from GPS position and is subject to GPS precision and latency errors. Acceleration data was sampled at 2 Hz. At this sample rate it is unlikely that peak acceleration data was recorded. Additionally, data recorded for G Force X (accelerations in the X-direction) contained a recorded baseline of 1 g. This value should baseline at zero. Data for G Force Z (accelerations in the Z axis) baselined at around 0.5g. This value should baseline a 1g. In general, the reliability of the accelerometer data is questionable and is only utilized in the context of this report to assist in identifying the time of impact with the pedestrian. The accelerometer data has not been offset. Select video log data parameters are shown below along with select video events in figures 1 through 3. Vehicle GPS location and time data is presented on Google Earth overlays in figure 4, 5 and 6. Figure 1 is a plot consisting of three-axis acceleration, vehicle heading, vehicle speed and VO gaze. The plotted time interval is from 9:18:00 PM to 10:00:00 MST. Data was tabulated for the VO Gaze between 9:19:57.7 PM and 9:58:46.5 PM when impact occurred. In total, the VO’s gaze was tabulated for 38 minutes and 48.8 seconds of the accident trip. Of the 38 minutes and 48.8 seconds, GPS data indicated the vehicle was stopped for 7 minutes and 15.43 seconds. The total time the vehicle was moving was 31 minutes and 33.37 seconds. During the time the vehicle was moving, the VO’s Gaze was classified as “downward” 245 times for a total of 10 minutes and 46.2 seconds. In total, while the vehicle was moving HWY18MH010 Onboard Image Recorder Factual Report Page 4 and on a public roadway, the VO had her gaze “downward” approximately 34.1% of the time. The average amount of time the VO spent with her gaze downward was 2.56 seconds. The maximum duration the VO diverted her gaze downward was 26.47 seconds and occurred at 9:31:05 MST when the vehicle was traveling west on Salado Pkwy and then north along N. Mill Ave. across the N. Mill Ave bridge where the accident would later occur on the VO’s next circuit through Tempe. Figure 2 is a plot of the same parameters for a plotted time interval between 9:56:00 and 9:59:00 PM PST, approximately 3 minutes prior to the accident time. From 09:56:00 until the accident, the VO’s gaze was classified as “downward” 23 times, for a total of 58.43 seconds of the 2 minutes and 46.5 seconds prior to impact. Figure 3 is a plot of the same parameters for a plotted time interval between 9:58:30 and 9:59:00 PM PST. Figure 4 is a Google Earth overlay showing the entire accident trip. Since the route was a circuit, the graphic has been annotated with arrows pointing in the direction of travel of the vehicle as well as corresponding numerical leg segments. Figure 5 is a Google Earth overlay showing the region of the accident on N. Mill Ave. The track on the left was the previous circuit made by the vehicle. The track on the right is annotated with times and indicates approximately when the vehicle crossed under the underpass on N. Mill Ave and when it came to a stop around 09:59:02. Figure 6 is a representation of the accident vehicle path as displayed on Google Earth Street View. The view is oriented looking south on N. Mill Ave. The track on the left was the previous circuit made by the vehicle. The track on the right was the vehicle’s path around the time of impact with the pedestrian. 4.4.1. Inward Facing Video: The pertinent portion of the recording began around 7:38:15, the VO was already seated in the vehicle with the driver’s side door open. The VO appeared to be interfacing with the vehicle’s self-driving system (SDS). The VO was at times visible holding and drinking from a soda-like bottle. The VO continued to sit in the vehicle, at times the VO appeared to interact with an object was located near the VO’s lap area but was out of frame of the inward facing camera. Other actions by the VO during this time were generally unremarkable. At 7:45, the VO fastened the driver’s side seatbelt and around 7:46, closed the vehicle’s door. HWY18MH010 Onboard Image Recorder Factual Report Page 5 At 7:48:15, the VO was visible holding a mobile device. Moments later, around 7:48:30, the VO was visible holding a second mobile device. The VO placed the mobile devices in a backpack, unfasted the respective seatbelt and exited the vehicle around 7:49:00 and available recorded video files ended for this time period, creating a discontinuity in recorded material. The recorded files and video resumed at 9:04:40. The VO was seated in the VO’s seat with the seatbelt fastened, the VO was manually driving the vehicle in a parking garage structure. Around 9:05:00, the VO brought the vehicle to a stop and was visible handling the backpack that was visible earlier in the recording. At 9:05:10, the VO removed a mobile device from the backpack and placed in the region of the center console of the vehicle, out of frame of the camera. The VO zipped the backpack and placed the backpack in the rear seat of the vehicle. Moments later, the VO began manually operating the vehicle in the parking structure. The car exited the parking structure and around 9:06:15 the VO brought the vehicle to a stop in a surface parking lot area. During this time, the VO was visible gazing downward toward her lap manipulating an unseen object that was out of the camera’s frame. At times, the VO appeared to be interacting with an unseen object that was located out of the camera’s frame in the area of the passenger’s seat. During this time, the VO also occasionally interacted with an object in a location consistent with the vehicle’s SDS. At other times, the VO was gazing toward an unseen object in the center console area (out of view of the camera) and singing. Around 9:16:20, the VO made movements consistent with a handling and manipulating an unseen object, then movements consistent with placing an object near the center console of the vehicle. At 9:17:30, the VO adjusted the seatbelt and took a drink from a soda-like bottle. The VO placed the bottle out of view of the camera and returned their gaze to a region near the vehicle’s center console. At other times, the VO gazed in the vicinity of the dashboard and the vehicle’s SDS. By 9:19:10, the VO began manually driving the vehicle on the surface parking lot. The following actions were deemed relevant to the accident investigation. Any other actions were considered not remarkable. The transcription data are recorded in a tabular format and are intended to be used in conjunction with the plotted data in figures 1 through 3 as well as the exported tabular data in attachment 1. 9:19:55: Vehicle entered public roadway. 9:20:25: VO adjusted rearview mirror. 9:20:42: VO drank from soda-like bottle. HWY18MH010 Onboard Image Recorder Factual Report Page 6 9:23:23: VO drank from soda-like bottle. 9:24:04: VO smirked. 9:26:32: VO turned head and smirked at another Uber Technologies vehicle while stopped at stop sign. 9:32:56: VO smirked. 9:34:12: VO drank from soda-like bottle. 9:38:20: VO yawned. 9:39:16: VO yawned. 9:41:08: VO laughed. 9:43:32: VO yawned and adjusted body in seating position. 9:49:52: VO smirked. 9:50:10: VO appeared to be singing. 9:51:21: VO appeared to be singing. 9:52:49: VO yawned. 9:53:09: VO was singing in the manner of an outburst. 9:54:20: VO nodded "no" and then "yes.” 9:54:29: VO nodded "no." 9:55:05: VO yawned and then smirked. 9:57:00: VO yawned. 9:57:40: VO drank from a soda-like bottle. 9:58:46.2: VO reacted abruptly while looking forward of the vehicle. 9:58:46.5: Impact occurred. HWY18MH010 Onboard Image Recorder Factual Report Page 7 4.4.2. Outward Facing Video: The outward facing video was reviewed in synchronization with the other video and parametric data that was available. In general, due to the lighting conditions, camera resolution and camera light metering effects, the outward facing video is not representative of how the VO would have perceived the environment around the vehicle. When examining the video, the pedestrian is visible on the camera recording about 2 seconds prior to the collision. The pedestrian shoes were the objects visible on the recording. The wheels of the bicycle were next to be visible by the camera. The pedestrian was wearing white shoes, blue jeans and a dark coat. The bicycle was being pushed by the pedestrian who was walking. About 1 second prior to impact, the pedestrian looked over her right shoulder toward the vehicle. The bicycle did not appear to have reflectors on either wheel. A small headlight on the bicycle appeared to be lit but was perpendicular to the Uber vehicle’s path of travel at the time of the accident. HWY18MH010 Onboard Image Recorder Factual Report Page 8 Figure 1. Select vehicle parameters and inward facing video data for the entire accident trip. HWY18MH010 Onboard Image Recorder Factual Report Page 9 Figure 2. Select vehicle parameters and inward facing video data for approximately three minutes prior to impact. HWY18MH010 Onboard Image Recorder Factual Report Page 10 Figure 3. Select vehicle parameters and inward facing video data for approximately thirty seconds prior to impact. HWY18MH010 Onboard Image Recorder Factual Report Page 11 Figure 4. An overview of the vehicle’s entire trip on the night of the accident. HWY18MH010 Onboard Image Recorder Factual Report Page 12 Figure 5. An overview of the vehicle’s tracks in the vicinity of the accident. HWY18MH010 Onboard Image Recorder Factual Report Page 13 Figure 6. A representation of the vehicle’s path on Google Earth Street View. HWY18MH010 Onboard Image Recorder Factual Report Page 14 Appendix A Table 1 describes data parameters provided by the GPS device. Date, Time, Latitude and Longitude are recorded by the device. Direction and Speed are derived from the recorded parameters. VO Glance was classified using NTSB laboratory software which assisted in annotating the driver’s glance based on the movement of the VO’s eyeballs. In the tabular attachment for VO Glance, the presents of data indicates a state change, where “1” indicates upward and “0” indicates downward Table 1: Recorded and Derived Parameters Parameter Name Parameter Description Date Date for recorded data point (MM/DD/YYYY) Time Time (MST) for recorded data point (HH:MM:SS.00) LAT Recorded Latitude (degrees) LON Recorded Longitude (degrees) Direction Derived GPS True Heading Speed Derived GPS Groundspeed VO Glance Driver Gaze, where 1 indicates upward and 0 indicates downward HWY18MH010 Onboard Image Recorder Factual Report Page 15