From: To: Subject: Date: Lauren from Replica Eliot Rose [External sender]Replica Product Release Notes // Resident and Transit Reports by Demographic Groups Now Available Monday, November 18, 2019 12:38:42 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Eliot, At Replica, we are constantly working to make it easier to surface and expose essential planning and transportation metrics. We’ve rolled out new filtered Resident and Transit Reports on our Homepage. Resident Reports by Demographic Group You can now view Resident Reports by household income, race and ethnicity, and by car ownership status. Transit Reports by Demographic Group You can now view Transit Reports by household income, race and ethnicity, and by car ownership status. Check out the new reports at https://my.replicahq.com/. As always, please reach out if you have any questions or feedback. Your feedback continues to push us forward and helps Replica design and develop more useful tools. Interested in participating in a one-on-one Replica Discovery Session with a member of our team? Sign-up here and we'll be in touch soon. Thank you, The Replica Team Unsubscribe from our emails Powered by Intercom From: To: Subject: Date: Lauren from Replica Eliot Rose [External sender]Now available in Replica - Save and share reports Friday, November 08, 2019 7:45:31 AM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Eliot, Great news! You can now save the reports you create in Replica. Any report you save will be accessible through the “My reports” section of the Replica Homepage so you can view for later use or share with your colleagues. Check out this new feature at https://my.replicahq.com. Stay tuned for new features coming down the pipeline in the next few days!  As always, please reach out if you have any questions or feedback. Your feedback continues to push us forward and helps Replica design and develop even more useful tools. Thank you, The Replica Team Lauren from Replica ri fr Powered by Intercom From: To: Cc: Subject: Date: Chris Johnson Nathan Preheim Eliot Rose; Joe Broach Post-Acceptance Replica Data Delivery Schedule Thursday, October 31, 2019 2:01:21 PM Hi NathanQuick (and hopefully easy) question for you to answer… Given where we are now in the process/schedule, for which quarters (and years) can we expect Replica data for, give or take a quarter? I’m asking this because we’re hip-deep in budgeting for FY 20-21 right now and I’m trying to be as strategic as possible. Thanks! -Chris From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Benjamin Y. Bloom; Chris Johnson; Michelle Bellia; Robert Kirkman Re: Metro review of Replica agreement Monday, January 14, 2019 8:51:09 AM That sounds great. Please let us know if you have any questions. Thanks. Nick On Fri, Jan 11, 2019 at 5:40 PM Eliot Rose wrote: Thanks Nick! Looking forward to reviewing this. As for the agreement, we’re close to our final review and I don’t expect any changes that are going to be surprising to you. I’ll let you know if any come up or if we have questions about the documentation, but otherwise I’m inclined to let our partners review the current version once they sign the IGA before we send it back to you to minimize back and forth. That could take up to 3 weeks because it’ll take another week to get the IGA signed and then they have up to two weeks for review. I’ll try to keep the process moving and notify you ahead of time if there are significant questions that come up.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, January 11, 2019 2:23 PM To: Eliot Rose Cc: Benjamin Y. Bloom; Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Metro review of Replica agreement   Hi Eliot (and team)   Thanks for taking time to speak this week. As promised, I'm attaching a new version of the data disclosure document that includes our vendor audit process and a summary for the third party privacy audit we had conducted on Replica. I believe you all are going to turn back one final redline for the agreement and then we should be good to go. Please let me know if you have any questions.   Thanks everyone. Have a great weekend! Nick       On Mon, Jan 7, 2019 at 10:48 AM Eliot Rose wrote: Thanks for the heads-up, Nick. It looks like our team could do 10:30 or 11:30 as well if that works better for you. Since we only have a half hour it’d be helpful if everyone’s able to join in for as much of the call as possible.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 8:23 AM To: Eliot Rose; Benjamin Y. Bloom Subject: Re: Metro review of Replica agreement   Hi Eliot   Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed.    Sorry for the last minute change (typing from plane now). Talk soon.   Nick     On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments —seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: -          Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) -          Timeline for signing the contract -          Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Hi Eliot   Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below.   Attaching a few items here:   1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation.   We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created.    Thanks. As always, please let me know if you have questions.    Nick     On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread.   We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.    In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know.   Happy Holidays! Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments —but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Nick Bowden Benjamin Y. Bloom Eliot Rose; Chris Johnson; Michelle Bellia; Robert Kirkman Re: Metro review of Replica agreement Monday, January 07, 2019 9:00:01 AM Why don't we do this. We can do call from 1045-1115. We can spend first 15 minutes talking about timing, data, methodology and then spend the last 15 on contract once Ben jumps on. That should allow us to cover it all.  On Mon, Jan 7, 2019 at 10:55 AM Benjamin Y. Bloom wrote: Hi Nick,   Unfortunately I’m tied up on another call at that time. I am, however, currently generally available today after 11am PST.   Best, Ben   Benjamin Y. Bloom T: 416.369.4167   F: 416.864.9223 www.mindengross.com MERITAS LAW FIRMS WORLDWIDE     From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 11:51 AM To: Eliot Rose Cc: Benjamin Y. Bloom ; Chris Johnson ; Michelle Bellia ; Robert Kirkman Subject: Re: Metro review of Replica agreement   If it's possible, moving it up 30 minutes to 1030 would be great. Appreciate the flexibility. I will update the invite.    Thanks. Talk soon. Nick     On Mon, Jan 7, 2019 at 10:48 AM Eliot Rose wrote: Thanks for the heads-up, Nick. It looks like our team could do 10:30 or 11:30 as well if that works better for you. Since we only have a half hour it’d be helpful if everyone’s able to join in for as much of the call as possible.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 8:23 AM To: Eliot Rose; Benjamin Y. Bloom Subject: Re: Metro review of Replica agreement   Hi Eliot   Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed.    Sorry for the last minute change (typing from plane now). Talk soon.   Nick     On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments—seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: -          Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) -          Timeline for signing the contract -          Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Hi Eliot   Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below.   Attaching a few items here:   1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation.   We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created.    Thanks. As always, please let me know if you have questions.    Nick     On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread.   We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.    In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know.   Happy Holidays! Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Benjamin Y. Bloom; Chris Johnson; Michelle Bellia; Robert Kirkman Re: Metro review of Replica agreement Monday, January 07, 2019 8:51:17 AM If it's possible, moving it up 30 minutes to 1030 would be great. Appreciate the flexibility. I will update the invite.  Thanks. Talk soon. Nick On Mon, Jan 7, 2019 at 10:48 AM Eliot Rose wrote: Thanks for the heads-up, Nick. It looks like our team could do 10:30 or 11:30 as well if that works better for you. Since we only have a half hour it’d be helpful if everyone’s able to join in for as much of the call as possible.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 8:23 AM To: Eliot Rose; Benjamin Y. Bloom Subject: Re: Metro review of Replica agreement   Hi Eliot   Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed.    Sorry for the last minute change (typing from plane now). Talk soon.   Nick     On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments— seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: -          Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) -          Timeline for signing the contract -          Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Hi Eliot   Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below.   Attaching a few items here:   1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation.   We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created.    Thanks. As always, please let me know if you have questions.    Nick     On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread.   We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.    In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know.   Happy Holidays! Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments— but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Subject: Date: Nick Bowden Eliot Rose; Benjamin Y. Bloom Re: Metro review of Replica agreement Monday, January 07, 2019 8:23:48 AM Hi Eliot Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed.  Sorry for the last minute change (typing from plane now). Talk soon. Nick   On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments— seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: -          Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) -          Timeline for signing the contract -          Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Hi Eliot   Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below.   Attaching a few items here:   1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation.   We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created.    Thanks. As always, please let me know if you have questions.    Nick     On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread.   We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.    In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know.   Happy Holidays! Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Subject: Date: Valerie Eisen prvs=90087f558=Eliot.Rose@oregonmetro.gov Out of Office RE: Replica agreement - public records request Friday, January 04, 2019 7:50:07 AM Thank you for your email. Please note I am out of the office until Monday, January 14, 2018 with limited access to email. If your matter is urgent, please contact Alyssa Harvey Dawson at alyssa@sidewalklabs.com -Best regards, Valerie ---------------------- Valerie Eisen Legal Counsel*   c 647.929.2575 e valerie.eisen@sidewalklabs * on a client work assignment providing on-site client services through Fasken Martineau DuMoulin LLP. From: To: Subject: Date: Nick Bowden Eliot Rose Re: Replica agreement - public records request Thursday, January 03, 2019 7:23:42 AM Let's plan on 11am PST Monday to have the discussion to wrap up the contract. I will send an invite now to you, feel free to share with whomever on your team that will attend. Thanks. Nick On Wed, Jan 2, 2019 at 7:47 PM Eliot Rose wrote: Great! We’re available Monday 11-1, Wednesday 9:30-10:30 and 12-1:30, and Thursday 11-2. Glad to know that the data are getting close to ready to go.   I looked over the contract with our attorneys and I think that’s in pretty good shape, just a few minor issues to discuss. I’ll send an updated versions with those issues flagged tomorrow. I’m also having our technical folks review the documentation you sent so we can let you know if we have any remaining questions on that.   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 1:18 PM To: Eliot Rose Subject: Re: Replica agreement - public records request   I think we can just focus the call on the contract and try to close the loop on the outstanding points. The FOIA process is pretty much dictated, so no need to discuss until you would like to.    Understand on the term sheet, no worries.    Thanks again for all your help on these items.   Nick     On Wed, Jan 2, 2019 at 1:08 PM Eliot Rose wrote: Thanks! Reviewing everything now and then talking with our legal at 2:30. Definitely going to check in on the term sheet – I understand your concerns but as I understand it that’s a signed agreement and therefore subject to disclosure.   Sending some times for us to talk later today once I get to check in with the team. Is your plan for the call to just loop in the attorneys and focus on the agreement, or should we plan to cover any outstanding issues re: the public records request and documentation as well?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 10:53 AM To: Eliot Rose Subject: Re: Replica agreement - public records request   Sorry about that! Just shared.   On Wed, Jan 2, 2019 at 12:47 PM Eliot Rose wrote: Hello Nick:   I just tried to open this up and it looks like you shared it with my Metro acct., which isn’t Google-enabled. Can you please share with me at ?   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot   We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data.   As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues.    Thanks again and sorry for all of the trouble! Happy new year! Nick       On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request   Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Thanks, Eliot! We appreciate it.   I will plan to review quickly once received.   Have a great weekend and happy holidays!   Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Hi Eliot   Happy holidays to you all as well.   Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary:   1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases  3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense)   Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started.   Thank you. Nick           On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team:   Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them.   Please let us know if you have any questions or concerns. -Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736               503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Valerie Eisen prvs=8997903eb=Eliot.Rose@oregonmetro.gov Out of Office RE: Replica agreement - public records request Wednesday, January 02, 2019 5:35:20 PM Thank you for your email. Please note I am out of the office until Monday, January 14, 2018 with limited access to email. If your matter is urgent, please contact Alyssa Harvey Dawson at alyssa@sidewalklabs.com -Best regards, Valerie ---------------------- Valerie Eisen Legal Counsel*   c 647.929.2575 e valerie.eisen@sidewalklabs * on a client work assignment providing on-site client services through Fasken Martineau DuMoulin LLP. From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Michelle Bellia; Valerie Eisen Re: Replica agreement - public records request Friday, December 21, 2018 6:36:57 PM Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Hi Eliot   Happy holidays to you all as well.   Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary:   1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases  3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense)   Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started.   Thank you. Nick           On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team:   Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the emails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them.   Please let us know if you have any questions or concerns. -Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736               503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Marie Hlavaty prvs=88620eadc=Eliot.Rose@oregonmetro.gov Please read RE: Metro review of Replica agreement Friday, December 21, 2018 11:48:11 AM Hello. Thank you for your message. I am no longer with Sidewalk. Please contact Alyssa Harvey Dawson (alyssa@sidewalklabs.com) or Valerie Eisen (valerie.eisen@sidewalklabs.com) who will be able to address your inquiry. Thank you. Regards, Marie From: To: Subject: Date: Marie Hlavaty prvs=88557b327=Eliot.Rose@oregonmetro.gov Please read Re: Replica agreement - public records request Thursday, December 20, 2018 3:34:49 PM Hello. Thank you for your message. I am no longer with Sidewalk. Please contact Alyssa Harvey Dawson (alyssa@sidewalklabs.com) or Valerie Eisen (valerie.eisen@sidewalklabs.com) who will be able to address your inquiry. Thank you. Regards, Marie From: To: Subject: Date: Nick Bowden Eliot Rose Re: Metro review of Replica agreement Friday, December 07, 2018 10:17:55 AM Perfect. On Fri, Dec 7, 2018 at 12:13 PM Eliot Rose wrote: Correct! Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 07, 2018 10:10 AM To: Eliot Rose Subject: Re: Metro review of Replica agreement That's totally fine. I assume the group is all public sector folks that will be involved and have access to Replica? On Fri, Dec 7, 2018 at 12:05 PM Eliot Rose wrote: I’ve been pulling together the data that we need and working with our partners who collect more detailed data sets. Any objection to me sharing the acceptance criteria doc with one of our regional data working groups? Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:40 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Also, forget to attach two items. The first is the most up-to-date template for Acceptance Criteria. This is the format we are using with both NYC and Chicago. Second, I attached the recently approved calibration report for KC Season 1. This will give you an idea of how we report against the acceptance criteria each quarter. On Wed, Nov 28, 2018 at 11:22 AM Nick Bowden wrote: Hi Eliot - Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry! Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Subject: Date: Nick Bowden Eliot Rose Re: Metro review of Replica agreement Friday, December 07, 2018 10:10:41 AM That's totally fine. I assume the group is all public sector folks that will be involved and have access to Replica? On Fri, Dec 7, 2018 at 12:05 PM Eliot Rose wrote: I’ve been pulling together the data that we need and working with our partners who collect more detailed data sets. Any objection to me sharing the acceptance criteria doc with one of our regional data working groups?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:40 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Also, forget to attach two items. The first is the most up-to-date template for Acceptance Criteria. This is the format we are using with both NYC and Chicago.    Second, I attached the recently approved calibration report for KC Season 1. This will give you an idea of how we report against the acceptance criteria each quarter.      On Wed, Nov 28, 2018 at 11:22 AM Nick Bowden wrote: Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Marie Hlavaty; Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Re: Metro review of Replica agreement Friday, November 30, 2018 3:43:30 PM Sorry I missed everyone! Thank you all for carrying on in my absence.  That sounds like a great plan. Happy to discuss anytime next week. Have a great weekend. Nick On Fri, Nov 30, 2018 at 5:53 PM Eliot Rose wrote: Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Robert Kirkman Nick Bowden; Eliot Rose Chris Johnson; Marie Hlavaty; Michelle Bellia; Valerie Eisen RE: Metro review of Replica agreement Thursday, November 29, 2018 10:24:24 AM I’m actually out of the office that day, but can join remotely.  Please make it a conference call or Hangout.   ~ Robb   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson ; Marie Hlavaty ; Michelle Bellia ; Robert Kirkman ; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Re: Metro review of Replica agreement Thursday, November 29, 2018 10:13:37 AM I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Michelle Bellia; Robert Kirkman; Chris Johnson Re: Metro review of Replica agreement Monday, November 26, 2018 12:02:04 PM No worries, we can run the comparison, just wanted to check before we started.  Hoping to have something back to you by end of week. Thanks! Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Michelle Bellia; Robert Kirkman; Chris Johnson Re: Metro review of Replica agreement Sunday, November 25, 2018 11:57:07 AM Thanks, Eliot.  You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.  If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Subject: Date: Nick Bowden Eliot Rose Re: Replica Portland: Next Steps Thursday, November 08, 2018 12:12:34 PM Hi Eliot! Just wanted to check and see how this is coming along. Let me know if you have any questions. Thanks. Nick On Thu, Oct 25, 2018 at 12:31 PM Eliot Rose wrote: Thanks Nick! That approach sounds good to me. We’ll work with our attorney to outline those use cases—I don’t think that they will be a surprise to you based on our conversations so far, but we want to make sure that they’re covered under the agreement.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, October 25, 2018 6:37 AM To: Eliot Rose Cc: Chris Johnson Subject: Re: Replica Portland: Next Steps   This reference is a vestige from an older version of the SaaS agreement that didn't get removed from the current version. It was intended to reference the MOU / Term sheet. It's easiest to ignore for now and we will redline out once we get your review. On the use cases you are anticipating, it's likely easiest to include those in your redline explicitly and we will accept the changes upon receipt. Does this work?   Thanks. Nick   On Wed, Oct 24, 2018 at 7:09 PM Eliot Rose wrote: PS – Can you please provide an example of the Order Forms and Documentation that are referenced in the contract? The language in Section 4.4 appears to prohibit some of the use cases that we’re interested in, but it also allows for exceptions as detailed in those two documents, so it will expedite our review if we can see them too.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, October 22, 2018 12:53 PM To: 'Nick Bowden' Cc: Chris Johnson Subject: RE: Replica Portland: Next Steps   Thanks for checking in, Nick. We’ve done both a technical and legal review of the agreement, and want to take one more pass before we get a redlined version back to you. That should give us some time for back-and-forth before Portland and TriMet are ready to enter into an MOU with Metro, which should happen in early November, so that we can get the agreement signed.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, October 22, 2018 11:51 AM To: Eliot Rose Cc: Chris Johnson Subject: Re: Replica Portland: Next Steps   Hi Eliot   Touching base on the agreement. How are things coming along on your end? We are planning to start working on the uncalibrated Portland Replica sometime late next month.    Thanks! Nick   On Tue, Oct 2, 2018 at 4:48 PM Nick Bowden wrote: Hi Eliot   Our legal has updated the contract (attached) to address some of the issues you notes.    Section 7.2 now makes it explicit that you can download or export data at any time. Sections 4.4 and 4.6 are the places where restrictions are identified for data use.    On any of these items, any suggestions that you have (not covered by the updates) are welcome. I want to make sure we are aligned with what you have identified.   Thanks! Nick     On Mon, Sep 24, 2018 at 6:42 PM Eliot Rose wrote: Hello Nick:   We’ve been reviewing the Replica SaaS contract that you sent over, and we have a couple of questions. ·         It’s important to our team that we be able to download and host the Replica data, and we’ve discussed that before with you and gotten the impression that we’re going to be able to do that through the pilot. The only place we see reference to downloading the data in the SaaS agreement is in the termination clause—is there any language about our ability to download data during the pilot elsewhere in the contract or its exhibits, like in the Order Form and Documentation (which weren’t included in full in the draft that you sent)? ·         Similarly, where can we find more information on restrictions on the access to the dataset? The contract refers to limitations on the number of users and on queries, but doesn’t detail what those limitations are. ·         Lastly, is there any information that you can share with us on the technical specs of the Replica database—approximate size for a region like ours, format, etc.? Our enterprise data manager, Robb, is cc’d here and can fill in some of the specifics if you need more information about what we’re looking to know.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, September 17, 2018 8:29 AM To: Eliot Rose Subject: Re: Replica Portland: Next Steps   Acceptance Criteria Template attached. Some of the numbers can't actually be determined until we do a variance analysis on the ground truth data. As an example, if the natural variance in freeway loop detectors is 18% (actually number from Kansas City), then the threshold for that particular metric of course can't actually be below 18%.    Let me know if you have any questions. In terms of to-do's / order of work: We should be able to work the SaaS agreement mark ups / acceptance criteria in parallel. The SaaS mark ups will likely come from legal, which acceptance agreement will come from you / others at Metro.  Assuming we can get everything finalized in the next 45 days, we will start the uncalibrated build in early November-ish. The only thing we will need to do this is GTFS for all transit.  Let's say early December is when we will start the data request from local agencies with a plan to have it collected by mid-Jan. Normalized by early feb. That gives us two months to do calibration runs, with final, fully calibrated Replica done by end of March.  Thanks! Nick   On Thu, Sep 13, 2018 at 11:10 PM Eliot Rose wrote: Thanks Nick! I’ll get you a data sample shortly. Looking forward to seeing the acceptance criteria.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, September 10, 2018 12:55 PM To: Eliot Rose Subject: Re: Replica Portland: Next Steps   Hi Eliot   Thanks for sending this and sorry for delayed response. Answers inline below:       On Tue, Sep 4, 2018 at 5:57 PM Eliot Rose wrote: Hello Nick:   I’ve been moving forward getting an IGA in place on our side to fund the Replica pilot, but I haven’t gotten the chance to follow up on your requests yet. Here’s a quick update and some questions to help keep this moving. 1.       Contract: I looked into it and we don’t have a standard SAS contract template, nor even a comparable contract that we feel like we can base this one on, so we should start from your sample contract. Attached is the latest version I have of that; is this the right one to be using? I've attached the most recent version, I *think it's the same as the one attached but did not go line by line.  2.       Acceptance criteria: same question as above; is this the right version to be working off of? No, we have an updated version, it continues to get modified as learn new things with each effort. I am working to finish today, and then will send your way.  3.       Sample data: working on this right now. Transit data is pretty straightforward, but in our region there’s no single regional source of traffic data—Metro compiles some, but for the complete set we’ll have to pull together data from 3 counties and the city of Portland. Do you want me to send you the different data sets as I access them so that you can think with us about how to best mash them up in this process, or just get you the regional dataset? Any one sample is fine for now. We will ultimately handled the normalizing or meshing, we don't require agencies to do any work, just send us what they have. I am just trying to get a feel for what's out there at this point. Anything you have is great.    Also, we continue to be interested in any documentation you’ve developed about the data sources and methodology used in Replica. You sent a research paper that covers the use of cell data well, but doesn’t discuss other sources in detail. I’m particularly interested in hearing more about how Replica incorporates Streetlight data, and how the end product differs from Streetlight. I have a sense of that from speaking to you, but some of the folks that are going to be approving the contract are well aware of Streetlight, and it will help to be able to articulate to them what the additional value add of Replica is.   We are finalizing the data disclosure document for Kansas City right now and once complete I will send your way. We actually use a combination of streelight, safegraph, cell companies, consumer marketing data, google data, and census data. So, the first component of differentiation is in the composite of data across multiple sources which provides a much larger samples and makes de-biasing the data much easier. Secondly, we produce a full-scale model, that represents all trips vs. aggregate counts with small factoring, which is a pretty significant effort / difference. Thirdly, trip purpose (why people travel) but household characteristics are associated with every trip, which means who is traveling is equally important as how and why people travel. Finally, and most important in my opinion, is we calibrate Replica to world conditions to ensure it can be used in policy making vs. just a source of additional data.    Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, August 21, 2018 3:38 PM To: 'Nick Bowden'; Chris Johnson; Robert Kirkman Subject: RE: Replica Portland: Next Steps   Hello Nick:   Sorry to be a little slow in responding to this; I was out on vacation for a long weekend. I think that you may have forgotten to attach the updated acceptance criteria. I’m attaching the most recent version I have for reference.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, August 17, 2018 1:46 PM To: Eliot Rose; Chris Johnson; Robert Kirkman Subject: Replica Portland: Next Steps   Good Afternoon   Thanks for taking time to talk this week. We are very excited to get started. Here are the follow up items from our call: 1. Contract Form / Process: You guys are going to check on using your own standard agreement vs. using our standard agreement. Once determined, we can start the procurement / legal review process.  2. Acceptance Criteria Document: Updated Template Version Attached. It takes a little localization, but by and large, about 90% consistent across regions.  3. Data Collection Process: I've attached a data requirements checklist to the email as well. This provides a run down of all the data required across all participating agencies. Again, the gathering of this will be handled in a very programatic way, so don't fret the process at this point.  4. Sample Date: If you have a sample set of traffic and transit counts you can share in their raw format, that would be great. Please let me know if I forgot anything. Have a great weekend.   Nick From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Sean Johnson; Lauren Massey; Chris Johnson Re: [External sender]Re: Portland Replica questions Wednesday, November 20, 2019 10:22:45 AM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! 1. The raw trip table, and the format of that deliverable, will be different than the fields and attributes available for download within the application. We're polling a handful of our users to best understand the schema they'd like to receive it in, and from that, we'll do our best to establish a format to gets you most of what you need. 2. High privacy risk trips and other unique activity sequences get removed during the calibration process. You won't find these data points in the raw data or through the web application. 3. OHAS. I've combed through all of our online Google Drive folders and I'm not able to locate this data set. If it was shared prior, I regret that I do not have access to it. Is that something you can share again? 4. I like your approach to giving folks a high-level overview prior to soliciting feedback on interest use cases and capabilities. Delivering those completed use case spreadsheets after Thanksgiving is fine. We're excited to start building out onboard training materials and the results from those documents will be essential to ensure we dive into the specific areas that hold most value for each agency. Thanks! -Nathan On Tue, Nov 19, 2019 at 4:03 PM Eliot Rose wrote: Thanks Nathan. A couple of follow-ups below. We’re writing our policies for who gets access to Replica data via different formats and what they have to do to protect it, and your answers will help us there. 1. I get that you’re trying to match the trip table to the format we use in our model. My question is, will the data that people are able to query and download from the web tool be in the same format that’s in the trip table? The web tool data that I can currently download via the KC/Chicago builds is more limited in the attributes and level of detail it contains than what we expect from the trip table. That’s not necessarily a bad thing; we just need to understand what level of protection the web tool data requires. 2. Am I correct that outlier trips will not be visible in the web tool? Also, is there anything you’re waiting on from us? You had a question about the OHAS data, but Chris and Joe were both under the impression that we shared that already. Maybe it got lost in the shuffle when the Drive folders were moved? We’re having a meeting next Tuesday at Metro where we give likely users a high-level overview of Replica and let them know about upcoming training opportunities, and also have them fill out the use case sheets, so I hope to have a bunch of completed sheets for you a little after Thanksgiving. I looked them over and they seem fine, but they’re pretty detailed, so I think that being able to show people the tool before having them fill the sheets out will get you better feedback. Thanks, Eliot Eliot Rose Technology Strategist Metro From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, November 19, 2019 10:20 AM To: Eliot Rose Cc: Sean Johnson; Lauren Massey; Chris Johnson Subject: [External sender]Re: Portland Replica questions CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Team Portland! What a weekend indeed. The Big Red was embarrassed at home by the Badgers. Convenient day to be out of the office - I couldn't handle the shame, Chris. :) 1. In terms of the trip table, we'll do our best to emulate the schema of your trip table. Can't promise that we'll be able to match it exactly. But we will do our best. 2. We work hard to remove all outlying data sets during the calibration process. Remember, we create personas which are abstractions of real people with real movements. Also, access to the trip table should be extremely limited. You'll have to decide who gets access to it, but I'd encourage you to think very carefully about that. 3. The more the merrier! Happy to do another session. Also, any feedback on the use case template I sent over? The more info you can provide, the better we can tailor those onboard sessions. Thanks, Nathan On Mon, Nov 18, 2019 at 7:02 PM Eliot Rose wrote: Hello Nathan and team: Hope y’all enjoyed a restful day of not checking in with us, and that Nathan’s entire day off was restful. Chris was bummed that he didn’t get to boast about the Badgers’ victory over the Huskers though. Chris and I caught up with each other on this project after both of us being out for a spell today, and we had a couple of questions. These will help us get the right policies in place about how our staff and partners use Replica. · Is the trip table in the same format as the data that you get when you query data through the web tool? Or is it more detailed? · Remind us - how does Replica remove / restrict access to potentially identifiable outlier trips? I know we’ve discussed the case of someone who might be identifiable in the data because they have a unique trip (e.g,. they’re that one person who commutes 20 miles each way to downtown along a route with no infrastructure). How do you identify / treat those trips in the data? Do they show up at all via the web tool or in the trip table? · Portland’s Replica info sessions are getting oversubscribed – if they can’t accommodate the demand in a single 30-person training session would you be open to doing a 2nd session at Portland when you visit? If not, I can tell them to call in the bouncers and velvet ropes. Thank you, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Joe Broach Nathan Preheim Eliot Rose; Chris Johnson Portland Replica // C-Tran data Tuesday, November 12, 2019 4:27:24 PM Hi Nathan,   One more piece of data news. C-Tran was finally able to wrangle their ridership data for Q3 & Q4 2018, and I’ve uploaded to the shared folder. We realize these are coming in late, and it’s a smaller agency. Given that some C-Tran targets are in the acceptance criteria, and that the cross-river trips are important regional flows, let us know how you want to proceed.   Cheers! joe From: To: Cc: Subject: Date: Joe Broach Nathan Preheim Eliot Rose; Chris Johnson Portland Replica // summary of additional data needs Tuesday, November 12, 2019 1:03:06 PM Hi gang,   I’m going to try to merge all these emails into one thread as I work through the asks.   Metro/Joe action items in BLUE Follow up questions in RED   · Freight data o Our main ongoing source of freight ground truth are Metro’s classified vehicle counts. You should have those from 2017 (next update will be 2019). The difference between light and all vehicles should give you most commercial truck traffic but, yeah, not light commercial traffic like vans, etc. o I’ll check with our lead freight modeler on what else is available o Are there specific data that would be most valuable to you? I’m not totally clear on how you’d use the freight data, so any tips would help me locate the data you need. · Trip table format o I’m checking with a couple of folks here on what’s most important. Obviously, we don’t generate a comparable trip table from our travel model, so I can’t give you a simple structure to match. We could use the household travel survey structure, but that’s dated now, and I’m not sure how useful it would be. I’ll update later today and then let Chris chime in if he has additions. o What network(s) do you use for trip assignment? OSM? I’m guessing we’ll want to compare results with our modeling networks, so we’d need to conflate the two…in case you all have any ideas. · OHAS survey data o Checking on how we can share most easily. I’ll get back today.      Please add in if I missed anything! After today, I’m OOTO until Thurs 11/21 and largely without email 11/14-11/19.   -joe   From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Joe Broach; Chris Johnson Re: [External sender]Portland Replica // Freight ground truth data? Monday, November 11, 2019 12:27:52 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot, Sorry, one more question. I know that we'd like to deliver a copy of the trip table. It's gonna be enormous - heads up. Likely 2GB per day zipped. What would really help us is to have a schema of your existing table to ensure that we're delivering everything you want/need. I can't promise we can accommodate it perfectly, but knowing how your trip table looks today will help get close. Is that something you can share with me? Thanks, Nathan On Mon, Nov 11, 2019 at 2:11 PM Eliot Rose wrote: Thanks Nathan! Adding Chris in, I know we have a new freight model but I'm not sure what data feeds it. No call today-- Metro is off for Veterans Day. Sorry I didn't catch that sooner. Any updates to share via email? Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Nov 11, 2019, at 9:54 AM, Nathan Preheim wrote: CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hello Joe and Eliot! We're all hands on deck getting your model tuned and ready to go. Curious if you all have access to freight ground truth? We'll model it without, but always ask if you have access to ground truth to help us better calibrate the results. In our world, freight includes heavy, medium, and lightweight commercial vehicles. Let me know! Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Joe Broach; Chris Johnson Re: [External sender]Portland Replica // Freight ground truth data? Monday, November 11, 2019 12:25:29 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! Nothing significant. We're processing through all of the ground truth and geo files. We're having an internal discussion tomorrow to ensure we've got all requirements locked down. I'll share a document with you tomorrow that identify exactly what we'll be delivering above and beyond the acceptance criteria. Thanks, Nathan On Mon, Nov 11, 2019 at 2:11 PM Eliot Rose wrote: Thanks Nathan! Adding Chris in, I know we have a new freight model but I'm not sure what data feeds it. No call today-- Metro is off for Veterans Day. Sorry I didn't catch that sooner. Any updates to share via email? Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Nov 11, 2019, at 9:54 AM, Nathan Preheim wrote: CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hello Joe and Eliot! We're all hands on deck getting your model tuned and ready to go. Curious if you all have access to freight ground truth? We'll model it without, but always ask if you have access to ground truth to help us better calibrate the results. In our world, freight includes heavy, medium, and lightweight commercial vehicles. Let me know! Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Nathan Preheim Joe Broach Eliot Rose [External sender]Portland Replica // Freight ground truth data? Monday, November 11, 2019 9:54:42 AM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hello Joe and Eliot! We're all hands on deck getting your model tuned and ready to go. Curious if you all have access to freight ground truth? We'll model it without, but always ask if you have access to ground truth to help us better calibrate the results. In our world, freight includes heavy, medium, and lightweight commercial vehicles. Let me know! Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Attachments: Nathan Preheim Eliot Rose; Joe Broach; Chris Johnson Lauren Massey; Sean Johnson [External sender]Portland Replica // Use Cases Monday, November 04, 2019 3:30:46 PM Portland Replica _ Use Cases.xlsx CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! I just shared a cloud version of this document with you. I'm also attaching an Excel version. As we start to prepare for onboarding, the more information we can gather from Portland agencies about relevant use cases the better. This simple worksheet helps us do just that. Feel free to distribute far and wide. An agency can complete it, a division can complete it, a specific role can complete it, or best yet, many different individuals can complete it. All we ask is that we map each completed sheet to a role and agency. Individuals should save their files with this kind of naming format... NP_Planner_Trimet KL_Modeler_Metro FD_Policy Analyst_City of Portland You get the idea. The first two letters are initials. Enough detail for you all to identify, but nearly anonymous for Replica. Does that make sense? As I mentioned, California Air Resource Board currently as a high water mark on returned worksheets at 14. I'll bet you can up that by at least 1. ;) Thanks! -Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com From: To: Cc: Subject: Date: Nathan Preheim Chris Johnson Eliot Rose; Joe Broach [External sender]Re: Post-Acceptance Replica Data Delivery Schedule Thursday, October 31, 2019 2:25:20 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Chris! Absolutely an easy question to answer. Here's how things will play out, assuming you all formally accept the model in December 2019.... We are launching with Fall 2018 and Winter 2018/2019. It's a 1 year agreement, from December 2019 until December 2020. Remember, we don't start the clock until the model has been formally accepted. During the one-year term you can expect the following seasonal refreshes.... Spring 2019 Summer 2019 Fall 2019 Potential Winter 2019/2020 I can guarantee 5 seasons worth of data, and depending on how systematized your ground truth data is we might be able to squeeze in Winter 2019/2020 just prior to the expiration date. Is that helpful? Thanks, Nathan On Thu, Oct 31, 2019 at 4:01 PM Chris Johnson wrote: Hi NathanQuick (and hopefully easy) question for you to answer… Given where we are now in the process/schedule, for which quarters (and years) can we expect Replica data for, give or take a quarter? I’m asking this because we’re hip-deep in budgeting for FY 20-21 right now and I’m trying to be as strategic as possible. Thanks! -Chris -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Lauren Massey; Sean Johnson [External sender]Portland Replica Onboard Tuesday, October 29, 2019 1:43:20 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Chris Johnson; Robert Kirkman Re: Replica Portland: Next Steps Friday, November 09, 2018 2:33:46 PM Appreciate the heads up, I don't anticipate any of this being an issue.  Will look for agreement next week. Thanks and have a great weekend! nick On Fri, Nov 9, 2018 at 4:21 PM Eliot Rose wrote: Thanks for reaching out, Nick.   We’ve got a close-to-final draft of the contract to share; it’s going through one more round of technical review and then we hope to send it middle of next week. In case it helps to have a heads-up on what you’ll see in that draft:   Revising this has been a long process because the contract language, particularly the usage restrictions in section 4.4, prevents many of the uses of the Replica data that we’re most interested in and that we’ve previously discussed with you. Our main change is to clarify that the following use cases are exempted from the restrictions in 4.4: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   We are also concerned about the broad requirements to “use  commercially  reasonable  efforts  to  prevent  unauthorized  access to or use of Services and Content” (4.3d) and broad restrictions on building similar products in 4.4n. We’ll be looking to SWL to refine those to focus on the specific security cases and derivative products that you’re concerned with.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 08, 2018 12:12 PM To: Eliot Rose Subject: Re: Replica Portland: Next Steps   Hi Eliot! Just wanted to check and see how this is coming along. Let me know if you have any questions.   Thanks. Nick   On Thu, Oct 25, 2018 at 12:31 PM Eliot Rose wrote: Thanks Nick! That approach sounds good to me. We’ll work with our attorney to outline those use cases—I don’t think that they will be a surprise to you based on our conversations so far, but we want to make sure that they’re covered under the agreement.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, October 25, 2018 6:37 AM To: Eliot Rose Cc: Chris Johnson Subject: Re: Replica Portland: Next Steps   This reference is a vestige from an older version of the SaaS agreement that didn't get removed from the current version. It was intended to reference the MOU / Term sheet. It's easiest to ignore for now and we will redline out once we get your review. On the use cases you are anticipating, it's likely easiest to include those in your redline explicitly and we will accept the changes upon receipt. Does this work?   Thanks. Nick   On Wed, Oct 24, 2018 at 7:09 PM Eliot Rose wrote: PS – Can you please provide an example of the Order Forms and Documentation that are referenced in the contract? The language in Section 4.4 appears to prohibit some of the use cases that we’re interested in, but it also allows for exceptions as detailed in those two documents, so it will expedite our review if we can see them too.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, October 22, 2018 12:53 PM To: 'Nick Bowden' Cc: Chris Johnson Subject: RE: Replica Portland: Next Steps   Thanks for checking in, Nick. We’ve done both a technical and legal review of the agreement, and want to take one more pass before we get a redlined version back to you. That should give us some time for back-and-forth before Portland and TriMet are ready to enter into an MOU with Metro, which should happen in early November, so that we can get the agreement signed.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, October 22, 2018 11:51 AM To: Eliot Rose Cc: Chris Johnson Subject: Re: Replica Portland: Next Steps   Hi Eliot   Touching base on the agreement. How are things coming along on your end? We are planning to start working on the uncalibrated Portland Replica sometime late next month.    Thanks! Nick   On Tue, Oct 2, 2018 at 4:48 PM Nick Bowden wrote: Hi Eliot   Our legal has updated the contract (attached) to address some of the issues you notes.    Section 7.2 now makes it explicit that you can download or export data at any time. Sections 4.4 and 4.6 are the places where restrictions are identified for data use.    On any of these items, any suggestions that you have (not covered by the updates) are welcome. I want to make sure we are aligned with what you have identified.   Thanks! Nick     On Mon, Sep 24, 2018 at 6:42 PM Eliot Rose wrote: Hello Nick:   We’ve been reviewing the Replica SaaS contract that you sent over, and we have a couple of questions. ·         It’s important to our team that we be able to download and host the Replica data, and we’ve discussed that before with you and gotten the impression that we’re going to be able to do that through the pilot. The only place we see reference to downloading the data in the SaaS agreement is in the termination clause—is there any language about our ability to download data during the pilot elsewhere in the contract or its exhibits, like in the Order Form and Documentation (which weren’t included in full in the draft that you sent)? ·         Similarly, where can we find more information on restrictions on the access to the dataset? The contract refers to limitations on the number of users and on queries, but doesn’t detail what those limitations are. ·         Lastly, is there any information that you can share with us on the technical specs of the Replica database—approximate size for a region like ours, format, etc.? Our enterprise data manager, Robb, is cc’d here and can fill in some of the specifics if you need more information about what we’re looking to know.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, September 17, 2018 8:29 AM To: Eliot Rose Subject: Re: Replica Portland: Next Steps   Acceptance Criteria Template attached. Some of the numbers can't actually be determined until we do a variance analysis on the ground truth data. As an example, if the natural variance in freeway loop detectors is 18% (actually number from Kansas City), then the threshold for that particular metric of course can't actually be below 18%.    Let me know if you have any questions. In terms of to-do's / order of work: We should be able to work the SaaS agreement mark ups / acceptance criteria in parallel. The SaaS mark ups will likely come from legal, which acceptance agreement will come from you / others at Metro.  Assuming we can get everything finalized in the next 45 days, we will start the uncalibrated build in early November-ish. The only thing we will need to do this is GTFS for all transit.  Let's say early December is when we will start the data request from local agencies with a plan to have it collected by mid-Jan. Normalized by early feb. That gives us two months to do calibration runs, with final, fully calibrated Replica done by end of March.  Thanks! Nick   On Thu, Sep 13, 2018 at 11:10 PM Eliot Rose wrote: Thanks Nick! I’ll get you a data sample shortly. Looking forward to seeing the acceptance criteria.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, September 10, 2018 12:55 PM To: Eliot Rose Subject: Re: Replica Portland: Next Steps   Hi Eliot   Thanks for sending this and sorry for delayed response. Answers inline below:       On Tue, Sep 4, 2018 at 5:57 PM Eliot Rose wrote: Hello Nick:   I’ve been moving forward getting an IGA in place on our side to fund the Replica pilot, but I haven’t gotten the chance to follow up on your requests yet. Here’s a quick update and some questions to help keep this moving. 1.       Contract: I looked into it and we don’t have a standard SAS contract template, nor even a comparable contract that we feel like we can base this one on, so we should start from your sample contract. Attached is the latest version I have of that; is this the right one to be using? I've attached the most recent version, I *think it's the same as the one attached but did not go line by line.  2.       Acceptance criteria: same question as above; is this the right version to be working off of? No, we have an updated version, it continues to get modified as learn new things with each effort. I am working to finish today, and then will send your way.  3.       Sample data: working on this right now. Transit data is pretty straightforward, but in our region there’s no single regional source of traffic data—Metro compiles some, but for the complete set we’ll have to pull together data from 3 counties and the city of Portland. Do you want me to send you the different data sets as I access them so that you can think with us about how to best mash them up in this process, or just get you the regional dataset? Any one sample is fine for now. We will ultimately handled the normalizing or meshing, we don't require agencies to do any work, just send us what they have. I am just trying to get a feel for what's out there at this point. Anything you have is great.    Also, we continue to be interested in any documentation you’ve developed about the data sources and methodology used in Replica. You sent a research paper that covers the use of cell data well, but doesn’t discuss other sources in detail. I’m particularly interested in hearing more about how Replica incorporates Streetlight data, and how the end product differs from Streetlight. I have a sense of that from speaking to you, but some of the folks that are going to be approving the contract are well aware of Streetlight, and it will help to be able to articulate to them what the additional value add of Replica is.   We are finalizing the data disclosure document for Kansas City right now and once complete I will send your way. We actually use a combination of streelight, safegraph, cell companies, consumer marketing data, google data, and census data. So, the first component of differentiation is in the composite of data across multiple sources which provides a much larger samples and makes de-biasing the data much easier. Secondly, we produce a full-scale model, that represents all trips vs. aggregate counts with small factoring, which is a pretty significant effort / difference. Thirdly, trip purpose (why people travel) but household characteristics are associated with every trip, which means who is traveling is equally important as how and why people travel. Finally, and most important in my opinion, is we calibrate Replica to world conditions to ensure it can be used in policy making vs. just a source of additional data.    Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, August 21, 2018 3:38 PM To: 'Nick Bowden'; Chris Johnson; Robert Kirkman Subject: RE: Replica Portland: Next Steps   Hello Nick:   Sorry to be a little slow in responding to this; I was out on vacation for a long weekend. I think that you may have forgotten to attach the updated acceptance criteria. I’m attaching the most recent version I have for reference.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, August 17, 2018 1:46 PM To: Eliot Rose; Chris Johnson; Robert Kirkman Subject: Replica Portland: Next Steps   Good Afternoon   Thanks for taking time to talk this week. We are very excited to get started. Here are the follow up items from our call: 1. Contract Form / Process: You guys are going to check on using your own standard agreement vs. using our standard agreement. Once determined, we can start the procurement / legal review process.  2. Acceptance Criteria Document: Updated Template Version Attached. It takes a little localization, but by and large, about 90% consistent across regions.  3. Data Collection Process: I've attached a data requirements checklist to the email as well. This provides a run down of all the data required across all participating agencies. Again, the gathering of this will be handled in a very programatic way, so don't fret the process at this point.  4. Sample Date: If you have a sample set of traffic and transit counts you can share in their raw format, that would be great. Please let me know if I forgot anything. Have a great weekend.   Nick   From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Sean Johnson; Lauren Massey; Chris Johnson [External sender]Re: Portland Replica questions Tuesday, November 19, 2019 10:20:17 AM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Team Portland! What a weekend indeed. The Big Red was embarrassed at home by the Badgers. Convenient day to be out of the office - I couldn't handle the shame, Chris. :) 1. In terms of the trip table, we'll do our best to emulate the schema of your trip table. Can't promise that we'll be able to match it exactly. But we will do our best. 2. We work hard to remove all outlying data sets during the calibration process. Remember, we create personas which are abstractions of real people with real movements. Also, access to the trip table should be extremely limited. You'll have to decide who gets access to it, but I'd encourage you to think very carefully about that. 3. The more the merrier! Happy to do another session. Also, any feedback on the use case template I sent over? The more info you can provide, the better we can tailor those onboard sessions. Thanks, Nathan On Mon, Nov 18, 2019 at 7:02 PM Eliot Rose wrote: Hello Nathan and team: Hope y’all enjoyed a restful day of not checking in with us, and that Nathan’s entire day off was restful. Chris was bummed that he didn’t get to boast about the Badgers’ victory over the Huskers though. Chris and I caught up with each other on this project after both of us being out for a spell today, and we had a couple of questions. These will help us get the right policies in place about how our staff and partners use Replica. · Is the trip table in the same format as the data that you get when you query data through the web tool? Or is it more detailed? · Remind us - how does Replica remove / restrict access to potentially identifiable outlier trips? I know we’ve discussed the case of someone who might be identifiable in the data because they have a unique trip (e.g,. they’re that one person who commutes 20 miles each way to downtown along a route with no infrastructure). How do you identify / treat those trips in the data? Do they show up at all via the web tool or in the trip table? · Portland’s Replica info sessions are getting oversubscribed – if they can’t accommodate the demand in a single 30-person training session would you be open to doing a 2nd session at Portland when you visit? If not, I can tell them to call in the bouncers and velvet ropes. Thank you, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Lauren Massey; Sean Johnson; Chris Johnson; Joe Broach Re: [External sender]Portland Replica Onboard Friday, November 08, 2019 6:25:12 AM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot, I like everything you propose. I think we'd like to modify our position to have multiple sessions per agency. Given our new focus on depth-versus-breadth, a single 3-4 hour session per agency would be more than satisfactory. If you want to schedule a morning session with one agency and an afternoon session for another agency that's fine. Or if you'd prefer to dedicate all mornings, or all afternoons, we are extremely flexible. As it stands right now, we'll use feedback gathered from our use case documents to help assess agency need, and tailor to each agency. I will also send you a schema of our data table. I'd love for you to circulate that to the most technical participants and review that, and then let us know which data fields are of particular interest. Thanks, Nathan On Wed, Nov 6, 2019 at 5:04 PM Eliot Rose wrote: Hello Nathan and team: I just went on a scheduling spree for our Replica rollout – see below. It’s a lot of details and more meeting rooms than we need, but I wanted to grab good rooms because they get booked quickly at Metro and use this to coordinate admin support for the meetings. Can you please answer some questions to help me firm this up? · Seems like 12/16-12/18 works well for your visit, but it’s potentially more time than we need. When within that window do you want to visit us? · How do you want to divide up the training sessions? You mentioned holding 2 sessions per agency – how long do you want each of those sessions to be, and does each of the two sessions cover different content? · Do prefer to hold joint sessions for all 3 agencies at Metro or schedule individual sessions with each agency? If you don’t have a preference, I’ll ask Portland and TriMet if they do. Most of the rooms I’ve booked have enough space for all of us if needed. · Is there anything that’s missing from the timeline below? Replica schedule: · This week: distribute use case sheet to Portland and TriMet · 11/26, 12-1, Metro room 370 - Internal Replica intro for Metro power users: Eliot will give an overview of Replica, demo some of the tool’s features, and have Metro staff fill out use case sheets. · 12/6, 12-1:30, Metro room 301 – Replica @ Metro emerging tech working group: Eliot will give an overview of Replica and demo some of the tool’s features for Metro and partner agency staff who focus on new modes like ride-hailing and micromobility and get input on how to structure the 12/18 workshop with partner agency staff. · 12/18: 9-12:30, Metro Council Chambers – Replica @ TPAC/MTAC workshop: Metro staff, and potentially Replica staff, will present Replica to Metro’s transportation and land use technical committees, which consist of staff from Metro partner agencies and community representatives. We will give an overview of the tool, facilitate small groups in exploring the data using Metro laptops, and describe how and when partner agencies will be able to access the tool. This is a long meeting with multiple agenda items; I’ve requested 45 minutes. · 12/16-12/18: Replica staff lead BYO laptop training sessions for Metro, PBOT and TriMet power users. I’m holding the following rooms/times at Metro, and PBOT and TriMet may be able to host sessions on site if needed: o 12/16: Holding meeting room 370A (capacity = 30) from 9-12 and 370A/370B (capacity = 60) from 12-5 o 12/17: Holding our Council Chamber (capacity = 125) from 8-12 and room 401 (capacity = 43) from 2-5 o 12/18: Holding room 301 (capacity = 20) for from 12-5 for afternoon sessions following the TPAC/MTAC workshop Thanks! Eliot Eliot Rose Technology Strategist Metro From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 1:44 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Lauren Massey; Sean Johnson Re: [External sender]Portland Replica Onboard Thursday, November 07, 2019 12:35:43 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! That's about it. Sometimes we make mice available, because zooming in and interacting with the map feature inside Explorer isn't perfect with a trackpad. Maybe a couple of extension cords to connect to outlets. We'll run a presentation from our work computers and will bring along all of the other training materials. So projector is key. We have all of the essential connection dongles. Lastly, robust WiFi is important because we'll likely have 10+ work stations all connected to Replica, which is a data heavy application. Thanks! -Nathan On Thu, Nov 7, 2019 at 2:13 PM Eliot Rose wrote: One more question – what supplies do you need for these meetings? Sounds like we need a projector and power sources for people to plug in their laptops – anything else? Eliot Rose Technology Strategist Metro From: Eliot Rose Sent: Wednesday, November 06, 2019 3:04 PM To: 'Nathan Preheim' Cc: Lauren Massey; Sean Johnson; Chris Johnson; Joe Broach Subject: RE: [External sender]Portland Replica Onboard Hello Nathan and team: I just went on a scheduling spree for our Replica rollout – see below. It’s a lot of details and more meeting rooms than we need, but I wanted to grab good rooms because they get booked quickly at Metro and use this to coordinate admin support for the meetings. Can you please answer some questions to help me firm this up? · Seems like 12/16-12/18 works well for your visit, but it’s potentially more time than we need. When within that window do you want to visit us? · How do you want to divide up the training sessions? You mentioned holding 2 sessions per agency – how long do you want each of those sessions to be, and does each of the two sessions cover different content? · Do prefer to hold joint sessions for all 3 agencies at Metro or schedule individual sessions with each agency? If you don’t have a preference, I’ll ask Portland and TriMet if they do. Most of the rooms I’ve booked have enough space for all of us if needed. · Is there anything that’s missing from the timeline below? Replica schedule: · This week: distribute use case sheet to Portland and TriMet · 11/26, 12-1, Metro room 370 - Internal Replica intro for Metro power users: Eliot will give an overview of Replica, demo some of the tool’s features, and have Metro staff fill out use case sheets. · 12/6, 12-1:30, Metro room 301 – Replica @ Metro emerging tech working group: Eliot will give an overview of Replica and demo some of the tool’s features for Metro and partner agency staff who focus on new modes like ride-hailing and micromobility and get input on how to structure the 12/18 workshop with partner agency staff. · 12/18: 9-12:30, Metro Council Chambers – Replica @ TPAC/MTAC workshop: Metro staff, and potentially Replica staff, will present Replica to Metro’s transportation and land use technical committees, which consist of staff from Metro partner agencies and community representatives. We will give an overview of the tool, facilitate small groups in exploring the data using Metro laptops, and describe how and when partner agencies will be able to access the tool. This is a long meeting with multiple agenda items; I’ve requested 45 minutes. · 12/16-12/18: Replica staff lead BYO laptop training sessions for Metro, PBOT and TriMet power users. I’m holding the following rooms/times at Metro, and PBOT and TriMet may be able to host sessions on site if needed: o 12/16: Holding meeting room 370A (capacity = 30) from 9-12 and 370A/370B (capacity = 60) from 12-5 o 12/17: Holding our Council Chamber (capacity = 125) from 8-12 and room 401 (capacity = 43) from 2-5 o 12/18: Holding room 301 (capacity = 20) for from 12-5 for afternoon sessions following the TPAC/MTAC workshop Thanks! Eliot Eliot Rose Technology Strategist Metro From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 1:44 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Chris Johnson Nathan Preheim Eliot Rose; Joe Broach RE: [External sender]Re: Post-Acceptance Replica Data Delivery Schedule Thursday, October 31, 2019 2:57:49 PM Yes! Thanks! That’s very helpful. Here’s a more nuances question for you… In the interest of maintaining continuity of the data sets, I’d like to know if we’ll be able to order “historical” seasons of Replica? As context, a Metro budget scenario that I see playing out is that our next batch of funding for this type of purchase might not be available until 7/1/2021. So, would we still have the option of purchasing the Spring 2020, Summer 2020, Fall 2020 and Winter 2020-21 sets in July 2021? Or does sitting out a budget cycle automatically align us to the subsequent 2021 vintage? Does that question make sense? -Chris   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Thursday, October 31, 2019 2:26 PM To: Chris Johnson Cc: Eliot Rose ; Joe Broach Subject: [External sender]Re: Post-Acceptance Replica Data Delivery Schedule CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Chris! Absolutely an easy question to answer. Here's how things will play out, assuming you all formally accept the model in December 2019.... We are launching with Fall 2018 and Winter 2018/2019. It's a 1 year agreement, from December 2019 until December 2020. Remember, we don't start the clock until the model has been formally accepted. During the one-year term you can expect the following seasonal refreshes.... Spring 2019 Summer 2019 Fall 2019 Potential Winter 2019/2020 I can guarantee 5 seasons worth of data, and depending on how systematized your ground truth data is we might be able to squeeze in Winter 2019/2020 just prior to the expiration date. Is that helpful? Thanks, Nathan On Thu, Oct 31, 2019 at 4:01 PM Chris Johnson wrote: Hi NathanQuick (and hopefully easy) question for you to answer… Given where we are now in the process/schedule, for which quarters (and years) can we expect Replica data for, give or take a quarter? I’m asking this because we’re hip-deep in budgeting for FY 20-21 right now and I’m trying to be as strategic as possible. Thanks! -Chris -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Attachments: Nathan Preheim Eliot Rose; Walle Brown Lauren Massey; Sean Johnson [External sender]Onboard Worksheets Monday, October 21, 2019 3:31:57 PM chicago_worksheet_all.pdf CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hello Eliot and Walle! I promised to send over some of the sample "projects" we created for Chicago Onboarding. We'll likely follow this lead when we onboard Portland, except the use cases will be more contextually relevant. As you'll see, the goal is to get folks into the platform and then learn by doing. Happy to work with you all to create our Portland versions as we get closer to launch. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Nick Bowden [External sender]Re: Responding to latest Replica public records request Wednesday, October 16, 2019 8:52:45 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Eliot, I'm attaching Nick to this email because he is aware of that prior request. I don't believe we will object to sharing previously assembled information. In terms of 2019 shared content, there are a couple of technical white-papers that we provided that delve into the nuances of how we construct our models. Those materials were made available inside a FAQ folder located on a shared folder. However, that folder existed on our prior Sidewalk Labs domain, and I have not re-shared that content. If you have saved, local copies, we would prefer contents not be disclosed as they likely include sensitive information that would be considered intellectual property. Nick, any other documents that we would request not be disclosed? Thanks, Nathan On Wed, Oct 16, 2019 at 6:56 PM Eliot Rose wrote: Hello Nathan: We’re preparing our response to the public records request that I notified you about last week. It’s a very detailed and comprehensive request, and we’re trying to expedite the release of the appropriate records while still giving you at least 3 business days to review the responsive materials, per our agreement. There are a couple of things that I’d like to draw to your attention to and get your feedback on. Can you please respond to this email by the end of the week? A significant amount of the materials that the reviewer requested are included in our previous response to the Wall Street Journal. We intend to provide that response, including the redactions that Metro and Replica previously agreed to, instead of re-compiling these materials and asking you to review them again. I’m assuming that you’re OK with that, and that you don’t need to review those materials again since they’ve already been reviewed by your team. Please let me know if you have any concerns with that approach. That response is attached for reference. Are there specific materials that you’ve provided us in 2019 that you consider trade secret and are likely to request that we not disclose in response to this request? If so, it would help streamline the process if you sent me a list of those materials and an explanation of why Replica considers them trade secret. We’ll still compile all of the responsive materials and let you review the response before sending, but with the WSJ request I recall that the backand-forth about what exactly constituted a trade secret took up a fair amount of time, so I think that having this information up front will help us get back to this requestor promptly. Here (in section 2) is the OR definition of trade secret in case it helps to refer to that. Thank you, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595?0055 nathan replicahg .com El From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Chris Johnson; Joe Broach [External sender]Re: Portland Replica updates Thursday, October 03, 2019 12:46:09 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! 4-6 weeks post all data sets received is when we can anticipate a release date of the calibrated Replica for the Portland region. If everything was delivered this week - or soon after - we could plan on early November. In terms of the event, I don't think we'll quite have the full experience ready to go. As you know, we have an un-calibrated version available to Explore, but we haven't yet enabled Reports or Downloads. I think you'd rather offer up a comprehensive experience, correct? If so, I'd suggest a re-schedule. Thanks, Nathan On Thu, Oct 3, 2019 at 12:28 PM Eliot Rose wrote: Thanks Joe! Nathan, can you please let me know when we can expect to see the first round of data for our region? I need to know whether we should postpone this 10/16 workshop we have planned. I’m available til 2:30 if you want to chat – 503.797.1825. Thanks! Eliot Rose Technology Strategist Metro From: Joe Broach Sent: Wednesday, October 02, 2019 10:51 PM To: Eliot Rose; Nathan Preheim Cc: Chris Johnson Subject: RE: Portland Replica updates Hi all, All auto and transit data are now posted except for C-TRAN (Vancouver transit), and we don’t currently have an ETA on that. Please check notes in the file headers and status sheet comments for additional details. Bike & ped data are staged, but after review I’ll need to take another pass tomorrow before finalizing and uploading. I hope to get to them by the end of the day and will update when I do. Cheers, joe From: Eliot Rose Sent: Monday, September 30, 2019 1:41 PM To: Nathan Preheim Cc: Joe Broach ; Chris Johnson Subject: Portland Replica updates Hello Nathan: Sorry we missed you today! Hope that the onboarding is going well. I just checked in with the team and the main news is that, barring any unforeseen challenges, Joe plans to finish uploading our ground truth data by the end of the day tomorrow, and he’ll also plan to send along a recap of all the files that are in the folder. Given that progress, when do you think that we’ll receive the first uncalibrated version of Replica? We have a presentation to our technical committeees on the tool scheduled for 10/16, and I was hoping to let them explore the online data as part of the exercise. We can reschedule if the Portland isn’t going to be ready by then. Thanks, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Nick Bowden Eliot Rose; Chris Johnson Nathan Preheim [External sender]Exciting News About Replica Friday, September 06, 2019 7:19:46 AM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Good Morning Eliot and Chris As you know, over the last three years, Replica has lived and grown inside of Sidewalk Labs. As Sidewalk Labs’ work in Toronto has intensified and Replica’s growth has continued, it made sense to formally separate the too entities and allow each to be uniquely focused on their own work. As of this week, Replica the product has become Replica the company.  Sidewalk Labs will continue to be an important partner and shareholder in Replica. We have the ability to tap into and collaborate with both Sidewalk Labs and the broader Alphabet family as needed. Most important, our work continues to be very much at the heart of Sidewalk Labs mission — accelerating urban innovation.  Before making the news public, we wanted to share with you, our customers and partners. We wouldn’t be in this position without your belief in our work and willingness to partner on a new approach to understanding the movement of people and goods. Our primary goal remains unchanged — build a product that helps you build the very best cities. Outside of new email addresses, nothing will change as it relates to our work with you. You will continue to interact with the same team and same product. In fact, as a stand alone company, we now have the ability to hire more people and add capacity more quickly. If you have any questions at all, I am happy to schedule time to discuss over the phone or happy to answer over email.  I've shared this with you, the two primary points of contact at Metro, please feel free to share with others at Metro and the City of Portland as you see fit. The public announcement won't come until next week. Thank you again.  Nick and the Replica Team From: To: Subject: Date: Nathan Preheim Eliot Rose; Joe Broach Speed Data Thursday, August 29, 2019 10:24:16 AM Hey Joe! Found out that median speed data is optional. Average is much more important.  Thanks! -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Nathan Preheim Joe Broach Eliot Rose Metro Replica // Ground Truth Data Tuesday, August 20, 2019 1:35:07 PM Hey Joe, We browsed thru the files that you sent over and they look great! High five on assembling the data into our preferred format. Perfect! There are two extra fields that we would LOVE to have in addition to what you provided...  "average_speed_kmh", "median_speed_kmh", Median speed is a real bonus. Do you happen to have average speed for each of these sensors? Please let me know. Thanks again for all of your assistance on data formatting! -Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Attachments: Chris Johnson nathan@sidewalklabs.com Eliot Rose; Joe Broach; Nick Bowden Replica Use Cases Wednesday, August 14, 2019 4:01:57 PM TransportationDataProgram_Scope+Schedule_v7.docx Hi NathanCan you do me favor and review the attachment when you get a chance? It’s a proposal for a new enterprise transportation data program that we (Metro Research Center) would like to stand up to support Metro’s planners. Plan A is for Replica to become *THE* foundational data source for the new transportation data program. Plan B is for us to cobble together the same old data sets that we’ve always used and perhaps sprinkle in some of the other emerging big data source. It would be awesome if I could get an initial read from you all if Replica is conducive (and/or if it’s not) to the use cases identified in the proposal. Feel free to reach out with any questions. Thanks! -Chris From: To: Subject: Date: Attachments: Nathan Preheim Eliot Rose Metro Replica // Before and after data examples Monday, July 29, 2019 8:15:00 AM Ground Truth Data Formatting v1.2.pdf Hey Eliot, I'm still reading through that RedTailMedia article! Dense. I had talked to Nick about your quote and we didn't object. I certainly would have followed up with you expediently if we did. Any local blowback as a result? In terms of an example of pre and post normalized data, the best thing to offer is an example of our ground truth schema. This defines exactly what we need. Think of it as "best case scenario" so don't stress if you can't accommodate everything within.  Thanks, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Nathan Preheim Eliot Rose Re: slides/demo and a public records request Tuesday, June 18, 2019 9:01:56 AM Hi Eliot! These folks have been added too. Totally agree on finding some middle ground on releasing acceptance criteria. Anything we can do to build confidence and dispel misperceptions is important. Nick is out this week and I'd have to confer with him on what is suitable for release.  I will indeed set up another discussion for later this week on criteria. What works best for you and Chris? Let me know and I'll get that discussion scheduled. Thanks, Nathan On Mon, Jun 17, 2019 at 12:32 PM Eliot Rose wrote: Thanks Nathan! We won’t disclose the draft acceptance criteria document that I sent. I understand that the testing process is unique to Replica and is part of your business model. At the same time, I wouldn’t be surprised if we continue to get inquiries like this, and I do think that sharing some information about the acceptance criteria would help people understand why this is such a unique opportunity for our region, because the testing process gives us confidence that this is a responsible investment in accurate, anonymized data.  Is there any information on the acceptance criteria that you would be comfortable with sharing publicly?   I’m planning to send a revised version of the acceptance criteria later today with comments. I think that we’ll want to discuss the scooter data and ped data before we finalize. Would you like to try and set up something later in the week for that, or can we tackle it tomorrow / hold off til next Monday’s check-in?   Looking forward to the data collection kickoff tomorrow. In addition to adding Mike Gilligan from TriMet, can you please add the following folks from Portland: Michael.Kerr@portlandoregon.gov   joshua.lynch@portlandoregon.gov   Michael.McDonald@portlandoregon.gov Aubrey.Lindstrom@portlandoregon.gov Happy to forward the invite to them instead if you’d prefer.   -Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Sunday, June 16, 2019 3:23 PM To: Eliot Rose Subject: Re: slides/demo and a public records request   Hi Eliot,   We heard back from legal. They consider the acceptance criteria document proprietary, as it's entirely unique to measuring Replica quality and releasing it has the potential to cause business harm. So I'd ask that you not share that document. We don't have any issues with sharing the contract.    Also, here's a high-level deck that you can use to jumpstart the conversation with the modelers group.    Thanks, Nathan       On Wed, Jun 12, 2019 at 6:27 PM Eliot Rose wrote: Thanks! I appreciate it.   One more thing to add to your list: Metro received a public records request from the local reporter who wrote the Geekwire story on Portland’s Replica pilot that ran recently. She is requesting the agreement between Metro and Replica and the acceptance criteria.   Per our agreement, I’m providing you with the versions of those documents that we intend to disclose. Can you please let me know if Replica considers any of this information confidential? If so, we’d appreciate it if you can provide any redactions by next Tuesday COB so that we can respond to the request in a timely fashion.   Thanks again, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, June 12, 2019 9:35 AM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation?   Hi Eliot!   Honestly, this should only take me minutes. I've got something that's pretty good as a starting point. It's just a matter of subtracting slides that aren't applicable.    I'll add this to my Friday task list. You'll have it by EOD Friday.   Thanks, Nathan     On Tue, Jun 11, 2019 at 5:45 PM Eliot Rose wrote: OK, thanks! I’m happy to cobble together something from existing slides if it saves you effort.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 11, 2019 3:25 PM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation?   Hi Eliot!   I can throw something together for you. Give me a coupe days to prepare a 5-6 slide overview. And yes, you can certainly demo KC Replica.   More soon!   Thanks, Nathan     On Tue, Jun 11, 2019 at 5:02 PM Eliot Rose wrote: Hello Nathan:   I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there.   I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place     -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com   -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com   -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Attachments: Nathan Preheim Eliot Rose Re: slides/demo and a public records request Sunday, June 16, 2019 3:23:29 PM Replica Demo Portland.pdf Hi Eliot, We heard back from legal. They consider the acceptance criteria document proprietary, as it's entirely unique to measuring Replica quality and releasing it has the potential to cause business harm. So I'd ask that you not share that document. We don't have any issues with sharing the contract.  Also, here's a high-level deck that you can use to jumpstart the conversation with the modelers group.  Thanks, Nathan On Wed, Jun 12, 2019 at 6:27 PM Eliot Rose wrote: Thanks! I appreciate it.   One more thing to add to your list: Metro received a public records request from the local reporter who wrote the Geekwire story on Portland’s Replica pilot that ran recently. She is requesting the agreement between Metro and Replica and the acceptance criteria.   Per our agreement, I’m providing you with the versions of those documents that we intend to disclose. Can you please let me know if Replica considers any of this information confidential? If so, we’d appreciate it if you can provide any redactions by next Tuesday COB so that we can respond to the request in a timely fashion.   Thanks again, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, June 12, 2019 9:35 AM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation?   Hi Eliot!   Honestly, this should only take me minutes. I've got something that's pretty good as a starting point. It's just a matter of subtracting slides that aren't applicable.    I'll add this to my Friday task list. You'll have it by EOD Friday.   Thanks, Nathan     On Tue, Jun 11, 2019 at 5:45 PM Eliot Rose wrote: OK, thanks! I’m happy to cobble together something from existing slides if it saves you effort.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 11, 2019 3:25 PM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation?   Hi Eliot!   I can throw something together for you. Give me a coupe days to prepare a 5-6 slide overview. And yes, you can certainly demo KC Replica.   More soon!   Thanks, Nathan     On Tue, Jun 11, 2019 at 5:02 PM Eliot Rose wrote: Hello Nathan:   I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there.   I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place     -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com   -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 rli i 1k From: To: Subject: Date: Nathan Preheim Eliot Rose Re: slides/demo and a public records request Friday, June 14, 2019 4:41:02 PM Eliot! I owe you a quick presentation. Coming this weekend. We’re seeking guidance on releasing the acceptance criteria. That’s a first for us so had to run it up the chain of command. No word back yet. I’ll check in again on status.  Lovely on scooter data. And keep me posted on pedestrian. Remember, we’ll model it. We just won’t have anything firm to compare without ground truth data.  Thanks On Fri, Jun 14, 2019 at 5:47 PM Eliot Rose wrote: Hello Nathan:   Just wanted to confirm that you saw the heads-up about the public records request that I sent on Wednesday. I want to make sure that you get a chance to review before we respond.   Also, I’ve been rounding up names from TriMet and Portland to add to the invite for the data collection meeting on Tuesday. Still getting names from Portland, but for TriMet you can add Mike Gilligan - GilligaM@trimet.org   I’ve been following up on the questions from our last check-in and revising the acceptance criteria. I may not have a version to share until right before the call, but know that we’re getting closer – it seems like we will have some solid scooter data to include in the evaluation, but we’re not sure that our ped data is going to meet some of the needs that we discussed on the last call.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, June 12, 2019 4:27 PM To: 'Nathan Preheim' Subject: slides/demo and a public records request   Thanks! I appreciate it.   One more thing to add to your list: Metro received a public records request from the local reporter who wrote the Geekwire story on Portland’s Replica pilot that ran recently. She is requesting the agreement between Metro and Replica and the acceptance criteria.   Per our agreement, I’m providing you with the versions of those documents that we intend to disclose. Can you please let me know if Replica considers any of this information confidential? If so, we’d appreciate it if you can provide any redactions by next Tuesday COB so that we can respond to the request in a timely fashion.   Thanks again, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, June 12, 2019 9:35 AM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation?   Hi Eliot!   Honestly, this should only take me minutes. I've got something that's pretty good as a starting point. It's just a matter of subtracting slides that aren't applicable.    I'll add this to my Friday task list. You'll have it by EOD Friday.   Thanks, Nathan     On Tue, Jun 11, 2019 at 5:45 PM Eliot Rose wrote: OK, thanks! I’m happy to cobble together something from existing slides if it saves you effort.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 11, 2019 3:25 PM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation?   Hi Eliot!   I can throw something together for you. Give me a coupe days to prepare a 5-6 slide overview. And yes, you can certainly demo KC Replica.   More soon!   Thanks, Nathan     On Tue, Jun 11, 2019 at 5:02 PM Eliot Rose wrote: Hello Nathan:   I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there.   I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place     -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com   -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Chris Johnson Re: Portland Replica // Status Update Thursday, May 30, 2019 9:08:29 AM Eliot! Great, thanks for the heads up. In talking with Nick I think it might be most advantageous to delay that trip a bit - and reschedule a tad closer to launch. That will likely be more fruitful. In the meantime we can go full bore on data collection so that we get this model up and running as quickly as possible. I can interface with anyone and everyone that will be sourcing data virtually.  Also, I enjoyed the recent article in GeekWire. We received a ton of favorable feedback about it and I was surprised at the breadth of reach that article had! Talk to you next week! Thanks, Nathan On Wed, May 29, 2019 at 9:22 AM Eliot Rose wrote: Thanks Nathan! This all makes sense. I just spoke with Chris. He’s going to take another pass at the acceptance criteria and then we’re going to circulate to Portland and TriMet for their sign-off and try to get any feedback from them by our call next Monday so that we can discuss on our call - we especially need our partners to weigh in on the transit and scooter-related criteria. When I send those out I’ll also direct people to the data collection resources in the folder so that they can prepare, and mention the timing of your visit. Chris is available 6/11-6/13 so you can book your trip.   Happy to have a call if there’s anything else to discuss!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, May 29, 2019 8:23 AM To: Eliot Rose; Chris Johnson Subject: Portland Replica // Status Update   Eliot,   Thanks for the call yesterday. Couple key takeaways…   First, it’s imperative we split the acceptance criteria flow from the data collection flow. These are independent activities. I’m confident that Chris will soon review and “accept” the proposed terms that we recently delivered. There might be a slight modification on a couple attributes, but we’re keen to nail this down.   Second, let’s get moving on data collection. This is typically the most intensive effort of the workflow as there are multiple agencies involved and lots of people that will need to be engaged. I’ve created a document that identifies all of the required attributes for both traffic and transit data. Complete details can be found in the Ground Truth Data Formatting document within the 3. DATA folder.   You and I have discussed data collection on a cursory level - time to dig in and start that effort in earnest.   There are template emails that can be used (Data Collection Template) to assist with the data requests. You can management that process, or I can step in and work directly with the data partners. Expediency is key here. I’d like to have all data collected within the next two weeks.   We’ve lost a little traction over the past couple of weeks and I’d like to regain momentum.    Please let me know if you have questions about next steps. Happy to jump on a quick call anytime Thursday to discuss.    Thanks, Nathan   -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Attachments: Nick Bowden Sherrie Blackledge Chris Johnson; nathan@sidewalklabs.com; Eliot Rose Re: Sidewalk Labs Agreement - Metro No. 935980 Monday, May 13, 2019 12:30:38 PM 935980 Replica Agreement.docx HI Sherrie Updated document attached. All references to SWL, Sidewalk Labs, and Sidewalk Labs Employees have been replaced.  Thanks. Nick On Mon, May 13, 2019 at 2:08 PM Sherrie Blackledge wrote: Nick, There is reference to Replica, SWL and Sidewalk Labs throughout the agreement (attached).  Would you please have your procurement  folks update the agreement and send it back to me?    Then I will have it resigned.   Thanks!   Sherrie Blackledge Senior Management Analyst   My gender pronouns: she, her, hers.   Metro oregonmetro.gov 600 NE Grand Avenue Portland, OR 97232-2736 503-797-1724   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, May 13, 2019 11:57 AM To: Chris Johnson Cc: Sherrie Blackledge; nathan@sidewalklabs.com; Eliot Rose Subject: Re: Sidewalk Labs Agreement - Metro No. 935980   Should just apply to the first page (Replica, Inc. instead of Sidewalk Labs) and the signature page. But, if it's easiest to do a "find and replace" that's fine too.   Thanks. Nick     On Mon, May 13, 2019 at 1:52 PM Chris Johnson wrote: Hi NickSherrie and just chatted about the change to the legal entity signing on your side (i.e., Sidewalk Labs Employees, LLC to from Replica, Inc.). Does that mean that the change needs to be reflected throughout the entire contract document, or just the signing page (i.e. page 9)? Thanks! -Chris   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, May 10, 2019 10:38 AM To: Sherrie Blackledge Cc: nathan@sidewalklabs.com; Chris Johnson ; Eliot Rose Subject: Re: Sidewalk Labs Agreement - Metro No. 935980   Updated W-9 attached.   Thank you.   Nick       On Fri, May 10, 2019 at 12:33 PM Sherrie Blackledge wrote: Will you please send an updated W9 (ours is attached) so we can update our files?   Sherrie Blackledge Senior Management Analyst   My gender pronouns: she, her, hers.   Metro oregonmetro.gov 600 NE Grand Avenue Portland, OR 97232-2736 503-797-1724   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, May 10, 2019 10:32 AM To: Sherrie Blackledge Cc: nathan@sidewalklabs.com; Chris Johnson; Eliot Rose Subject: Re: Sidewalk Labs Agreement - Metro No. 935980   Hi All   Thank you very much. We actually need to make a minor change (I know!) to the legal entity signing the agreement on our side. It should be Replica, Inc. not Sidewalk Labs Employees, LLC. No other changes, in between the last version and signing, Sidewalk Labs created wholly-owned legal entities for each of it's products.    Rather than sign on our end, then send over a consent to assignment to new entity and have to re-sign, feels like it easier to do now. Apologize for the timing, it wasn't something I was anticipating was going to happen in the middle of this all.   Again, nothing else changes in the agreement, timeline, product, services, etc..   Let me know if you have any questions. Sorry again! Nick     On Thu, May 9, 2019 at 4:52 PM Sherrie Blackledge wrote: Please find attached a PDF file of the Metro-signed contract document for the Sidewalk Labs Software Subscription.  Once your organization has signed, please return a fully-executed version to my attention.    Please contact me with any questions you may have.  Thanks for your help in this matter.     Sherrie Blackledge Senior Management Analyst   My gender pronouns: she, her, hers.   Metro oregonmetro.gov 600 NE Grand Avenue Portland, OR 97232-2736 503-797-1724   From: To: Subject: Date: Attachments: Nathan Preheim Eliot Rose; Chris Johnson; Nick Bowden Metro Replica // Updated Acceptance Criteria Friday, May 10, 2019 11:43:12 AM Portland - Acceptance Criteria (External).docx Portland - Acceptance Criteria (External).pdf Hello Eliot! You'll find updated acceptance criteria attached. Please review internally, and let us know if you have any questions or feedback.  Thanks again for working to get that contract pushed through! We're ready to go! Thanks, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Chris Johnson; Joe Broach; Maribeth Todd Re: Requirements Meeting Recap and Updated Acceptance Criteria Monday, May 06, 2019 11:32:09 AM Hi Eliot! In terms of clarification on data sharing, I think we're all on the same page. Yes, we are completely onboard with you all validating the final outputs. We don't use ground-truth for model creation, in fact we have already created an uncalibrated Replica for Portland. The ground-truth is used to calibrate (vs. validate), obviously meaning fit the model to the counts we see on the ground. It is typical, as you and Nick discussed, to hold back some of the ground-truth (so as it is consistent in time period and dimensionality) to validate for your own purposes. We certainly don't need everything shared, but getting 80% +/- of the available ground-truth as we start simulation runs for the purpose of fitting is pretty standard practice.  Ideally, the 80% should stretch across all data sets (traffic, transit, TNCs, scooters, etc..). An example of this in practice, New York provided traffic counts for 8 or 9 major thoroughfares and held back 1-2 for the purposes of ensuring internally consistency. The counts, both provided and held back, were from the same season, collected in the same way, and baselined against Replica consistently. This is a practice we encourage for the sake of building trust in the model, particularly in places where ground-truth doesn't exist.  That sound alright? In terms of that validation/ground truth data, it would be extremely helpful if you were able to provide the following inputs ASAP. There will absolutely be other ground-truth data requests, but these data sets are essential to start moving the Portland Replica model from uncalibrated to calibrated - and will really help us get a jump on that effort.  Auto: Hourly counts for one (or more) weekday(s) within the date range of Fall of 2018 (September 1 thru November 30th).  * Major bridges  * At least one sensor on other major highways: I-5, I-205, I-84, 26, 217    - For ease of matching it’s best if they’re not at a major split/interchange Is that feasible? I've got a meeting scheduled with Alexei tomorrow and we'll be reviewing the acceptance criteria revisions before I send them over. Also, I just shared the Google Drive with everyone you suggested. Thanks, Nathan On Fri, May 3, 2019 at 5:53 PM Eliot Rose wrote: Thanks Nathan:   Looking forward to receiving your feedback on the draft acceptance criteria. The follow-ups that I noted from the meeting are below, anything that I missed? Also, can you please make sure that everyone who was at the meeting on our side has access to the Google Drive? Attendee list is below.   Before we dive into the follow-ups, we’d like to clarify your expectations for data sharing during the validation process. We intend to withhold some of the data that we use to validate Replica, both because there are privacy concerns involved (as you heard with Portland’s TNC data) and because we want to ensure that data is used solely for validation and not as an input to the model, which is a standard best practice for model validation. I had run that by Nick, and he seemed OK with it, so our contract reflects that intention, but on yesterday’s call it seemed like Alexei was operating off of the assumption that we were going to share all of the validation data with you. I understand that you need some validation data because you intend to validate Replica in parallel with us. Is your expectation that we’ll share all of the data we use for validation with you? How do we verify that you’re using data that we share with you solely for validation and not to train the model? I’m happy to set up a call to discuss.   My follow-ups from the meeting for Metro: ·         Begin compiling validation data so that we can add more detail to the acceptance criteria. Here are the data sets that I heard the most focus on making sure that we understand / consider adding to the process before we dive into acceptance testing: o   Traffic counts o   Bike/ped counts o   Ridership data for other transit systems (C-Tran, Portland Streetcar) o   Crosswalk of Metro employment zones x Census tracts o   Bikeshare / scooter data o   Demographic data on race / ethnicity ·         Send along the TriMet report about demographics of transit riders to see about including transit demographics in validation criteria ·         Schedule a follow-up with City of Portland re: TNC data (but let’s resolve the data sharing discussion above first)   Attendee list (I know that Peter and Stephanie, the last two folks listed don’t have access to the Drive folder):   Chris Johnson ; Maribeth Todd ; Kerr, Michael ; joshua.lynch@portlandoregon.gov; Martin, Kevin ; McHugh, Bibiana ; steelem@trimet.org; Lindstrom, Aubrey ; Joe Broach ; McDonald, Mike ; Gilligan, Mike ; ningsheng.zhou@portlandoregon.gov; Council Chamber Owen, Jeffrey ; Hesse, Eric ; Sherman, Jacob ; Bryan Nguyen ; Hurley, Peter Peter.T.Hurley@portlandoregon.gov; Lonsdale, Stephanie Stephanie.Lonsdale@portlandoregon.gov   Til soon, and have a great weekend! -Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Friday, May 03, 2019 11:30 AM To: Eliot Rose Subject: Requirements Meeting Recap and Updated Acceptance Criteria   Hi Eliot!   Thanks for coordinating details on your side to support the requirement kickoff call. Lots of useful dialog from all parties involved. We are updating the acceptance criteria based on feedback. I should have that back to you soon - early next week at the latest.    The most germane next steps are acceptance criteria finalization and a fully executed contract. Sounds like both of these items will get resolved within the next week or so.    Thanks again, Nathan   -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Nathan Preheim Eliot Rose Requirements Meeting Recap and Updated Acceptance Criteria Friday, May 03, 2019 11:30:13 AM Hi Eliot! Thanks for coordinating details on your side to support the requirement kickoff call. Lots of useful dialog from all parties involved. We are updating the acceptance criteria based on feedback. I should have that back to you soon - early next week at the latest.  The most germane next steps are acceptance criteria finalization and a fully executed contract. Sounds like both of these items will get resolved within the next week or so.  Thanks again, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Attachments: Nathan Preheim Eliot Rose Kickoff Meeting Presentation Thursday, May 02, 2019 8:31:34 AM Portland - Requirements Kickoff.pdf Hi Eliot! Attached you'll find a revised PDF presentation. I'll be sharing my screen, but thought it would be helpful for you all to have the presentation just in case.  Also, I merged your acceptance criteria into the Requirements Document. That document is available here: https://docs.google.com/document/d/1XCN9kOKJ2GpBEFJRCJ_hUdQ3kYQIUYTTdTmBRBLR6Y/edit# Let me know if you'd like an electronic version of that as well. Thanks, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Chris Johnson Nick Bowden Eliot Rose; Michelle Bellia; Sherrie Blackledge RE: Revised Portland Metro Replica agreement Monday, April 22, 2019 2:23:52 PM Hi NickJust a quick update, I signed off late last week and sent it along to our financial folks for a risk assessment. Hang in there --  we’re close!! Cheers, Chris   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, April 17, 2019 8:06 AM To: Chris Johnson Cc: Eliot Rose ; Michelle Bellia ; Sherrie Blackledge Subject: Re: Revised Portland Metro Replica agreement Hi All Just wanted to circle back with you and see when you are planning to send over the final version of the contract? Thanks. Nick On Wed, Apr 10, 2019 at 1:42 PM Chris Johnson wrote: Hi NickSherrie will be in touch with you re: signing on your side. Thanks! -Chris   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, April 10, 2019 11:38 AM To: Eliot Rose Cc: Michelle Bellia ; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yay! Can we route for signature on our side? On Wed, Apr 10, 2019 at 12:02 PM Eliot Rose wrote: Thanks Nick! That makes sense, and we checked in with our contracting folks and it sounds like it makes sense on their end.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, April 10, 2019 7:28 AM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement We, and the agreement, generally treat the signature date as the start of the contract. The license term and payments are tied to acceptance date. So, contract starts at signature, license term (12 months) starts at acceptance. Everywhere we have worked, we both sign contract now and then we have used email or a one paragraph letter from you to indicate acceptance has been met. Does that make sense? Nick On Tue, Apr 9, 2019 at 6:12 PM Eliot Rose wrote: Next question:   Our contracts folks are used to working on contracts with specific beginning and end dates, and were asking about putting dates on this agreement (e.g., 4/15/19-12/31/20). But in rereading 11.1, it says that the contract starts when we accept the data. Should we put a start date on the contract, or add that at acceptance?   This also raised a question about signatures – are you expecting us to sign the contract now, or once we accept the data?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, April 09, 2019 9:03 AM To: Eliot Rose Cc: Nathan Preheim; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement No need to keep those sections, sorry missed that in the first round of emails. Once I get confirmation, we will route for signature. Thanks. Nick On Mon, Apr 8, 2019 at 3:45 PM Eliot Rose wrote: Thanks Nick:   30 days sounds good. I added that into the attached draft. Please let me check in with our contracts folks to make sure that they’ve completed their review before you route for sig. I’ll keep you posted when I hear back from them. Also, still waiting for a response from you on whether you want to leave in the sections marked [reserved] (2 and 12.1).   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, April 08, 2019 9:20 AM To: Eliot Rose; Nathan Preheim Cc: Lisa Hefty; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We typically have used 30 days as a reasonable amount of time for you all to determine if acceptance criteria has been met. It usually is much shorter than that, but it provides enough time for you all to review. If that sounds okay, we can add on our end and then route for signature. I will let Nathan respond on the particulars for the kick-off meeting. Let me know if contract is good to go per above and we can get signed on our end. Thanks. Nick On Fri, Apr 5, 2019 at 6:07 PM Eliot Rose wrote: Hello Nick:   Formatted contract attached. Other than changing the formatting we added a missing “shall” to 13.1. Only add’l remaining question (other than the ones below) is whether you want to leave in the sections marked [reserved] (2 and 12.1).   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, April 03, 2019 3:41 PM To: 'Nick Bowden' Cc: 'Nathan Preheim'; Chris Johnson Subject: RE: Revised Portland Metro Replica agreement Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot - We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by midweek at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Nick Bowden; Chris Johnson Re: [External email]Re: Revised Portland Metro Replica agreement Tuesday, April 16, 2019 7:03:07 AM Hi Eliot! It's not something we typically do but yes, we can kickoff prior to signed contract.   Only final details I need are a confirmation of participants and a couple day/times that work for the group. I'll then follow up with an invitation and kickoff resources. Thanks much! -Nathan On Thu, Apr 11, 2019 at 6:30 PM Eliot Rose wrote: Hello Nathan and Nick:   We had a call with the project team today; I’ll be getting you a complete list of names and emails for the kickoff by the end of the week and some scheduling options midweek next week. A couple scheduling questions: we’re working with our contracting folks to get this ready for sig ASAP (and I appreciate your quick responses to move that along), but there’s a chance it won’t be signed by the week of 4/22 when we hold the kickoff. Are you comfortable holding the kickoff before we get a final contract signed? Also, we’ve been working to draft our acceptance criteria. Would it be helpful to share those in advance of the kickoff, or should we hold off until we get a chance to discuss the process during the kickoff?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, April 08, 2019 2:15 PM To: Eliot Rose Cc: Nick Bowden; Chris Johnson Subject: [External email]Re: Revised Portland Metro Replica agreement   Hi Eliot!   The week of 4/22 looks good for a kickoff call. I too think we should start smaller. Let's focus on the 3 core contracting agencies - it's up to you all to define requirements. There will be plenty of opportunities to engage the other players as we get into the data collection process.   I have a host of online materials to share when I send out the calendar invite. Who specifically, name and email addresses, should I include?   Looking forward to working with you!   Thanks, Nathan     On Wed, Apr 3, 2019 at 5:40 PM Eliot Rose wrote: Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 102:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement   Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week.   Thanks. Nick     On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline.  Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign.   We are done! Thanks! Nick   On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   Yes, that addition is okay.    Awesome on legal name.   Thanks. Nick     On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.      Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation.  We added a cap on the indemnification. Hopefully this gets us to the finish line.    Thanks. Nick       On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement   Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. -          Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? -          On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   HI Eliot   Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so.   Thanks. Nick     On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5.   On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.    Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call.   On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There -   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by mid-week at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place     -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Attachments: Nick Bowden Chris Johnson; Sherrie Blackledge Eliot Rose; Michelle Bellia Re: Revised Portland Metro Replica agreement Thursday, April 11, 2019 1:57:15 PM Portland--SWL-W9.pdf Hi Sherrie and Chris Our W-9 is attached. Thanks. Nick On Wed, Apr 10, 2019 at 1:42 PM Chris Johnson wrote: Hi NickSherrie will be in touch with you re: signing on your side. Thanks! -Chris   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, April 10, 2019 11:38 AM To: Eliot Rose Cc: Michelle Bellia ; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   Yay! Can we route for signature on our side?   On Wed, Apr 10, 2019 at 12:02 PM Eliot Rose wrote: Thanks Nick! That makes sense, and we checked in with our contracting folks and it sounds like it makes sense on their end.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, April 10, 2019 7:28 AM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   We, and the agreement, generally treat the signature date as the start of the contract. The license term and payments are tied to acceptance date. So, contract starts at signature, license term (12 months) starts at acceptance. Everywhere we have worked, we both sign contract now and then we have used email or a one paragraph letter from you to indicate acceptance has been met.   Does that make sense? Nick     On Tue, Apr 9, 2019 at 6:12 PM Eliot Rose wrote: Next question:   Our contracts folks are used to working on contracts with specific beginning and end dates, and were asking about putting dates on this agreement (e.g., 4/15/19-12/31/20). But in rereading 11.1, it says that the contract starts when we accept the data. Should we put a start date on the contract, or add that at acceptance?   This also raised a question about signatures – are you expecting us to sign the contract now, or once we accept the data?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, April 09, 2019 9:03 AM To: Eliot Rose Cc: Nathan Preheim; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   No need to keep those sections, sorry missed that in the first round of emails.    Once I get confirmation, we will route for signature.    Thanks. Nick     On Mon, Apr 8, 2019 at 3:45 PM Eliot Rose wrote: Thanks Nick:   30 days sounds good. I added that into the attached draft. Please let me check in with our contracts folks to make sure that they’ve completed their review before you route for sig. I’ll keep you posted when I hear back from them. Also, still waiting for a response from you on whether you want to leave in the sections marked [reserved] (2 and 12.1).   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, April 08, 2019 9:20 AM To: Eliot Rose; Nathan Preheim Cc: Lisa Hefty; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We typically have used 30 days as a reasonable amount of time for you all to determine if acceptance criteria has been met. It usually is much shorter than that, but it provides enough time for you all to review. If that sounds okay, we can add on our end and then route for signature.    I will let Nathan respond on the particulars for the kick-off meeting.    Let me know if contract is good to go per above and we can get signed on our end.   Thanks. Nick     On Fri, Apr 5, 2019 at 6:07 PM Eliot Rose wrote: Hello Nick:   Formatted contract attached. Other than changing the formatting we added a missing “shall” to 13.1. Only add’l remaining question (other than the ones below) is whether you want to leave in the sections marked [reserved] (2 and 12.1).   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, April 03, 2019 3:41 PM To: 'Nick Bowden' Cc: 'Nathan Preheim'; Chris Johnson Subject: RE: Revised Portland Metro Replica agreement   Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement   Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week.   Thanks. Nick     On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline.  Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign.   We are done! Thanks! Nick   On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   Yes, that addition is okay.    Awesome on legal name.   Thanks. Nick     On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.      Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation.  We added a cap on the indemnification. Hopefully this gets us to the finish line.    Thanks. Nick       On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement   Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. -          Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? -          On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   HI Eliot   Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so.   Thanks. Nick     On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5.   On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.    Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call.   On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by midweek at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Nick Bowden Portland Replica // Google Drive Folder Wednesday, April 10, 2019 12:17:10 PM Hi Eliot! I got a jump on some pre-kickoff call activities. The Google Drive folder that we'll use for collaboration and communication has been created - I just shared it with you and everyone you suggested, including... Michael Kerr Chris Johnson Robert Kirkman Martin Kevin Tim McHugh Mike Gilligan Sharing is a very fluid process. We can either add or subtract folder access anytime so don't hesitate to let me know if there is anyone else that you'd like to grant access. The folder can be accessed here: https://drive.google.com/drive/folders/0B2goRUpd2W8LMVBHbGFkd0g2SHc You'll need a Google account to access the folder. Don't worry if you don't already have one. Takes a minute to create and you can use your work email address if you'd like. Feel free to browse around. Contents will make much more sense after our kickoff meeting. :) A couple of items worth mentioning... - Inside the *** FAQ *** sub-folder you'll notice a Use Case file. Give that a look to start to better understand Replica capabilities. - I do ask that all Portland Region Replica stakeholders complete this simple form. That'll provide me with comprehensive contact information. That form takes 20 seconds to complete and is accessible here: https://forms.gle/keUP9UBNRcaAy7Ku6 - We will use an Issue List to stay current on open issues and tasks. This simple file will help us ensure that work items are documented and assigned: https://docs.google.com/spreadsheets/d/1l-QKd_weXsRl5QdafsnQvAnszOn2aq_pdvSyoSA4CQ/edit#gid=1817381524 I'll wait to hear back from you regarding date and attendees for our kickoff call. Thanks! -Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 402-595-0055 nathan@ Sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Nathan Preheim Eliot Rose Nick Bowden; Chris Johnson [External email]Re: Revised Portland Metro Replica agreement Monday, April 08, 2019 2:15:34 PM Hi Eliot! The week of 4/22 looks good for a kickoff call. I too think we should start smaller. Let's focus on the 3 core contracting agencies - it's up to you all to define requirements. There will be plenty of opportunities to engage the other players as we get into the data collection process. I have a host of online materials to share when I send out the calendar invite. Who specifically, name and email addresses, should I include? Looking forward to working with you! Thanks, Nathan On Wed, Apr 3, 2019 at 5:40 PM Eliot Rose wrote: Hello Nick and Nathan: Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best. Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome. I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good? Thanks, Eliot Eliot Rose Technology Strategist Metro From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick: As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.   Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot - Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer. Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys. One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below? (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh? Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot - Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick: I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that. This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset? Thanks, Eliot Eliot Rose Technology Strategist Metro From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick: Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end. I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.    There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along. Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot - Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week. Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot - We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits. Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership. Thanks, Eliot Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There - Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.   Eliot Rose Technology Strategist Metro From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Nick Bowden Eliot Rose; Nathan Preheim Lisa Hefty; Michelle Bellia; Chris Johnson Re: Revised Portland Metro Replica agreement Monday, April 08, 2019 9:20:51 AM Hi Eliot We typically have used 30 days as a reasonable amount of time for you all to determine if acceptance criteria has been met. It usually is much shorter than that, but it provides enough time for you all to review. If that sounds okay, we can add on our end and then route for signature.  I will let Nathan respond on the particulars for the kick-off meeting.  Let me know if contract is good to go per above and we can get signed on our end. Thanks. Nick On Fri, Apr 5, 2019 at 6:07 PM Eliot Rose wrote: Hello Nick:   Formatted contract attached. Other than changing the formatting we added a missing “shall” to 13.1. Only add’l remaining question (other than the ones below) is whether you want to leave in the sections marked [reserved] (2 and 12.1).   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, April 03, 2019 3:41 PM To: 'Nick Bowden' Cc: 'Nathan Preheim'; Chris Johnson Subject: RE: Revised Portland Metro Replica agreement   Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement   Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week.   Thanks. Nick     On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline.  Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign.   We are done! Thanks! Nick   On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   Yes, that addition is okay.    Awesome on legal name.   Thanks. Nick     On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.      Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation.  We added a cap on the indemnification. Hopefully this gets us to the finish line.    Thanks. Nick       On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement   Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. -          Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? -          On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   HI Eliot   Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so.   Thanks. Nick     On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5.   On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.    Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call.   On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by mid-week at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Nick Bowden Eliot Rose Re: Revised Portland Metro Replica agreement Sunday, March 31, 2019 2:52:01 PM Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline.  Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   Yes, that addition is okay.    Awesome on legal name.   Thanks. Nick     On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.      Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation.  We added a cap on the indemnification. Hopefully this gets us to the finish line.    Thanks. Nick       On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement   Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. -          Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? -          On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   HI Eliot   Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so.   Thanks. Nick     On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5.   On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.    Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call.   On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by mid-week at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland OR 97232?2736 503?797?1825 1? Metro Making a great place From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Michelle Bellia; Chris Johnson Re: Revised Portland Metro Replica agreement Wednesday, March 20, 2019 12:00:53 PM Yes, that addition is okay.  Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.      Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation.  We added a cap on the indemnification. Hopefully this gets us to the finish line.    Thanks. Nick       On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement   Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. -          Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? -          On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   HI Eliot   Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so.   Thanks. Nick     On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5.   On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.    Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call.   On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by mid-week at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Attachments: Nick Bowden Eliot Rose Re: Revised Portland Metro Replica agreement Wednesday, March 20, 2019 9:47:35 AM SaaSPortland Metro 3.20.2019 SWL edit.docx Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation.  We added a cap on the indemnification. Hopefully this gets us to the finish line.  Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement   Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. -          Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? -          On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   HI Eliot   Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so.   Thanks. Nick     On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5.   On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.    Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call.   On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by mid-week at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Chris Johnson; Michelle Bellia; Robert Kirkman Re: Revised Portland Metro Replica agreement Wednesday, March 06, 2019 6:15:35 PM Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.  Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by mid-week at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 Metro Making a great place From: To: Subject: Date: Attachments: Nick Bowden Eliot Rose Schedule Monday, February 25, 2019 7:17:02 AM Replica_PORTLAND_SCHEDULE - Gantt Chart.pdf Hi Eliot As discussed last week, attached is a rough schedule for project. I worked back from an early June delivery date (acceptance being met) to ensure we have some buffer time. It puts some downward pressure on us quickly getting the contract and acceptance criteria done by March 15th, two weeks from Friday. Is this realistic?  Let me know if you have any questions. Thanks. Nick From: To: Subject: Date: Nick Bowden Eliot Rose Re: Portland Replica update Tuesday, February 19, 2019 7:33:16 AM Hi Eliot The list of folks you provided all have access to Replica Kansas City. Visit here: https://replicaexplorer.sidewalklabs.com/ Just use the email you provided to sign up. Happy to answer any questions for people if needed. Thanks. nick On Thu, Feb 14, 2019 at 6:10 PM Eliot Rose wrote: Thursday at 10 PT is confirmed. I’ll send an invite. That will probably be just me and Michael Kerr, who’s managing the project from the Portland side. I think that his main question will be about whether it’s possible to manage permissions so that the download function is turned off for some users and open to others. But it’ll be good for you two to be able to hear from him directly about the privacy concerns that the city has more generally.   Here’s the list of folks who’s like access to the KC tool.   Eliot Rose Eliot.Rose@oregonmetro.gov Chris Johnson Chris.Johnson@oregonmetro.gov   Robert Kirkman Robert.Kirkman@oregonmetro.gov McHugh, Tim McHughT@trimet.org Gilligan, Mike GilligaM@trimet.org Kerr, Michael Michael.Kerr@portlandoregon.gov Lindstrom, Aubrey Aubrey.Lindstrom@portlandoregon.gov Nayame, Jacqueline Jacqueline.Nayame@portlandoregon.gov McDonald, Mike Michael.McDonald@portlandoregon.gov McEwen, Kirk Kirk.McEwen@portlandoregon.gov Zhou, Ningsheng Ningsheng.Zhou@portlandoregon.gov Hesse, Eric Eric.Hesse@portlandoregon.gov Hurley, Peter Peter.T.Hurley@portlandoregon.gov Marx, Michelle Michelle.Marx@portlandoregon.gov Geller, Roger Roger.Geller@portlandoregon.gov Bertelsen, April April.Bertelsen@portlandoregon.gov Pearce, Art Art.Pearce@portlandoregon.gov   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, February 14, 2019 12:24 PM To: Eliot Rose Subject: Re: Portland Replica update   Any email works.    On Thu, Feb 14, 2019 at 12:46 PM Eliot Rose wrote: Thanks! I’ll confirm that time with the team. And yes, it would be helpful to see KC’s build. I’ll get you emails shortly—do they need to be Google-enabled emails like last time?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, February 14, 2019 10:29 AM To: Eliot Rose Subject: Re: Portland Replica update   Happy to join. How about next thursday, 10am PST?   You guys should have access to the product, but all I need is email addresses for people and I can provide access to KC.    Thanks. Nick     On Thu, Feb 14, 2019 at 11:59 AM Eliot Rose wrote: I’ve been talking with Portland more about their concerns about what data they’ll be able to access. I think that it would help them to understand in more detail how the query/download function within the Replica tool works. Would you be up for joining in a half-hour call next week with them to discuss? If so, please let me know some windows that work for you – Wednesday, Thursday AM, and Friday AM are pretty good on our end. Also, if it’s possible to share a link to an updated version of the tool that they can view in advance of the call that would help as well.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, February 13, 2019 8:09 AM To: Eliot Rose Subject: Re: Portland Replica update   Yes, we would provide the data table outside of the tool / product itself. Either through an API or a dump of the data table into a cloud bucket you all maintain.    I am happy to make an intro to Jean-Louis if you believe it will help. I think it's important to note that he doesn't have any expertise as it relates to the contextual use or communication of Replica. His expertise is simply in testing privacy and security frameworks within applications. If there are particular questions about privacy that remain, I am positively certain we can provide the necessary information / documentation required. Let me know how you would like to proceed.   Thanks. Nick     On Tue, Feb 12, 2019 at 12:09 PM Eliot Rose wrote: Thanks Nick:   Appreciate all of the information; I’ll forward it on to members of the group.   I suspect that we’ll end up requesting that Metro have access to the entire data table, but not the other partners in the group. We haven’t discussed the method by which that would be shared, but I was assuming from the conversation that it would be provided separately from the online tool. Is that correct?   Would you be open to us reaching out to Jean-Louis Tambay to ask him for some more of the background on his findings? I understand why you can’t share his report, but I want to make sure that we do our due diligence on privacy. Staff continue to get a lot of questions about SWL/Replica from the public, and it might help for us to be able to run those questions by him.   Thanks, Eliot   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies.    2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us.    3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes.    4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.      Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 12, 2019 8:45 AM To: Eliot Rose Subject: Re: Portland Replica update   Hi Eliot   Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions:   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies.    2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us.    3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes.    4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.    Let me know if you want to chat by phone.   Thanks. Nick     On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick:   Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call.  I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract.   Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: ·         Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? ·         You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. ·         There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria?      ·         How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts.   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Nick Bowden Eliot Rose Re: Portland Replica update Wednesday, February 13, 2019 8:09:24 AM Yes, we would provide the data table outside of the tool / product itself. Either through an API or a dump of the data table into a cloud bucket you all maintain.  I am happy to make an intro to Jean-Louis if you believe it will help. I think it's important to note that he doesn't have any expertise as it relates to the contextual use or communication of Replica. His expertise is simply in testing privacy and security frameworks within applications. If there are particular questions about privacy that remain, I am positively certain we can provide the necessary information / documentation required. Let me know how you would like to proceed. Thanks. Nick On Tue, Feb 12, 2019 at 12:09 PM Eliot Rose wrote: Thanks Nick:   Appreciate all of the information; I’ll forward it on to members of the group.   I suspect that we’ll end up requesting that Metro have access to the entire data table, but not the other partners in the group. We haven’t discussed the method by which that would be shared, but I was assuming from the conversation that it would be provided separately from the online tool. Is that correct?   Would you be open to us reaching out to Jean-Louis Tambay to ask him for some more of the background on his findings? I understand why you can’t share his report, but I want to make sure that we do our due diligence on privacy. Staff continue to get a lot of questions about SWL/Replica from the public, and it might help for us to be able to run those questions by him.   Thanks, Eliot   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies.    2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us.    3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes.    4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.      Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 12, 2019 8:45 AM To: Eliot Rose Subject: Re: Portland Replica update   Hi Eliot   Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions:   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies.    2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us.    3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes.    4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.    Let me know if you want to chat by phone.   Thanks. Nick     On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick:   Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call.  I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract.   Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: ·         Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? ·         You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. ·         There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria?      ·         How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts.   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Nick Bowden Eliot Rose Re: Portland Replica update Tuesday, February 12, 2019 8:45:25 AM Hi Eliot Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions: 1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies.  2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us.  3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes.  4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.  Let me know if you want to chat by phone. Thanks. Nick On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick:   Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call.  I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract.   Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: ·         Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? ·         You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. ·         There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria?      ·         How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts.   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Nick Bowden Eliot Rose Re: Metro review of Replica agreement Tuesday, January 29, 2019 12:17:21 PM Hi Eliot Just wanted to check in and see how the review process is going? On another note, one thing we have been doing to ease the onboarding process is starting conversations with partner organizations early in the process. Intent of these is to give them a little introduction to the product and also learn about the projects they are working on prior to launching. Is it possible to look at time mid to late February for us to visit and talk with 3-6 of the key agencies involved? Typically try and do an hour per agency over a 2 day period. Thanks. Nick On Fri, Jan 11, 2019 at 5:40 PM Eliot Rose wrote: Thanks Nick! Looking forward to reviewing this. As for the agreement, we’re close to our final review and I don’t expect any changes that are going to be surprising to you. I’ll let you know if any come up or if we have questions about the documentation, but otherwise I’m inclined to let our partners review the current version once they sign the IGA before we send it back to you to minimize back and forth. That could take up to 3 weeks because it’ll take another week to get the IGA signed and then they have up to two weeks for review. I’ll try to keep the process moving and notify you ahead of time if there are significant questions that come up.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, January 11, 2019 2:23 PM To: Eliot Rose Cc: Benjamin Y. Bloom; Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Metro review of Replica agreement   Hi Eliot (and team)   Thanks for taking time to speak this week. As promised, I'm attaching a new version of the data disclosure document that includes our vendor audit process and a summary for the third party privacy audit we had conducted on Replica. I believe you all are going to turn back one final redline for the agreement and then we should be good to go. Please let me know if you have any questions.   Thanks everyone. Have a great weekend! Nick       On Mon, Jan 7, 2019 at 10:48 AM Eliot Rose wrote: Thanks for the heads-up, Nick. It looks like our team could do 10:30 or 11:30 as well if that works better for you. Since we only have a half hour it’d be helpful if everyone’s able to join in for as much of the call as possible.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 8:23 AM To: Eliot Rose; Benjamin Y. Bloom Subject: Re: Metro review of Replica agreement   Hi Eliot   Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed.    Sorry for the last minute change (typing from plane now). Talk soon.   Nick     On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments —seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: -          Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) -          Timeline for signing the contract -          Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Hi Eliot   Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below.   Attaching a few items here:   1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation.   We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created.    Thanks. As always, please let me know if you have questions.    Nick     On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread.   We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.    In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know.   Happy Holidays! Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments —but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Attachments: Nick Bowden Eliot Rose Benjamin Y. Bloom; Chris Johnson; Michelle Bellia; Robert Kirkman Re: Metro review of Replica agreement Friday, January 11, 2019 2:23:30 PM Replica_DataDisclouse_Summary_011119.pdf Hi Eliot (and team) Thanks for taking time to speak this week. As promised, I'm attaching a new version of the data disclosure document that includes our vendor audit process and a summary for the third party privacy audit we had conducted on Replica. I believe you all are going to turn back one final redline for the agreement and then we should be good to go. Please let me know if you have any questions. Thanks everyone. Have a great weekend! Nick On Mon, Jan 7, 2019 at 10:48 AM Eliot Rose wrote: Thanks for the heads-up, Nick. It looks like our team could do 10:30 or 11:30 as well if that works better for you. Since we only have a half hour it’d be helpful if everyone’s able to join in for as much of the call as possible.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 8:23 AM To: Eliot Rose; Benjamin Y. Bloom Subject: Re: Metro review of Replica agreement   Hi Eliot   Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed.    Sorry for the last minute change (typing from plane now). Talk soon.   Nick     On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments— seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: -          Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) -          Timeline for signing the contract -          Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Hi Eliot   Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below.   Attaching a few items here:   1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation.   We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created.    Thanks. As always, please let me know if you have questions.    Nick     On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread.   We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.    In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know.   Happy Holidays! Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments— but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Michelle Bellia; Valerie Eisen; Clifford Higgins Re: Replica agreement - public records request Thursday, January 03, 2019 2:59:01 PM Sorry for not clarifying. It is fine if you turn over the term sheet with the pricing formula redacted. The rest of the information in the term sheet is about timing and product outputs, both of which are contained in the email thread and / or are available publicly. Is this an okay solution? Thanks. Nick On Thu, Jan 3, 2019 at 4:46 PM Eliot Rose wrote: Thanks again for the quick response, Nick. I didn’t see an answer from you about the attachments in the PDF, but based on what you’re telling me about how SWL views trade secrets we plan to disclose the term sheet as redacted in the attached version. The request specifically mentions signed agreements between us and SWL, so there’s a high likelihood that the requestor would petition the DA to challenge the withholding the term sheet, and we won’t be able to defend withholding it. We plan to provide the emails with your redactions incorporated and other attachments omitted.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, January 03, 2019 8:08 AM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen; Clifford Higgins Subject: Re: Replica agreement - public records request   SWL_Review - Eliot Rose Portland Emails_Final.pdf Thanks, Eliot. I've attached an updated redaction request PDF. We do believe these requests fall under the broad definition of either trade secrets or confidentiality as defined by Oregon Law. Broadly speaking, we are the only provider of a Software-as-a-Service, fully calibrated travel-demand model in the United States at this time, which has resulted and will likely continue to result in competing firms looking to uncover as much about our product and business model as possible. Specifically; 1. Our business model of working with MPOs to provide services to all cities and counties, using a per capita pricing model, is the first of its kind and something, particularly the business plan and pricing formula things we consider to be trade secrets. We don't object to the total price paid (as is shown in the term sheet and contract), but the distribution model and pricing formula is something we consider a trade secret at this time.  2. References to other customers, their specific outputs, use cases, acceptance reports, and the like we consider to be trade secrets as well.  3. Our technical process and procedure of combining several disparate data sets, using a synthetic population generator, and then assigning that population movements through an activity generator we consider to be a significant technical trade secret. I suspect the difference with many other data vendors is that most, if not all, have a single source of data and do aggregation as a primary means of privacy-protections. Additionally, the specific sources of this data and how / when it is used is something we consider to be a significant trade secret.  4. Non-public documents that detail the above process or provide more than readily available documentation about our process, methodology, or technical procedures we also consider to fall under the trade secret definition.  5. We also consider our customer release schedule to be confidential. 6. Finally, because the final contract is not being released, any specific reference of clauses or documents associated specifically with the final contract have been redacted. In this second review, I've removed redactions to anything that would be considered common knowledge about our product, process, or customers. Moving forward, happy to flag these items in communications.   Hope this helps. Happy to discuss if needed.    Thanks. Nick           On Wed, Jan 2, 2019 at 7:35 PM Eliot Rose wrote: Hello Nick:   Thanks for turning around your review of our response to the public records request so quickly. I reviewed your suggested redactions and discussed with our attorneys’ office today. I’d encourage you to take one more pass at this before tomorrow (Thursday) COB and make sure that all the redactions you’re requesting are defensible given Oregon law’s definitions of confidential and trade secret (below), and remove those that aren’t. Also, I saw that many of the attachments linked to in the PDF are highlighted—are these attachments that you’re requesting we remove? Happy to hop on a call to discuss any of this tomorrow; I’m available from 8-10 or from 12:30-2:30.   If the requestor pushes back and asks to see the redacted information their request will go before a DA, who may question the redactions as a whole and order us to release a version with all redactions removed if he/she feels they are not well-based in the law. I understand the intent behind many of your proposed redactions given what you’ve told me about how Sidewalk views trade secrets, but some of them seem questionable, especially requesting that we withhold the signed term sheet. Our signed agreements generally tend to be a matter of public record, and this redaction seems likely to be challenged by the requestor. If you’re concerned about the pricing information becoming public record we could just redact the Cost and Cost Certainty sections of the term sheet. The redactions to the text of my emails and the request that we withhold the methodology article that you shared with me, which has been published in a peer-reviewed journal, also seem questionable (assuming that your highlights mean that you want us to withhold that attachment).   Unfortunately, we’ve also got to ask you to flag all associated text that recurs throughout the response. It would also be helpful if you can provide us with a more detailed explanation about why the redactions that you are requesting are necessary – with some of the more established data products we work with high-level descriptions of the methodology and cost, like some of what you’ve flagged for redaction, aren’t always trade secret. Is there something about the unique nature of Replica that makes them a trade secret in this case?   I checked in with the City of Portland today, and I learned that they were also hit with a public records request. Because of the timing of the request—after they approved the IGA, but before they had reviewed the draft agreement—they complied in full, which means that the requestor likely has access to anything that you’ve shared directly with the broader group of agencies that were involved in the conversation, which includes the data sample that you shared with us. In the future can you please flag any documents that you consider to be confidential or trade secret up front?   Thanks, Eliot   Confidential:   192.355(4) Information submitted to a public body in confidence and not otherwise required by law to be submitted, where such information should reasonably be considered confidential, the public body has obliged itself in good faith not to disclose the information, and when the public interest would suffer by the disclosure. Trade secret:   192.345(2) [U]nless the public interest requires disclosure in the particular instance, [t]rade secrets. “Trade secrets,” as used in this section, may include, but are not limited to, any formula, plan, pattern, process, tool, mechanism, compound, procedure, production data, or compilation of information which is not patented, which is known only to certain individuals within an organization and which is used in a business it conducts, having actual or potential commercial value, and which gives its user an opportunity to obtain a business advantage over competitors who do not know or use it.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot   We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data.   As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues.    Thanks again and sorry for all of the trouble! Happy new year! Nick       On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request   Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Thanks, Eliot! We appreciate it.   I will plan to review quickly once received.   Have a great weekend and happy holidays!   Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Hi Eliot   Happy holidays to you all as well.   Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary:   1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases  3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense)   Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started.   Thank you. Nick           On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team:   Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them.   Please let us know if you have any questions or concerns. -Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736               503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Michelle Bellia; Valerie Eisen; Clifford Higgins Re: Replica agreement - public records request Thursday, January 03, 2019 8:08:01 AM SWL_Review - Eliot Rose Portland Emails_Final.pdf Thanks, Eliot. I've attached an updated redaction request PDF. We do believe these requests fall under the broad definition of either trade secrets or confidentiality as defined by Oregon Law. Broadly speaking, we are the only provider of a Software-as-a-Service, fully calibrated travel-demand model in the United States at this time, which has resulted and will likely continue to result in competing firms looking to uncover as much about our product and business model as possible. Specifically; 1. Our business model of working with MPOs to provide services to all cities and counties, using a per capita pricing model, is the first of its kind and something, particularly the business plan and pricing formula things we consider to be trade secrets. We don't object to the total price paid (as is shown in the term sheet and contract), but the distribution model and pricing formula is something we consider a trade secret at this time.  2. References to other customers, their specific outputs, use cases, acceptance reports, and the like we consider to be trade secrets as well.  3. Our technical process and procedure of combining several disparate data sets, using a synthetic population generator, and then assigning that population movements through an activity generator we consider to be a significant technical trade secret. I suspect the difference with many other data vendors is that most, if not all, have a single source of data and do aggregation as a primary means of privacy-protections. Additionally, the specific sources of this data and how / when it is used is something we consider to be a significant trade secret.  4. Non-public documents that detail the above process or provide more than readily available documentation about our process, methodology, or technical procedures we also consider to fall under the trade secret definition.  5. We also consider our customer release schedule to be confidential. 6. Finally, because the final contract is not being released, any specific reference of clauses or documents associated specifically with the final contract have been redacted. In this second review, I've removed redactions to anything that would be considered common knowledge about our product, process, or customers. Moving forward, happy to flag these items in communications. Hope this helps. Happy to discuss if needed.  Thanks. Nick On Wed, Jan 2, 2019 at 7:35 PM Eliot Rose wrote: Hello Nick:   Thanks for turning around your review of our response to the public records request so quickly. I reviewed your suggested redactions and discussed with our attorneys’ office today. I’d encourage you to take one more pass at this before tomorrow (Thursday) COB and make sure that all the redactions you’re requesting are defensible given Oregon law’s definitions of confidential and trade secret (below), and remove those that aren’t. Also, I saw that many of the attachments linked to in the PDF are highlighted—are these attachments that you’re requesting we remove? Happy to hop on a call to discuss any of this tomorrow; I’m available from 8-10 or from 12:302:30.   If the requestor pushes back and asks to see the redacted information their request will go before a DA, who may question the redactions as a whole and order us to release a version with all redactions removed if he/she feels they are not well-based in the law. I understand the intent behind many of your proposed redactions given what you’ve told me about how Sidewalk views trade secrets, but some of them seem questionable, especially requesting that we withhold the signed term sheet. Our signed agreements generally tend to be a matter of public record, and this redaction seems likely to be challenged by the requestor. If you’re concerned about the pricing information becoming public record we could just redact the Cost and Cost Certainty sections of the term sheet. The redactions to the text of my emails and the request that we withhold the methodology article that you shared with me, which has been published in a peer-reviewed journal, also seem questionable (assuming that your highlights mean that you want us to withhold that attachment).   Unfortunately, we’ve also got to ask you to flag all associated text that recurs throughout the response. It would also be helpful if you can provide us with a more detailed explanation about why the redactions that you are requesting are necessary – with some of the more established data products we work with high-level descriptions of the methodology and cost, like some of what you’ve flagged for redaction, aren’t always trade secret. Is there something about the unique nature of Replica that makes them a trade secret in this case?   I checked in with the City of Portland today, and I learned that they were also hit with a public records request. Because of the timing of the request—after they approved the IGA, but before they had reviewed the draft agreement—they complied in full, which means that the requestor likely has access to anything that you’ve shared directly with the broader group of agencies that were involved in the conversation, which includes the data sample that you shared with us. In the future can you please flag any documents that you consider to be confidential or trade secret up front?   Thanks, Eliot   Confidential:   192.355(4) Information submitted to a public body in confidence and not otherwise required by law to be submitted, where such information should reasonably be considered confidential, the public body has obliged itself in good faith not to disclose the information, and when the public interest would suffer by the disclosure. Trade secret:   192.345(2) [U]nless the public interest requires disclosure in the particular instance, [t]rade secrets. “Trade secrets,” as used in this section, may include, but are not limited to, any formula, plan, pattern, process, tool, mechanism, compound, procedure, production data, or compilation of information which is not patented, which is known only to certain individuals within an organization and which is used in a business it conducts, having actual or potential commercial value, and which gives its user an opportunity to obtain a business advantage over competitors who do not know or use it.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot   We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data.   As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues.    Thanks again and sorry for all of the trouble! Happy new year! Nick       On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request   Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Thanks, Eliot! We appreciate it.   I will plan to review quickly once received.   Have a great weekend and happy holidays!   Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Hi Eliot   Happy holidays to you all as well.   Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary:   1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases  3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense)   Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started.   Thank you. Nick           On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team:   Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them.   Please let us know if you have any questions or concerns. -Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736               503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Nick Bowden Eliot Rose Re: Replica agreement - public records request Wednesday, January 02, 2019 1:21:05 PM Just saw this thread.  On deployment, we actually just finished an uncalibrated version of Portland last week, which is really cool. Once we get contract hammered out, we can immediately turn to acceptance criteria and validation plan. We will also need to get moving pretty quickly on collecting ground-truth data, but we have a checklist for that process. I think if you and I make it a goal to try and get contract finalized and signed by end of next week, we will still be on track for late March calibrated deployment. On Wed, Jan 2, 2019 at 1:30 PM Eliot Rose wrote: …and I should have added – thanks to you and your team for your patience and thoroughness about reviewing this. I’m checking to see if we have ways of our end of ensuring that all your intended redactions are carried over in the various quoted text.   Also, as I revisit these emails and our discussion of timelines, I’m noting that we’re behind the Q4 2018 validation texting, but it seems from the more recent discussions that we can start on that soon in 2019. Can you confirm that? I know that y’all have a lot of other work in progress.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, January 02, 2019 11:08 AM To: 'Nick Bowden' Subject: RE: Replica agreement - public records request   Thanks! Reviewing everything now and then talking with our legal at 2:30. Definitely going to check in on the term sheet – I understand your concerns but as I understand it that’s a signed agreement and therefore subject to disclosure.   Sending some times for us to talk later today once I get to check in with the team. Is your plan for the call to just loop in the attorneys and focus on the agreement, or should we plan to cover any outstanding issues re: the public records request and documentation as well?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 10:53 AM To: Eliot Rose Subject: Re: Replica agreement - public records request   Sorry about that! Just shared.   On Wed, Jan 2, 2019 at 12:47 PM Eliot Rose wrote: Hello Nick:   I just tried to open this up and it looks like you shared it with my Metro acct., which isn’t Google-enabled. Can you please share with me at   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot   We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data.   As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues.    Thanks again and sorry for all of the trouble! Happy new year! Nick       On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request   Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Thanks, Eliot! We appreciate it.   I will plan to review quickly once received.   Have a great weekend and happy holidays!   Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Hi Eliot   Happy holidays to you all as well.   Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary:   1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases  3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense)   Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started.   Thank you. Nick           On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team:   Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them.   Please let us know if you have any questions or concerns. -Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736               503-797-1825 www.oregonmetro.gov   Metro Making a great place From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Michelle Bellia; Valerie Eisen Re: Replica agreement - public records request Monday, December 31, 2018 1:10:57 PM SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues.  Thanks again and sorry for all of the trouble! Happy new year! Nick   On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request   Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Thanks, Eliot! We appreciate it.   I will plan to review quickly once received.   Have a great weekend and happy holidays!   Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request   Hi Eliot   Happy holidays to you all as well.   Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary:   1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases  3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense)   Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started.   Thank you. Nick           On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team:   Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them.   Please let us know if you have any questions or concerns. -Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736               503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Attachments: Nick Bowden Eliot Rose Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Re: Metro review of Replica agreement Monday, December 31, 2018 11:08:07 AM (4) SaaSPortland Comparison Metro 20181228.docx Replica_DataDisclouse_010119.pdf Shared Replica Data Model and Schema.pdf Screen Shot 2018-12-26 at 1.38.21 PM.png Hi Eliot Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below. Attaching a few items here: 1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation. We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created.  Thanks. As always, please let me know if you have questions.  Nick On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread.   We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.    In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know.   Happy Holidays! Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Nick Bowden Eliot Rose Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Re: Metro review of Replica agreement Wednesday, December 26, 2018 7:54:23 AM Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread. We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.  In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know. Happy Holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Subject: Date: Attachments: Nick Bowden Eliot Rose; Marie Hlavaty; Valerie Eisen Re: Metro review of Replica agreement Friday, November 30, 2018 10:49:49 AM Replica - Website Terms of Use (2).docx Thanks for sending this Eliot.  There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions. On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this. I am hoping I will be on the call but wanted to send this just in case. Thanks. Nick On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Attachments: Nick Bowden Eliot Rose Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Re: Metro review of Replica agreement Wednesday, November 28, 2018 9:41:12 AM Replica Acceptance Criteria Template_DRAFT (1).docx Calibration Report – Replica KC Jan-Mar 2018 v1.0 (2).pdf Also, forget to attach two items. The first is the most up-to-date template for Acceptance Criteria. This is the format we are using with both NYC and Chicago.  Second, I attached the recently approved calibration report for KC Season 1. This will give you an idea of how we report against the acceptance criteria each quarter.  On Wed, Nov 28, 2018 at 11:22 AM Nick Bowden wrote: Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.  If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.  Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Sean Johnson; Lauren Massey; Chris Johnson RE: [External sender]Re: Portland Replica questions Wednesday, November 20, 2019 11:01:00 AM Great, thanks Nathan! I just followed up separately with Chris and Joe to see about re-uploading OHAS.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Wednesday, November 20, 2019 10:22 AM To: Eliot Rose Cc: Sean Johnson; Lauren Massey; Chris Johnson Subject: Re: [External sender]Re: Portland Replica questions CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! 1. The raw trip table, and the format of that deliverable, will be different than the fields and attributes available for download within the application. We're polling a handful of our users to best understand the schema they'd like to receive it in, and from that, we'll do our best to establish a format to gets you most of what you need. 2. High privacy risk trips and other unique activity sequences get removed during the calibration process. You won't find these data points in the raw data or through the web application. 3. OHAS. I've combed through all of our online Google Drive folders and I'm not able to locate this data set. If it was shared prior, I regret that I do not have access to it. Is that something you can share again? 4. I like your approach to giving folks a high-level overview prior to soliciting feedback on interest use cases and capabilities. Delivering those completed use case spreadsheets after Thanksgiving is fine. We're excited to start building out onboard training materials and the results from those documents will be essential to ensure we dive into the specific areas that hold most value for each agency. Thanks! -Nathan On Tue, Nov 19, 2019 at 4:03 PM Eliot Rose wrote: Thanks Nathan. A couple of follow-ups below. We’re writing our policies for who gets access to Replica data via different formats and what they have to do to protect it, and your answers will help us there. 1. I get that you’re trying to match the trip table to the format we use in our model. My question is, will the data that people are able to query and download from the web tool be in the same format that’s in the trip table? The web tool data that I can currently download via the KC/Chicago builds is more limited in the attributes and level of detail it contains than what we expect from the trip table. That’s not necessarily a bad thing; we just need to understand what level of protection the web tool data requires. 2. Am I correct that outlier trips will not be visible in the web tool? Also, is there anything you’re waiting on from us? You had a question about the OHAS data, but Chris and Joe were both under the impression that we shared that already. Maybe it got lost in the shuffle when the Drive folders were moved?   We’re having a meeting next Tuesday at Metro where we give likely users a high-level overview of Replica and let them know about upcoming training opportunities, and also have them fill out the use case sheets, so I hope to have a bunch of completed sheets for you a little after Thanksgiving. I looked them over and they seem fine, but they’re pretty detailed, so I think that being able to show people the tool before having them fill the sheets out will get you better feedback.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, November 19, 2019 10:20 AM To: Eliot Rose Cc: Sean Johnson; Lauren Massey; Chris Johnson Subject: [External sender]Re: Portland Replica questions CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Team Portland! What a weekend indeed. The Big Red was embarrassed at home by the Badgers. Convenient day to be out of the office - I couldn't handle the shame, Chris. :) 1. In terms of the trip table, we'll do our best to emulate the schema of your trip table. Can't promise that we'll be able to match it exactly. But we will do our best. 2. We work hard to remove all outlying data sets during the calibration process. Remember, we create personas which are abstractions of real people with real movements. Also, access to the trip table should be extremely limited. You'll have to decide who gets access to it, but I'd encourage you to think very carefully about that. 3. The more the merrier! Happy to do another session. Also, any feedback on the use case template I sent over? The more info you can provide, the better we can tailor those onboard sessions. Thanks, Nathan On Mon, Nov 18, 2019 at 7:02 PM Eliot Rose wrote: Hello Nathan and team: Hope y’all enjoyed a restful day of not checking in with us, and that Nathan’s entire day off was restful. Chris was bummed that he didn’t get to boast about the Badgers’ victory over the Huskers though. Chris and I caught up with each other on this project after both of us being out for a spell today, and we had a couple of questions. These will help us get the right policies in place about how our staff and partners use Replica. · Is the trip table in the same format as the data that you get when you query data through the web tool? Or is it more detailed? · Remind us - how does Replica remove / restrict access to potentially identifiable outlier trips? I know we’ve discussed the case of someone who might be identifiable in the data because they have a unique trip (e.g,. they’re that one person who commutes 20 miles each way to downtown along a route with no infrastructure). How do you identify / treat those trips in the data? Do they show up at all via the web tool or in the trip table? · Portland’s Replica info sessions are getting oversubscribed – if they can’t accommodate the demand in a single 30-person training session would you be open to doing a 2nd session at Portland when you visit? If not, I can tell them to call in the bouncers and velvet ropes. Thank you, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Sean Johnson; Lauren Massey; Chris Johnson RE: [External sender]Re: Portland Replica questions Tuesday, November 19, 2019 2:02:00 PM Thanks Nathan. A couple of follow-ups below. We’re writing our policies for who gets access to Replica data via different formats and what they have to do to protect it, and your answers will help us there. 1. I get that you’re trying to match the trip table to the format we use in our model. My question is, will the data that people are able to query and download from the web tool be in the same format that’s in the trip table? The web tool data that I can currently download via the KC/Chicago builds is more limited in the attributes and level of detail it contains than what we expect from the trip table. That’s not necessarily a bad thing; we just need to understand what level of protection the web tool data requires. 2. Am I correct that outlier trips will not be visible in the web tool? Also, is there anything you’re waiting on from us? You had a question about the OHAS data, but Chris and Joe were both under the impression that we shared that already. Maybe it got lost in the shuffle when the Drive folders were moved?   We’re having a meeting next Tuesday at Metro where we give likely users a high-level overview of Replica and let them know about upcoming training opportunities, and also have them fill out the use case sheets, so I hope to have a bunch of completed sheets for you a little after Thanksgiving. I looked them over and they seem fine, but they’re pretty detailed, so I think that being able to show people the tool before having them fill the sheets out will get you better feedback.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, November 19, 2019 10:20 AM To: Eliot Rose Cc: Sean Johnson; Lauren Massey; Chris Johnson Subject: [External sender]Re: Portland Replica questions CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Team Portland! What a weekend indeed. The Big Red was embarrassed at home by the Badgers. Convenient day to be out of the office - I couldn't handle the shame, Chris. :) 1. In terms of the trip table, we'll do our best to emulate the schema of your trip table. Can't promise that we'll be able to match it exactly. But we will do our best. 2. We work hard to remove all outlying data sets during the calibration process. Remember, we create personas which are abstractions of real people with real movements. Also, access to the trip table should be extremely limited. You'll have to decide who gets access to it, but I'd encourage you to think very carefully about that. 3. The more the merrier! Happy to do another session. Also, any feedback on the use case template I sent over? The more info you can provide, the better we can tailor those onboard sessions. Thanks, Nathan On Mon, Nov 18, 2019 at 7:02 PM Eliot Rose wrote: Hello Nathan and team: Hope y’all enjoyed a restful day of not checking in with us, and that Nathan’s entire day off was restful. Chris was bummed that he didn’t get to boast about the Badgers’ victory over the Huskers though. Chris and I caught up with each other on this project after both of us being out for a spell today, and we had a couple of questions. These will help us get the right policies in place about how our staff and partners use Replica. · Is the trip table in the same format as the data that you get when you query data through the web tool? Or is it more detailed? · Remind us - how does Replica remove / restrict access to potentially identifiable outlier trips? I know we’ve discussed the case of someone who might be identifiable in the data because they have a unique trip (e.g,. they’re that one person who commutes 20 miles each way to downtown along a route with no infrastructure). How do you identify / treat those trips in the data? Do they show up at all via the web tool or in the trip table? · Portland’s Replica info sessions are getting oversubscribed – if they can’t accommodate the demand in a single 30-person training session would you be open to doing a 2nd session at Portland when you visit? If not, I can tell them to call in the bouncers and velvet ropes. Thank you, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Sean Johnson; Lauren Massey; Chris Johnson Portland Replica questions Monday, November 18, 2019 5:02:00 PM Hello Nathan and team:   Hope y’all enjoyed a restful day of not checking in with us, and that Nathan’s entire day off was restful. Chris was bummed that he didn’t get to boast about the Badgers’ victory over the Huskers though.   Chris and I caught up with each other on this project after both of us being out for a spell today, and we had a couple of questions. These will help us get the right policies in place about how our staff and partners use Replica. · Is the trip table in the same format as the data that you get when you query data through the web tool? Or is it more detailed? · Remind us - how does Replica remove / restrict access to potentially identifiable outlier trips? I know we’ve discussed the case of someone who might be identifiable in the data because they have a unique trip (e.g,. they’re that one person who commutes 20 miles each way to downtown along a route with no infrastructure). How do you identify / treat those trips in the data? Do they show up at all via the web tool or in the trip table? · Portland’s Replica info sessions are getting oversubscribed – if they can’t accommodate the demand in a single 30-person training session would you be open to doing a 2nd session at Portland when you visit? If not, I can tell them to call in the bouncers and velvet ropes.   Thank you, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Gilligan, Mike; Kerr, Michael; Nathan Preheim McHugh, Tim; Chris Johnson RE: Scheduling Replica visit, 12/16-12/18 Friday, November 15, 2019 10:58:00 AM Hello all:   Here is the draft final schedule for the Replica visit and training sessions from 12/16-12/18. Mike (PBOT) and Mike (TriMet), can you please add address info for your sessions? Once we confirm this I’ll send out calendar invites.   12/16, 8-12 AM: PBOT training session: The City’s 1900 Building in the large conference room (classroom seating, large screens).  12/16: 1-5 PM: Metro Training session: 600 NE Grand Ave., room 370B (seats approx.. 30, has a projector, will be BYO laptop) 12/17: 8-12 AM: TriMet training session: PC training room (approx. 20 pcs that our folks can use with a project/laptop hookups for the presenter/trainer) 12/18: 9:30-12 AM: Replica @ Metro MTAC/TPAC workshop (optional Replica attendance): 600 NE Grand Ave., Council Chambers (45 min. agenda item, timing TBD)   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Gilligan, Mike [mailto:GilligaM@trimet.org] Sent: Friday, November 08, 2019 1:41 PM To: Eliot Rose Cc: McHugh, Tim Subject: [External sender]RE: Scheduling Replica visit, 12/16-12/18     CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. 1. 2. 1st choice Tuesday 12/17 AM, 2nd choice Tuesday 12/17 PM. We can host, I already booked our PC training room so we’ll have approximately 20 pcs that our folks can use with a project/laptop hookups for the presenter/trainer.   I’ll be reaching out to what I consider our potential “power users”, with a target of approximately 20 TriMet folks. In addition, once I get a list of names form department leads, I will ask others to contribute to the use case form.   From: Eliot Rose Sent: Friday, November 8, 2019 10:30 AM To: Kerr, Michael ; Gilligan, Mike ; McHugh, Tim ; Martin, Kevin Subject: Scheduling Replica visit, 12/16-12/18   Hello all:   We nailed down some dates with Replica when they can visit Portland and offer training sessions. They’ll be in town 12/16 and 12/17. They would like to schedule a 3-4 hour training session with the likely Replica “power users” from each of our 3 agencies during that time. Can you please respond to items #1 and 2 below – ideally by next Tuesday so that we can nail down the schedule – and complete the other steps below to help me arrange these?   1. Let me know which time slots work best for folks on your end. Please pick a first and a second choice in case we have conflicts:   a. Monday 12/16 AM (8-12 or so) b. Monday 12/16 PM (1-5 or so) c. Monday 12/17 AM d. Monday 12/17 PM 2. Let me know if you’d like to host or prefer that Metro host. I’d encourage you to host at your own offices to make it easy for folks to attend, but I’m holding some meeting rooms here in case you need them. If you host, you’ll want to book a meeting room with a projector, good wifi, and outlets for folks to plug in on. These training sessions will be BYO laptop, so if people on your end don’t have laptops you’ll need to provide computers for them too. 3. Invite folks from your agency to participate. Replica is a little vague on what defines a “power user,” but in Metro’s case I’m interpreting it to mean people who provide data and analysis support to different departments and people who manage projects that are likely going to draw on Replica data. I’m trying to keep the group to no more than 30 people, and ideally closer to 10-20, so that there’s time for in-depth assistance. 4. Distribute Replica’s use case form (see attached email, which also contains some more info from Replica on how to fill it out) to participants to fill out – or if you prefer, set up a meeting w/ folks ahead of time to fill out a shared version, which is what I’m doing. Replica uses these to tailor training sessions to each agency. Due back to me by 12/6.   Thank you, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nathan Preheim Joe Broach; Chris Johnson RE: [External sender]3rd times a charm... Tuesday, November 12, 2019 10:21:00 AM Portland - Acceptance Criteria v13.docx 935980 - Amendment 1 - Replica, Inc 08192019.pdf Hello Nathan:   Acceptance criteria attached, both in Word and in the contract if it helps to have them in context. Joe will follow up about the trip table and freight data, and Chris may need to weigh in on that when he gets back tomorrow as well, but I just checked with Joe and he noted that the vehicle counts that we sent included a Class field that distinguishes freight from passenger vehicles. Did you see that in the data?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Monday, November 11, 2019 12:41 PM To: Eliot Rose Subject: [External sender]3rd times a charm... CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hey Eliot, Sorry, back to back to back messages. Would you mind sending me the final acceptance criteria we all agreed to? During the SWL migration there's a chance I didn't port over the final final. Thank you very much, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Joe Broach; Chris Johnson Re: [External sender]Portland Replica // Freight ground truth data? Monday, November 11, 2019 12:11:10 PM Thanks Nathan! Adding Chris in, I know we have a new freight model but I'm not sure what data feeds it.  No call today-- Metro is off for Veterans Day. Sorry I didn't catch that sooner. Any updates to share via email?  Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Nov 11, 2019, at 9:54 AM, Nathan Preheim wrote:   CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hello Joe and Eliot! We're all hands on deck getting your model tuned and ready to go. Curious if you all have access to freight ground truth? We'll model it without, but always ask if you have access to ground truth to help us better calibrate the results.  In our world, freight includes heavy, medium, and lightweight commercial vehicles.  Let me know! Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim   Customer Operations   (402) 595-0055   nathan@replicahq.com   http://www.replicahq.com       From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Lauren Massey; Sean Johnson; Chris Johnson; Joe Broach RE: [External sender]Portland Replica Onboard Friday, November 08, 2019 10:33:00 AM Thanks Nathan! I just sent out an email to Portland and TriMet asking them to schedule time slots that work best for people on their end and hold meeting rooms, as well as distribute and return the use case forms. I suggested that they include no more than 30 people and preferably closer to 10-20 on each session and get the forms back to me to compile and send to you by 12/6. I also tried to steer everyone toward time slots on 12/16 and 12/17 so that you’d have the option to head home or stick around for the workshop / other introductory sessions with the broader set of agency partners TBD on Wednesday. Let me know if there’s anything else I can do to make these a success.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Friday, November 08, 2019 6:25 AM To: Eliot Rose Cc: Lauren Massey; Sean Johnson; Chris Johnson; Joe Broach Subject: Re: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot, I like everything you propose. I think we'd like to modify our position to have multiple sessions per agency. Given our new focus on depth-versus-breadth, a single 3-4 hour session per agency would be more than satisfactory. If you want to schedule a morning session with one agency and an afternoon session for another agency that's fine. Or if you'd prefer to dedicate all mornings, or all afternoons, we are extremely flexible. As it stands right now, we'll use feedback gathered from our use case documents to help assess agency need, and tailor to each agency. I will also send you a schema of our data table. I'd love for you to circulate that to the most technical participants and review that, and then let us know which data fields are of particular interest. Thanks, Nathan On Wed, Nov 6, 2019 at 5:04 PM Eliot Rose wrote: Hello Nathan and team:   I just went on a scheduling spree for our Replica rollout – see below. It’s a lot of details and more meeting rooms than we need, but I wanted to grab good rooms because they get booked quickly at Metro and use this to coordinate admin support for the meetings. Can you please answer some questions to help me firm this up? · Seems like 12/16-12/18 works well for your visit, but it’s potentially more time than we need. When within that window do you want to visit us? · How do you want to divide up the training sessions? You mentioned holding 2 sessions per agency – how long do you want each of those sessions to be, and does each of the two sessions cover different content? · Do prefer to hold joint sessions for all 3 agencies at Metro or schedule individual sessions with each agency? If you don’t have a preference, I’ll ask Portland and TriMet if they do. Most of the rooms I’ve booked have enough space for all of us if needed. · Is there anything that’s missing from the timeline below?   Replica schedule: · This week: distribute use case sheet to Portland and TriMet · 11/26, 12-1, Metro room 370 - Internal Replica intro for Metro power users: Eliot will give an overview of Replica, demo some of the tool’s features, and have Metro staff fill out use case sheets. · 12/6, 12-1:30, Metro room 301 – Replica @ Metro emerging tech working group: Eliot will give an overview of Replica and demo some of the tool’s features for Metro and partner agency staff who focus on new modes like ride-hailing and micromobility and get input on how to structure the 12/18 workshop with partner agency staff. · 12/18: 9-12:30, Metro Council Chambers – Replica @ TPAC/MTAC workshop: Metro staff, and potentially Replica staff, will present Replica to Metro’s transportation and land use technical committees, which consist of staff from Metro partner agencies and community representatives. We will give an overview of the tool, facilitate small groups in exploring the data using Metro laptops, and describe how and when partner agencies will be able to access the tool. This is a long meeting with multiple agenda items; I’ve requested 45 minutes. · 12/16-12/18: Replica staff lead BYO laptop training sessions for Metro, PBOT and TriMet power users. I’m holding the following rooms/times at Metro, and PBOT and TriMet may be able to host sessions on site if needed: o 12/16: Holding meeting room 370A (capacity = 30) from 9-12 and 370A/370B (capacity = 60) from 12-5 o 12/17: Holding our Council Chamber (capacity = 125) from 8-12 and room 401 (capacity = 43) from 2-5 o 12/18: Holding room 301 (capacity = 20) for from 12-5 for afternoon sessions following the TPAC/MTAC workshop   Thanks! Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 1:44 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595?0055 nathan replicahg .com From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Lauren Massey; Sean Johnson RE: [External sender]Portland Replica Onboard Thursday, November 07, 2019 12:37:00 PM Thanks! Please let me know about the other scheduling questions when you get the chance. Our admin folks are freaking out because I booked so many rooms and I want to ease their minds. J   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Thursday, November 07, 2019 12:35 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: Re: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! That's about it. Sometimes we make mice available, because zooming in and interacting with the map feature inside Explorer isn't perfect with a trackpad. Maybe a couple of extension cords to connect to outlets. We'll run a presentation from our work computers and will bring along all of the other training materials. So projector is key. We have all of the essential connection dongles. Lastly, robust WiFi is important because we'll likely have 10+ work stations all connected to Replica, which is a data heavy application. Thanks! -Nathan On Thu, Nov 7, 2019 at 2:13 PM Eliot Rose wrote: One more question – what supplies do you need for these meetings? Sounds like we need a projector and power sources for people to plug in their laptops – anything else?   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, November 06, 2019 3:04 PM To: 'Nathan Preheim' Cc: Lauren Massey; Sean Johnson; Chris Johnson; Joe Broach Subject: RE: [External sender]Portland Replica Onboard Hello Nathan and team:   I just went on a scheduling spree for our Replica rollout – see below. It’s a lot of details and more meeting rooms than we need, but I wanted to grab good rooms because they get booked quickly at Metro and use this to coordinate admin support for the meetings. Can you please answer some questions to help me firm this up? · Seems like 12/16-12/18 works well for your visit, but it’s potentially more time than we need. When within that window do you want to visit us? · How do you want to divide up the training sessions? You mentioned holding 2 sessions per agency – how long do you want each of those sessions to be, and does each of the two sessions cover different content? · Do prefer to hold joint sessions for all 3 agencies at Metro or schedule individual sessions with each agency? If you don’t have a preference, I’ll ask Portland and TriMet if they do. Most of the rooms I’ve booked have enough space for all of us if needed. · Is there anything that’s missing from the timeline below?   Replica schedule: · This week: distribute use case sheet to Portland and TriMet · 11/26, 12-1, Metro room 370 - Internal Replica intro for Metro power users: Eliot will give an overview of Replica, demo some of the tool’s features, and have Metro staff fill out use case sheets. · 12/6, 12-1:30, Metro room 301 – Replica @ Metro emerging tech working group: Eliot will give an overview of Replica and demo some of the tool’s features for Metro and partner agency staff who focus on new modes like ride-hailing and micromobility and get input on how to structure the 12/18 workshop with partner agency staff. · 12/18: 9-12:30, Metro Council Chambers – Replica @ TPAC/MTAC workshop: Metro staff, and potentially Replica staff, will present Replica to Metro’s transportation and land use technical committees, which consist of staff from Metro partner agencies and community representatives. We will give an overview of the tool, facilitate small groups in exploring the data using Metro laptops, and describe how and when partner agencies will be able to access the tool. This is a long meeting with multiple agenda items; I’ve requested 45 minutes. · 12/16-12/18: Replica staff lead BYO laptop training sessions for Metro, PBOT and TriMet power users. I’m holding the following rooms/times at Metro, and PBOT and TriMet may be able to host sessions on site if needed: o 12/16: Holding meeting room 370A (capacity = 30) from 9-12 and 370A/370B (capacity = 60) from 12-5 o 12/17: Holding our Council Chamber (capacity = 125) from 8-12 and room 401 (capacity = 43) from 2-5 o 12/18: Holding room 301 (capacity = 20) for from 12-5 for afternoon sessions following the TPAC/MTAC workshop   Thanks! Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 1:44 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Lauren Massey; Sean Johnson RE: [External sender]Portland Replica Onboard Thursday, November 07, 2019 12:13:00 PM One more question – what supplies do you need for these meetings? Sounds like we need a projector and power sources for people to plug in their laptops – anything else?   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, November 06, 2019 3:04 PM To: 'Nathan Preheim' Cc: Lauren Massey; Sean Johnson; Chris Johnson; Joe Broach Subject: RE: [External sender]Portland Replica Onboard Hello Nathan and team:   I just went on a scheduling spree for our Replica rollout – see below. It’s a lot of details and more meeting rooms than we need, but I wanted to grab good rooms because they get booked quickly at Metro and use this to coordinate admin support for the meetings. Can you please answer some questions to help me firm this up? · Seems like 12/16-12/18 works well for your visit, but it’s potentially more time than we need. When within that window do you want to visit us? · How do you want to divide up the training sessions? You mentioned holding 2 sessions per agency – how long do you want each of those sessions to be, and does each of the two sessions cover different content? · Do prefer to hold joint sessions for all 3 agencies at Metro or schedule individual sessions with each agency? If you don’t have a preference, I’ll ask Portland and TriMet if they do. Most of the rooms I’ve booked have enough space for all of us if needed. · Is there anything that’s missing from the timeline below?   Replica schedule: · This week: distribute use case sheet to Portland and TriMet · 11/26, 12-1, Metro room 370 - Internal Replica intro for Metro power users: Eliot will give an overview of Replica, demo some of the tool’s features, and have Metro staff fill out use case sheets. · 12/6, 12-1:30, Metro room 301 – Replica @ Metro emerging tech working group: Eliot will give an overview of Replica and demo some of the tool’s features for Metro and partner agency staff who focus on new modes like ride-hailing and micromobility and get input on how to structure the 12/18 workshop with partner agency staff. · 12/18: 9-12:30, Metro Council Chambers – Replica @ TPAC/MTAC workshop: Metro staff, and potentially Replica staff, will present Replica to Metro’s transportation and land use · technical committees, which consist of staff from Metro partner agencies and community representatives. We will give an overview of the tool, facilitate small groups in exploring the data using Metro laptops, and describe how and when partner agencies will be able to access the tool. This is a long meeting with multiple agenda items; I’ve requested 45 minutes. 12/16-12/18: Replica staff lead BYO laptop training sessions for Metro, PBOT and TriMet power users. I’m holding the following rooms/times at Metro, and PBOT and TriMet may be able to host sessions on site if needed: o 12/16: Holding meeting room 370A (capacity = 30) from 9-12 and 370A/370B (capacity = 60) from 12-5 o 12/17: Holding our Council Chamber (capacity = 125) from 8-12 and room 401 (capacity = 43) from 2-5 o 12/18: Holding room 301 (capacity = 20) for from 12-5 for afternoon sessions following the TPAC/MTAC workshop   Thanks! Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 1:44 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan replicahg .Com IEI From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Lauren Massey; Sean Johnson; Chris Johnson; Joe Broach RE: [External sender]Portland Replica Onboard Wednesday, November 06, 2019 3:04:00 PM Hello Nathan and team:   I just went on a scheduling spree for our Replica rollout – see below. It’s a lot of details and more meeting rooms than we need, but I wanted to grab good rooms because they get booked quickly at Metro and use this to coordinate admin support for the meetings. Can you please answer some questions to help me firm this up? · Seems like 12/16-12/18 works well for your visit, but it’s potentially more time than we need. When within that window do you want to visit us? · How do you want to divide up the training sessions? You mentioned holding 2 sessions per agency – how long do you want each of those sessions to be, and does each of the two sessions cover different content? · Do prefer to hold joint sessions for all 3 agencies at Metro or schedule individual sessions with each agency? If you don’t have a preference, I’ll ask Portland and TriMet if they do. Most of the rooms I’ve booked have enough space for all of us if needed. · Is there anything that’s missing from the timeline below?   Replica schedule: · This week: distribute use case sheet to Portland and TriMet · 11/26, 12-1, Metro room 370 - Internal Replica intro for Metro power users: Eliot will give an overview of Replica, demo some of the tool’s features, and have Metro staff fill out use case sheets. · 12/6, 12-1:30, Metro room 301 – Replica @ Metro emerging tech working group: Eliot will give an overview of Replica and demo some of the tool’s features for Metro and partner agency staff who focus on new modes like ride-hailing and micromobility and get input on how to structure the 12/18 workshop with partner agency staff. · 12/18: 9-12:30, Metro Council Chambers – Replica @ TPAC/MTAC workshop: Metro staff, and potentially Replica staff, will present Replica to Metro’s transportation and land use technical committees, which consist of staff from Metro partner agencies and community representatives. We will give an overview of the tool, facilitate small groups in exploring the data using Metro laptops, and describe how and when partner agencies will be able to access the tool. This is a long meeting with multiple agenda items; I’ve requested 45 minutes. · 12/16-12/18: Replica staff lead BYO laptop training sessions for Metro, PBOT and TriMet power users. I’m holding the following rooms/times at Metro, and PBOT and TriMet may be able to host sessions on site if needed: o 12/16: Holding meeting room 370A (capacity = 30) from 9-12 and 370A/370B (capacity = 60) from 12-5 o 12/17: Holding our Council Chamber (capacity = 125) from 8-12 and room 401 (capacity = 43) from 2-5 o 12/18: Holding room 301 (capacity = 20) for from 12-5 for afternoon sessions following the TPAC/MTAC workshop   Thanks! Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 1:44 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Subject: Date: Eliot Rose Nathan Preheim RE: [External sender]Portland Replica // Use Cases Wednesday, November 06, 2019 9:17:00 AM Thanks Nathan! I’ll see what I can do to beat ARB’s record. The spreadsheet format gives them an edge though… they LOVE spreadsheets over there.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Monday, November 04, 2019 3:30 PM To: Eliot Rose; Joe Broach; Chris Johnson Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica // Use Cases CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! I just shared a cloud version of this document with you. I'm also attaching an Excel version. As we start to prepare for onboarding, the more information we can gather from Portland agencies about relevant use cases the better. This simple worksheet helps us do just that. Feel free to distribute far and wide. An agency can complete it, a division can complete it, a specific role can complete it, or best yet, many different individuals can complete it. All we ask is that we map each completed sheet to a role and agency. Individuals should save their files with this kind of naming format... NP_Planner_Trimet KL_Modeler_Metro FD_Policy Analyst_City of Portland You get the idea. The first two letters are initials. Enough detail for you all to identify, but nearly anonymous for Replica. Does that make sense? As I mentioned, California Air Resource Board currently as a high water mark on returned worksheets at 14. I'll bet you can up that by at least 1. ;) Thanks! -Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Lauren Massey; Sean Johnson RE: [External sender]Portland Replica Onboard Wednesday, October 30, 2019 3:56:00 PM Awesome, thanks! I just emailed Portland and TriMet asking for their input on dates too.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 5:29 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: Re: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Eliot, Let me consult with the broader team and get back to you soon. Appreciate the heads up on that event. A logical opportunity. More soon. Thanks On Tue, Oct 29, 2019 at 7:27 PM Eliot Rose wrote: Thanks Nathan!   A couple of questions: How long is each session? And are you available by any chance the first half of the 3rd week in December? Our workshop with all partners in the region, is 12/18, which will be a good chance to promote Replica more broadly. Sorry I didn’t mention that. If not, no worries; we’ll make some time in the 1st/2nd week work. There’s some good Metro meeting room availability during the latter half of those weeks.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 1:44 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Lauren Massey; Sean Johnson RE: [External sender]Portland Replica Onboard Tuesday, October 29, 2019 5:27:00 PM Thanks Nathan!   A couple of questions: How long is each session? And are you available by any chance the first half of the 3rd week in December? Our workshop with all partners in the region, is 12/18, which will be a good chance to promote Replica more broadly. Sorry I didn’t mention that. If not, no worries; we’ll make some time in the 1st/2nd week work. There’s some good Metro meeting room availability during the latter half of those weeks.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, October 29, 2019 1:44 PM To: Eliot Rose Cc: Lauren Massey; Sean Johnson Subject: [External sender]Portland Replica Onboard CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! We checked calendars today and the first or second week of December will work for us. I envision 1 or 2 full days of stacked meeting for all sessions. That will mostly be a function of number of participants. We think 2 sessions per agency (6 total) will be sufficient. Our team is prepared to consolidate that into a single day if necessary or we can extend to two. Preferred days either week are entirely up to you. Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Subject: Date: Eliot Rose nathan@sidewalklabs.com RE: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) Monday, October 28, 2019 1:33:00 PM We’re getting errors when we try the meeting link. We’re going to call in shortly! Eliot Rose Technology Strategist Metro -----Original Appointment----From: nathan@sidewalklabs.com [mailto:nathan@sidewalklabs.com] Sent: Tuesday, May 14, 2019 11:25 AM To: nathan@sidewalklabs.com; Chris Johnson; Eliot Rose Subject: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) When: Monday, October 28, 2019 3:30 PM-4:00 PM America/Chicago. Where: Google Hangout You have been invited to the following event. more details » Metro + Replica // Weekly Sync When Weekly from 3:30pm to 4pm on Monday Central Time - Chicago Where Google Hangout (map) Joining info meet.google.com/xoj-qapf-jdn Or dial: +1 413-341-4472  PIN: 134354#  More phone numbers Calendar eliot.rose@oregonmetro.gov Who • nathan@sidewalklabs.com - organizer • chris.johnson@oregonmetro.gov • eliot.rose@oregonmetro.gov A weekly check in to ensure this project is progressing swimmingly. The Portland - Issue List will drive the discussion: https://docs.google.com/spreadsheets/d/1l-QKd_weXsRl5QdafsnQ-vAnszOn2aq_pdvSyoSA4CQ/edit#gid=1817381524 Going (eliot.rose@oregonmetro.gov)?   All events in this series:   Yes - Maybe - No    more options » Invitation from Google Calendar You are receiving this courtesy email at the account eliot.rose@oregonmetro.gov because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn More.  << File: invite.ics >> From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Walle Brown Metro - New Replica team member Monday, October 21, 2019 11:24:00 AM Hello Nathan:   Bryan had to move on, so I’m looping in another planner, Walle Brown, to take on some of the work that he was doing helping us prepare some outreach and instructional materials. Can you please give Walle (cc’d) access to Replica? You can also remove Bryan’s login.   Thanks, and talk soon! Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Nick Bowden RE: [External sender]Re: Responding to latest Replica public records request Thursday, October 17, 2019 1:59:00 PM Thanks Nathan! Apologies for not looping in Nick; I should have thought of that. I’ll consider it OK to share the prior response unless I hear otherwise from Nick before tomorrow COB, and appreciate any other thoughts Nick has on documents that might include sensitive information.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Wednesday, October 16, 2019 8:53 PM To: Eliot Rose Cc: Nick Bowden Subject: [External sender]Re: Responding to latest Replica public records request CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Eliot, I'm attaching Nick to this email because he is aware of that prior request. I don't believe we will object to sharing previously assembled information. In terms of 2019 shared content, there are a couple of technical white-papers that we provided that delve into the nuances of how we construct our models. Those materials were made available inside a FAQ folder located on a shared folder. However, that folder existed on our prior Sidewalk Labs domain, and I have not re-shared that content. If you have saved, local copies, we would prefer contents not be disclosed as they likely include sensitive information that would be considered intellectual property. Nick, any other documents that we would request not be disclosed? Thanks, Nathan On Wed, Oct 16, 2019 at 6:56 PM Eliot Rose wrote: Hello Nathan: We’re preparing our response to the public records request that I notified you about last week. It’s a very detailed and comprehensive request, and we’re trying to expedite the release of the appropriate records while still giving you at least 3 business days to review the responsive materials, per our agreement. There are a couple of things that I’d like to draw to your attention to and get your feedback on. Can you please respond to this email by the end of the week? A significant amount of the materials that the reviewer requested are included in our previous response to the Wall Street Journal. We intend to provide that response, including the redactions that Metro and Replica previously agreed to, instead of re-compiling these materials and asking you to review them again. I’m assuming that you’re OK with that, and that you don’t need to review those materials again since they’ve already been reviewed by your team. Please let me know if you have any concerns with that approach. That response is attached for reference. Are there specific materials that you’ve provided us in 2019 that you consider trade secret and are likely to request that we not disclose in response to this request? If so, it would help streamline the process if you sent me a list of those materials and an explanation of why Replica considers them trade secret. We’ll still compile all of the responsive materials and let you review the response before sending, but with the WSJ request I recall that the backand-forth about what exactly constituted a trade secret took up a fair amount of time, so I think that having this information up front will help us get back to this requestor promptly. Here (in section 2) is the OR definition of trade secret in case it helps to refer to that. Thank you, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Subject: Date: Attachments: Eliot Rose Nathan Preheim Responding to latest Replica public records request Wednesday, October 16, 2019 4:56:00 PM Eliot Rose Portland Emails Redacted redaction process applied and attach....pdf Hello Nathan:   We’re preparing our response to the public records request that I notified you about last week. It’s a very detailed and comprehensive request, and we’re trying to expedite the release of the appropriate records while still giving you at least 3 business days to review the responsive materials, per our agreement. There are a couple of things that I’d like to draw to your attention to and get your feedback on. Can you please respond to this email by the end of the week?   A significant amount of the materials that the reviewer requested are included in our previous response to the Wall Street Journal. We intend to provide that response, including the redactions that Metro and Replica previously agreed to, instead of re-compiling these materials and asking you to review them again. I’m assuming that you’re OK with that, and that you don’t need to review those materials again since they’ve already been reviewed by your team. Please let me know if you have any concerns with that approach. That response is attached for reference.   Are there specific materials that you’ve provided us in 2019 that you consider trade secret and are likely to request that we not disclose in response to this request? If so, it would help streamline the process if you sent me a list of those materials and an explanation of why Replica considers them trade secret. We’ll still compile all of the responsive materials and let you review the response before sending, but with the WSJ request I recall that the back-and-forth about what exactly constituted a trade secret took up a fair amount of time, so I think that having this information up front will help us get back to this requestor promptly. Here (in section 2) is the OR definition of trade secret in case it helps to refer to that.   Thank you, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Joe Broach; Chris Johnson Portland Replica check-ins Monday, October 14, 2019 2:41:00 PM Hello Nathan:   Sorry we missed you today! I just checked in with Joe and Chris. The only remaining dataset we have left on our to-share list is transit data from C-TRAN, which serves one of the four counties in our region. We’ve reached out a contact at C-TRAIN for that data and are hopeful we’ll be able to get it to you soon, but our understanding is that it’s not on the critical path, and that you’re already moving ahead with calibrating the model and can continue to do so without that data.   If I’m right about that, then we expect to see delivery of the first calibrated dataset in midNovember, and that we’ll devote our check-ins between now and then to status updates from you. Does that sound right to you? And if so, should we continue to hold weekly check-ins by phone, or switch to email updates? Joe is available if you have any questions about the data or want to discuss any additional calibration data needs.   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Nathan Preheim New public records request Thursday, October 10, 2019 8:27:13 AM Hello Nathan:  We just got another public records request re: Replica - and it’s pretty exhaustive. I’ll provide info on what we intend to disclose in a couple of days once we get it assembled, and then we’ll send it over to you to review per the agreement. But I wanted to give you a heads-up now since there’s likely going to be a lot of material.  -Eliot Eliot Rose Technology Strategist Metro eliot.rose@oregonmetro.gov From: To: Subject: Date: Eliot Rose Nathan Preheim RE: [External sender]Re: Portland Replica updates Thursday, October 03, 2019 3:26:00 PM Great, thanks! Good to have an option. I checked in w/ the chairs and I think the most likely scenario is that we’ll punt the workshop. I want to get people excited about this ASAP, but they’ll get more excited if they’re looking at data for their communities.   Can you please remind me what type of onboarding you plan to do for users once we accept the data? I want to be mindful of the add’l educational opportunities that you’ll be making available when I present this.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Thursday, October 03, 2019 12:55 PM To: Eliot Rose Subject: Re: [External sender]Re: Portland Replica updates CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! 100%! Might be a good way to build anticipation. Walk through the entire Replica experience, in anticipation for the Portland region. The Seasonal Comparison report. The Transit Report. The interesting Chicago and KC use cases. I'd strongly encourage that. Confident your audience would enjoy it, even without Portland-centric data. Thanks, Nathan On Thu, Oct 3, 2019 at 2:49 PM Eliot Rose wrote: Awesome, thanks! That’s helpful to know. Sounds like it’s better to punt to November. Given that I’m getting notice to folks late I think there might be some pressure to fill agenda space on the 16th. Would it be an option to present data from Chicago or KC?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Thursday, October 03, 2019 12:46 PM To: Eliot Rose Cc: Chris Johnson; Joe Broach Subject: [External sender]Re: Portland Replica updates CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! 4-6 weeks post all data sets received is when we can anticipate a release date of the calibrated Replica for the Portland region. If everything was delivered this week - or soon after - we could plan on early November. In terms of the event, I don't think we'll quite have the full experience ready to go. As you know, we have an un-calibrated version available to Explore, but we haven't yet enabled Reports or Downloads. I think you'd rather offer up a comprehensive experience, correct? If so, I'd suggest a re-schedule. Thanks, Nathan On Thu, Oct 3, 2019 at 12:28 PM Eliot Rose wrote: Thanks Joe! Nathan, can you please let me know when we can expect to see the first round of data for our region? I need to know whether we should postpone this 10/16 workshop we have planned. I’m available til 2:30 if you want to chat – 503.797.1825. Thanks! Eliot Rose Technology Strategist Metro From: Joe Broach Sent: Wednesday, October 02, 2019 10:51 PM To: Eliot Rose; Nathan Preheim Cc: Chris Johnson Subject: RE: Portland Replica updates Hi all, All auto and transit data are now posted except for C-TRAN (Vancouver transit), and we don’t currently have an ETA on that. Please check notes in the file headers and status sheet comments for additional details. Bike & ped data are staged, but after review I’ll need to take another pass tomorrow before finalizing and uploading. I hope to get to them by the end of the day and will update when I do. Cheers, joe From: Eliot Rose Sent: Monday, September 30, 2019 1:41 PM To: Nathan Preheim Cc: Joe Broach ; Chris Johnson Subject: Portland Replica updates Hello Nathan: Sorry we missed you today! Hope that the onboarding is going well. I just checked in with the team and the main news is that, barring any unforeseen challenges, Joe plans to finish uploading our ground truth data by the end of the day tomorrow, and he’ll also plan to send along a recap of all the files that are in the folder. Given that progress, when do you think that we’ll receive the first uncalibrated version of Replica? We have a presentation to our technical committeees on the tool scheduled for 10/16, and I was hoping to let them explore the online data as part of the exercise. We can reschedule if the Portland isn’t going to be ready by then. Thanks, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Subject: Date: Eliot Rose Nathan Preheim RE: [External sender]Re: Portland Replica updates Thursday, October 03, 2019 12:49:00 PM Awesome, thanks! That’s helpful to know. Sounds like it’s better to punt to November. Given that I’m getting notice to folks late I think there might be some pressure to fill agenda space on the 16th. Would it be an option to present data from Chicago or KC?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Thursday, October 03, 2019 12:46 PM To: Eliot Rose Cc: Chris Johnson; Joe Broach Subject: [External sender]Re: Portland Replica updates CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Eliot! 4-6 weeks post all data sets received is when we can anticipate a release date of the calibrated Replica for the Portland region. If everything was delivered this week - or soon after - we could plan on early November. In terms of the event, I don't think we'll quite have the full experience ready to go. As you know, we have an un-calibrated version available to Explore, but we haven't yet enabled Reports or Downloads. I think you'd rather offer up a comprehensive experience, correct? If so, I'd suggest a re-schedule. Thanks, Nathan On Thu, Oct 3, 2019 at 12:28 PM Eliot Rose wrote: Thanks Joe! Nathan, can you please let me know when we can expect to see the first round of data for our region? I need to know whether we should postpone this 10/16 workshop we have planned. I’m available til 2:30 if you want to chat – 503.797.1825. Thanks! Eliot Rose Technology Strategist Metro From: Joe Broach Sent: Wednesday, October 02, 2019 10:51 PM To: Eliot Rose; Nathan Preheim Cc: Chris Johnson Subject: RE: Portland Replica updates Hi all, All auto and transit data are now posted except for C-TRAN (Vancouver transit), and we don’t currently have an ETA on that. Please check notes in the file headers and status sheet comments for additional details. Bike & ped data are staged, but after review I’ll need to take another pass tomorrow before finalizing and uploading. I hope to get to them by the end of the day and will update when I do. Cheers, joe From: Eliot Rose Sent: Monday, September 30, 2019 1:41 PM To: Nathan Preheim Cc: Joe Broach ; Chris Johnson Subject: Portland Replica updates Hello Nathan: Sorry we missed you today! Hope that the onboarding is going well. I just checked in with the team and the main news is that, barring any unforeseen challenges, Joe plans to finish uploading our ground truth data by the end of the day tomorrow, and he’ll also plan to send along a recap of all the files that are in the folder. Given that progress, when do you think that we’ll receive the first uncalibrated version of Replica? We have a presentation to our technical committeees on the tool scheduled for 10/16, and I was hoping to let them explore the online data as part of the exercise. We can reschedule if the Portland isn’t going to be ready by then. Thanks, Eliot Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Importance: Eliot Rose Nathan Preheim Chris Johnson; Joe Broach RE: Portland Replica updates Thursday, October 03, 2019 10:28:00 AM High Thanks Joe! Nathan, can you please let me know when we can expect to see the first round of data for our region? I need to know whether we should postpone this 10/16 workshop we have planned. I’m available til 2:30 if you want to chat – 503.797.1825. Thanks!   Eliot Rose Technology Strategist Metro   From: Joe Broach Sent: Wednesday, October 02, 2019 10:51 PM To: Eliot Rose; Nathan Preheim Cc: Chris Johnson Subject: RE: Portland Replica updates   Hi all,   All auto and transit data are now posted except for C-TRAN (Vancouver transit), and we don’t currently have an ETA on that. Please check notes in the file headers and status sheet comments for additional details.   Bike & ped data are staged, but after review I’ll need to take another pass tomorrow before finalizing and uploading. I hope to get to them by the end of the day and will update when I do.   Cheers, joe   From: Eliot Rose Sent: Monday, September 30, 2019 1:41 PM To: Nathan Preheim Cc: Joe Broach ; Chris Johnson Subject: Portland Replica updates   Hello Nathan:   Sorry we missed you today! Hope that the onboarding is going well. I just checked in with the team and the main news is that, barring any unforeseen challenges, Joe plans to finish uploading our ground truth data by the end of the day tomorrow, and he’ll also plan to send along a recap of all the files that are in the folder.   Given that progress, when do you think that we’ll receive the first uncalibrated version of Replica? We have a presentation to our technical committeees on the tool scheduled for 10/16, and I was hoping to let them explore the online data as part of the exercise. We can reschedule if the Portland isn’t going to be ready by then.   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Joe Broach; Chris Johnson Portland Replica updates Monday, September 30, 2019 1:41:00 PM Hello Nathan:   Sorry we missed you today! Hope that the onboarding is going well. I just checked in with the team and the main news is that, barring any unforeseen challenges, Joe plans to finish uploading our ground truth data by the end of the day tomorrow, and he’ll also plan to send along a recap of all the files that are in the folder.   Given that progress, when do you think that we’ll receive the first uncalibrated version of Replica? We have a presentation to our technical committeees on the tool scheduled for 10/16, and I was hoping to let them explore the online data as part of the exercise. We can reschedule if the Portland isn’t going to be ready by then.   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Joe Broach RE: [External sender]Re: Data folder transition Tuesday, September 24, 2019 2:25:00 PM Full list (I think; excluding inter-regional services). This site tends to be pretty up to date: · Blue Star (airport shuttle) · C-Tran · CCC Xpress · Ride Connection · Sandy Area Metro · South Metro Area Regional Transit · Swan Island Shuttle · TriMet (I believe their feed includes Portland Streetcar) · Washington Park Shuttle · Water Ave. Shuttle       Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Tuesday, September 24, 2019 7:00 AM To: Eliot Rose Cc: Joe Broach Subject: Re: [External sender]Re: Data folder transition CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Eliot! I'll take the complete list if you don't mind. I've got time reserved on the calendar today to search and retrieve Portland-area GTFS data. Thank you, Nathan On Mon, Sep 23, 2019 at 6:44 PM Eliot Rose wrote: …also, Joe just reminded me that there are a number of smaller agencies in our region with their own GTFS feeds. I focused on the ones that carry significant numbers of riders, but please let me know if you want a complete list!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, September 23, 2019 2:28 PM To: 'Nathan Preheim'; Joe Broach Cc: Chris Johnson Subject: RE: [External sender]Re: Data folder transition Good catching up today, Nathan. Here’s a quick list of the transit agencies in our region so that you know which GTFS feeds to pull down: · TriMet · Portland Streetcar · South Metro Area Rapid Transit (SMART) · C-TRAN (Clark County, WA)   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Monday, September 23, 2019 1:28 PM To: Joe Broach Cc: Eliot Rose; Chris Johnson Subject: [External sender]Re: Data folder transition CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Joe! Saw your most recent file update - thanks for that! I duplicated it and placed it inside our Replica shared drive. Also, I'd moved the Data Collection Status worksheet back into the folder you have access to. Thanks, Nathan On Mon, Sep 23, 2019 at 2:45 PM Joe Broach wrote: Hey Nathan, We can discuss on today’s call but wanted to make sure I understand the GDrive folder transition to replicahq. A couple notes/questions: · I uploaded a file list to the new Data folder of all the files that had been posted to the old folder. Please let me know if I should re-upload any to the new one. · Where should we be updating the data status? Is there a new link for that also? Thanks! joe -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Subject: Date: Eliot Rose Nathan Preheim; Joe Broach RE: [External sender]Re: Data folder transition Monday, September 23, 2019 4:44:00 PM …also, Joe just reminded me that there are a number of smaller agencies in our region with their own GTFS feeds. I focused on the ones that carry significant numbers of riders, but please let me know if you want a complete list!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, September 23, 2019 2:28 PM To: 'Nathan Preheim'; Joe Broach Cc: Chris Johnson Subject: RE: [External sender]Re: Data folder transition Good catching up today, Nathan. Here’s a quick list of the transit agencies in our region so that you know which GTFS feeds to pull down: · TriMet · Portland Streetcar · South Metro Area Rapid Transit (SMART) · C-TRAN (Clark County, WA)   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Monday, September 23, 2019 1:28 PM To: Joe Broach Cc: Eliot Rose; Chris Johnson Subject: [External sender]Re: Data folder transition CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Joe! Saw your most recent file update - thanks for that! I duplicated it and placed it inside our Replica shared drive. Also, I'd moved the Data Collection Status worksheet back into the folder you have access to. Thanks, Nathan On Mon, Sep 23, 2019 at 2:45 PM Joe Broach wrote: Hey Nathan, We can discuss on today’s call but wanted to make sure I understand the GDrive folder transition to replicahq. A couple notes/questions: · I uploaded a file list to the new Data folder of all the files that had been posted to the old folder. Please let me know if I should re-upload any to the new one. · Where should we be updating the data status? Is there a new link for that also? Thanks! joe -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim; Joe Broach Chris Johnson RE: [External sender]Re: Data folder transition Monday, September 23, 2019 2:27:00 PM Good catching up today, Nathan. Here’s a quick list of the transit agencies in our region so that you know which GTFS feeds to pull down: · TriMet · Portland Streetcar · South Metro Area Rapid Transit (SMART) · C-TRAN (Clark County, WA)   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Monday, September 23, 2019 1:28 PM To: Joe Broach Cc: Eliot Rose; Chris Johnson Subject: [External sender]Re: Data folder transition CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Joe! Saw your most recent file update - thanks for that! I duplicated it and placed it inside our Replica shared drive. Also, I'd moved the Data Collection Status worksheet back into the folder you have access to. Thanks, Nathan On Mon, Sep 23, 2019 at 2:45 PM Joe Broach wrote: Hey Nathan, We can discuss on today’s call but wanted to make sure I understand the GDrive folder transition to replicahq. A couple notes/questions: · I uploaded a file list to the new Data folder of all the files that had been posted to the old folder. Please let me know if I should re-upload any to the new one. · Where should we be updating the data status? Is there a new link for that also? Thanks! joe -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Subject: Eliot Rose nathan@replicahq.com Accepted: [External sender]Updated invitation: Metro // Replica - Weekly Sync @ Mon Sep 23, 2019 3:30pm 4pm (CDT) (eliot.rose@oregonmetro.gov) From: To: Subject: Eliot Rose nathan@replicahq.com Accepted: [External sender]Invitation: Metro // Replica - Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) From: To: Subject: Date: Eliot Rose Nick Bowden Re: [External sender]Re: Metro Contract No. 935980, Amendment No. 1 Thursday, September 19, 2019 10:42:37 AM Thanks Nick! And congratulations on the CEO position. Seems like you’ve been acting in that role for a while, but Nathan made it sound like it’s now official with the official establishment of Replica as its own company.  Eliot Rose Technology Strategist Metro eliot.rose@oregonmetro.gov On Sep 19, 2019, at 10:22 AM, Nick Bowden wrote:   CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Sherrie Signed version attached. Thank you! Nick On Tue, Sep 10, 2019 at 1:47 PM Sherrie Blackledge wrote: Please find attached a PDF file of the above-referenced document for Scope Clarification of the Software Subscription.  Please review and sign, then return a PDF/scanned version to me at your earliest convenience.  Once Metro has signed, I will send you a fully-executed original for your files.   Please contact me with any questions you may have.  Thanks for your help in this matter.     Sherrie Blackledge Senior Management Analyst   My gender pronouns: she, her, hers.   Metro oregonmetro.gov 600 NE Grand Avenue Portland, OR 97232-2736 503-797-1724   <935980 REPLICA Amendment 1_SIGNED.pdf> From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Joe Broach RE: [External sender]Metro Replica // Status Call Monday, September 16, 2019 12:45:00 PM Good to hear from you, Nathan! I was just realizing that I didn’t have your new email address on file.   We’re OK to skip today’s meeting. We don’t have anything pressing to discuss on our end. And now Joe knows where to find you if any questions come up for him.   I got the folder link and forwarded it onto Joe. Can you also please give him access to the webtool with the KC data?   Thanks for keeping the contract moving – I think that’s it for now.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@replicahq.com] Sent: Monday, September 16, 2019 12:20 PM To: Eliot Rose Subject: [External sender]Metro Replica // Status Call CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Hi Eliot, I'm traveling today and will be unable to talk at our usual time. I just sent you a new link to upload data sets. We are no longer operating on the Sidewalk Labs domain. Can you please notify Joe? https://drive.google.com/drive/folders/11iLiHSSeX62017NEu1trDj2QW-YJgA1H He might need to request access. Sorry about that. Also, we received the updated contract with the addendum. Thanks - we'll get that returned to you soon. Anything else urgent? Thanks, Nathan -–––––––––––––––––––––––––      Nathan Preheim Customer Operations (402) 595-0055 nathan@replicahq.com http://www.replicahq.com    From: To: Cc: Subject: Date: Eliot Rose Chris Johnson; nathan@sidewalklabs.com Joe Broach RE: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (chris.johnson@oregonmetro.gov) Monday, September 16, 2019 11:11:00 AM Joe and I are still planning on checking in with you, Nathan! Also, Joe needs access to the web tool – can you please help with that? Til soon, Eliot Eliot Rose Technology Strategist Metro _____________________________________________ From: Chris Johnson Sent: Monday, September 16, 2019 11:10 AM To: nathan@sidewalklabs.com; Eliot Rose Subject: RE: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (chris.johnson@oregonmetro.gov) Eliot and NathanI’m in Maui this week. Carry on without me ;) Cheers! Chris -----Original Appointment----From: nathan@sidewalklabs.com [mailto:nathan@sidewalklabs.com] Sent: Tuesday, May 14, 2019 11:25 AM To: nathan@sidewalklabs.com; Chris Johnson; Eliot Rose Subject: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (chris.johnson@oregonmetro.gov) When: Monday, September 16, 2019 3:30 PM-4:00 PM America/Chicago. Where: Google Hangout You have been invited to the following event. more details » Metro + Replica // Weekly Sync When         Weekly from 3:30pm to 4pm on Monday Central Time - Chicago         Where         Google Hangout (map)         Joining info         meet.google.com/xoj-qapf-jdn                 Or dial: +1 413-341-4472  PIN: 134354#  More phone numbers                 Calendar         chris.johnson@oregonmetro.gov         Who         •         nathan@sidewalklabs.com - organizer         •         chris.johnson@oregonmetro.gov         •         eliot.rose@oregonmetro.gov                 A weekly check in to ensure this project is progressing swimmingly. The Portland - Issue List will drive the discussion: https://docs.google.com/spreadsheets/d/1l-QKd_weXsRl5QdafsnQ-vAnszOn2aq_pdvSyoSA4CQ/edit#gid=1817381524 Going (chris.johnson@oregonmetro.gov)?   All events in this series:   Yes - Maybe - No    more options »         Invitation from Google Calendar You are receiving this courtesy email at the account chris.johnson@oregonmetro.gov because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others         regardless of their own invitation status, or to modify your RSVP. Learn More. From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson; Joe Broach RE: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) Monday, August 05, 2019 1:19:00 PM Great, thanks! I believe TriMet data still hasn’t been uploaded to Drive; Joe is staging everything for upload and documenting any data points that we might want to withhold for independent validation. I bet he can give us a more thorough update when he’s back in the office.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, August 05, 2019 1:17 PM To: Eliot Rose Cc: Chris Johnson; Joe Broach Subject: Re: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) Hi Eliot! Email update is fine. I’ll check the folder today. Keep up the momentum - talk to you next week. Thanks, Nathan On Mon, Aug 5, 2019 at 1:15 PM Eliot Rose wrote: Hello Nathan:   Joe is out today, so we don’t have an in-depth update on data collection. Do you want to skip our call and do an email update instead?   I know Joe has been hard at work and I imagine that he has a few updates. All I can tell you is that we have received all of the TriMet data in the format you specified, and the spreadsheet has been updated to reflect that.   -Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: nathan@sidewalklabs.com [mailto:nathan@sidewalklabs.com] Sent: Tuesday, May 14, 2019 11:25 AM To: nathan@sidewalklabs.com; Chris Johnson; Eliot Rose Subject: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) When: Monday, August 05, 2019 3:30 PM-4:00 PM America/Chicago. Where: Google Hangout     You have been invited to the following event. more details » Metro + Replica // Weekly Sync When Where Joining info Weekly from 3:30pm to 4pm on Monday Central Time - Chicago Google Hangout (map) meet.google.com/xoj-qapf-jdn Or dial: +1 413-341-4472 PIN: 134354# More phone numbers Calendar eliot.rose@oregonmetro.gov • Who • • nathan@sidewalklabs.com - organizer chris.johnson@oregonmetro.gov eliot.rose@oregonmetro.gov A weekly check in to ensure this project is progressing swimmingly. The Portland - Issue List will drive the discussion: https://docs.google.com/spreadsheets/d/1l-QKd_weXsRl5QdafsnQ-vAnszOn2aq_pdvSyoSA4CQ/edit#gid=1817381524 Going (eliot.rose@oregonmetro.gov)?   All events in this series:   Yes - Maybe - No    more options » Invitation from Google Calendar You are receiving this courtesy email at the account eliot.rose@oregonmetro.gov because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn More. << File: invite.ics >>   -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose nathan@sidewalklabs.com; Chris Johnson; Joe Broach RE: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) Monday, August 05, 2019 1:15:00 PM Hello Nathan: Joe is out today, so we don’t have an in-depth update on data collection. Do you want to skip our call and do an email update instead? I know Joe has been hard at work and I imagine that he has a few updates. All I can tell you is that we have received all of the TriMet data in the format you specified, and the spreadsheet has been updated to reflect that. -Eliot Eliot Rose Technology Strategist Metro -----Original Appointment----From: nathan@sidewalklabs.com [mailto:nathan@sidewalklabs.com] Sent: Tuesday, May 14, 2019 11:25 AM To: nathan@sidewalklabs.com; Chris Johnson; Eliot Rose Subject: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) When: Monday, August 05, 2019 3:30 PM-4:00 PM America/Chicago. Where: Google Hangout You have been invited to the following event. more details » Metro + Replica // Weekly Sync When         Weekly from 3:30pm to 4pm on Monday Central Time - Chicago         Where         Google Hangout (map)         Joining info         meet.google.com/xoj-qapf-jdn                 Or dial: +1 413-341-4472  PIN: 134354#  More phone numbers                 Calendar         eliot.rose@oregonmetro.gov         Who         •         nathan@sidewalklabs.com - organizer         •         chris.johnson@oregonmetro.gov         •         eliot.rose@oregonmetro.gov                 A weekly check in to ensure this project is progressing swimmingly. The Portland - Issue List will drive the discussion: https://docs.google.com/spreadsheets/d/1l-QKd_weXsRl5QdafsnQ-vAnszOn2aq_pdvSyoSA4CQ/edit#gid=1817381524 Going (eliot.rose@oregonmetro.gov)?   All events in this series:   Yes - Maybe - No    more options »    Invitation from Google Calendar You are receiving this courtesy email at the account eliot.rose@oregonmetro.gov because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others         regardless of their own invitation status, or to modify your RSVP. Learn More.  << File: invite.ics >> From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Metro Replica // Before and after data examples Monday, July 29, 2019 9:38:00 AM Yes, it was a dense article. I think that most of us – as well as other readers who might not have deep knowledge of the topic – are still processing. Happy to discuss today.   I’ll share the schema w/ Joe and we can discuss any further questions on the call today. Are there any significant changes between this version 1.2 and the v1.1 that’s posted on Drive that you’d like to call our attention to?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, July 29, 2019 8:16 AM To: Eliot Rose Subject: Metro Replica // Before and after data examples Hey Eliot, I'm still reading through that RedTailMedia article! Dense. I had talked to Nick about your quote and we didn't object. I certainly would have followed up with you expediently if we did. Any local blowback as a result? In terms of an example of pre and post normalized data, the best thing to offer is an example of our ground truth schema. This defines exactly what we need. Think of it as "best case scenario" so don't stress if you can't accommodate everything within. Thanks, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim Replica DB and more media coverage Wednesday, July 24, 2019 12:36:00 PM Hello Nathan:   Kate Kaye, a reporter who’s covered our Replica pilot in the past, has reached out about another story, asking about the contract we have with you, which she obtained through a public records request. She asked about quoting us regarding our interpretation of the contract clauses relating to public records requests, and I wanted to share that quote with you. Please give me a call if you have any concerns.   "We have not waived our right to disagree with Replica about what should be disclosed. If Metro believes that material requested should be disclosed under Oregon public records law we can elect to disclose that material. What this language is saying is that, if Replica did disagree with our decision to disclose material, we wouldn’t object if they wanted to make an argument against disclosure directly to the district attorney – but there’s nothing in this language that prevents us from coming to our own determination about what should be disclosed. "   Also, thanks for letting us know about the size of the Replica database! Just to confirm, the O-D table is 10 GB in size, right? And what’s the format – is it a relational database or a flat file?   -Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Replica check-ins Tuesday, July 09, 2019 5:32:00 PM Thanks Nathan!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, July 09, 2019 2:10 PM To: Eliot Rose Cc: Joe Broach Subject: Re: Replica check-ins Joe! You should have the invitation on your calendar. Talk to you next week. Thanks, Nathan On Mon, Jul 8, 2019 at 6:52 PM Eliot Rose wrote: Hello Nathan: Can you please add Joe Broach (cc’d) to our weekly Replica check-ins? Thanks! Eliot Rose (he/him/his) Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Replica check-ins Tuesday, July 09, 2019 11:33:00 AM Also – and we can discuss this at future check-ins if you want – at some point soon it would help us to see a dataset that you’ve taken from our group and normalized. Looking at the before-and-after would help us understand how to normalize the datasets that we are withholding. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, July 08, 2019 4:53 PM To: 'Nathan Preheim' Cc: Joe Broach Subject: Replica check-ins   Hello Nathan:   Can you please add Joe Broach (cc’d) to our weekly Replica check-ins? Thanks!   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Joe Broach Replica check-ins Monday, July 08, 2019 4:52:00 PM Hello Nathan:   Can you please add Joe Broach (cc’d) to our weekly Replica check-ins? Thanks!   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Portland Replica // Weekly Sync Monday, July 01, 2019 2:47:00 PM Thanks Nathan!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, July 01, 2019 2:35 PM To: Eliot Rose Subject: Re: Portland Replica // Weekly Sync Eliot! I've give that document a thorough review. Keen to see how you qualified "wiggle room". :) Apologies - my user error. The development team built some new functionality that lets me create user accounts, and I wasn't executing it properly. Fixed on my side. Bryan's account has been created. He should be good to go. As a reminder, be sure Bryan first clicks on the Sign Up. That'll prompt him to create a new password. Thanks again, Nathan On Mon, Jul 1, 2019 at 12:52 PM Eliot Rose wrote: Hello Nathan:   Let’s bump the meeting. I’m going to work with Chris to get the data collection moving quickly so that we have something to check in on next time.   I’m attaching an updated version of the acceptance criteria. No changes to the criteria other than to finalize what we discussed last week, but I brought in more front matter, mostly drawn from the next that is in the acceptance criteria section of the signed contract, and added some language about giving you all some wiggle room in case there are trade-offs between meeting different sets of criteria (highlighted). I haven’t run this by Chris yet, but if you want to let me know what you think it will help us keep moving.   Also, I’ve been able to log into the new Replica homepage, but Bryan hasn’t – when he tries to create a new account he receives the email to confirm the account, but when he clicks the link in that email he gets the message, “You don't have access to Replica.” If it helps, his email is Bryan.Nguyen@oregonmetro.gov.   Thanks!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, July 01, 2019 11:59 AM To: Eliot Rose Subject: Portland Replica // Weekly Sync Hi Eliot! Checking in to see if you have any noteworthy updates. If so, by all means let's keep the meeting on the calendar. If not, I'd propose that we table until next week. Finalizing acceptance criteria and moving forward with data collection are the only items that are on my radar right now. Let me know your thoughts! Onward, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Attachments: Eliot Rose Nathan Preheim RE: Portland Replica // Weekly Sync Monday, July 01, 2019 12:51:00 PM Portland - Acceptance Criteria v13.docx Hello Nathan:   Let’s bump the meeting. I’m going to work with Chris to get the data collection moving quickly so that we have something to check in on next time.   I’m attaching an updated version of the acceptance criteria. No changes to the criteria other than to finalize what we discussed last week, but I brought in more front matter, mostly drawn from the next that is in the acceptance criteria section of the signed contract, and added some language about giving you all some wiggle room in case there are trade-offs between meeting different sets of criteria (highlighted). I haven’t run this by Chris yet, but if you want to let me know what you think it will help us keep moving.   Also, I’ve been able to log into the new Replica homepage, but Bryan hasn’t – when he tries to create a new account he receives the email to confirm the account, but when he clicks the link in that email he gets the message, “You don't have access to Replica.” If it helps, his email is Bryan.Nguyen@oregonmetro.gov.   Thanks!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, July 01, 2019 11:59 AM To: Eliot Rose Subject: Portland Replica // Weekly Sync Hi Eliot! Checking in to see if you have any noteworthy updates. If so, by all means let's keep the meeting on the calendar. If not, I'd propose that we table until next week. Finalizing acceptance criteria and moving forward with data collection are the only items that are on my radar right now. Let me know your thoughts! Onward, Nathan -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim FW: Replica transit race/income criteria Thursday, June 27, 2019 10:43:00 AM Hey Nathan! Looking forward to chatting in a minute. Chris will be out but I checked in with him in advance and I think we’ll be able to move things forward just fine with only me on the call from Metro’s side. Below for discussion is a proposed approach to validating race/ethnicity and income of transit riders; I can explain more on the call. TriMet hasn’t approved this yet, and I’m asking for their take as well, but I’d be curious to know if you feel comfortable with this approach.     Race <1,000 Trips/day 52% Income Max. MOE 28% 1k-2k Trips/day 23% 13% 2k-5k Trips/day 22% 11% 5k-10k Trips/day 15% 7% 10k-20k Trips/day 12% 5% >20k Trips/day 9% 4% Avg. Weekday Ridership Max. MOE Trips/day (%RMSE) Avg. Weekday Ridership (ground truth) <1,000 Trips/day Acceptable Preferable Race (White/non-hispanic and Hispanic/non-white, Absolute) Acceptable Preferable Personal income (>$40k/year and <$40k/year, Absolute) Acceptable Preferable +/- 150% +/- 100% 75% 50% 40% 30% 1k-2k Trips/day +/- 100% +/- 65% 35% 25% 20% 15% 2k-5k Trips/day +/- 65% +/- 35% 25% 20% 15% 10% 5k-10k Trips/day +/- 35% +/- 25% 20% 15% 15% 10% 10k-20k Trips/day +/- 25% +/- 20% 15% 10% 10% 5% >20k Trips/day +/- 20% +/- 15% 15% 10% 10% 5%   Eliot Rose Technology Strategist Metro   From: To: Subject: Date: Eliot Rose Nathan Preheim; Bryan Nguyen RE: Latest acceptance criteria draft and next check-in Wednesday, June 26, 2019 4:20:00 PM Thanks Nathan!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, June 26, 2019 1:32 PM To: Eliot Rose; Bryan Nguyen Subject: Re: Latest acceptance criteria draft and next check-in Hi Eliot / Bryan! Good news. Bryan's Replica account has been created. Here's an article that will guide you through the account creation process, Bryan. https://intercom.help/replica/articles/3094438-signing-in-to-your-replica-account Also, we pushed a significant update to the Homepage yesterday. Lots of new functionality. Let me know if you have any specific questions. Reminder - you've been granted access to the Kansas City Replica, so all of the content is KC-centric. Thanks! -Nathan On Tue, Jun 25, 2019 at 12:36 PM Eliot Rose wrote: Just checking in on this – were you able to give Bryan access to the KC tool? Thanks! Eliot Rose Technology Strategist Metro From: Eliot Rose Sent: Monday, June 17, 2019 5:45 PM To: 'Nathan Preheim' Subject: RE: Latest acceptance criteria draft and next check-in Also, can you please give Bryan Nguyen (Bryan.Nguyen@oregonmetro.gov) access to the KC demo version of the tool that you shared with our team? He’s going to help create some instructional materials for when we roll out the tool, and I’d like him to get familiar with it. Thanks! Eliot Rose Technology Strategist Metro From: Eliot Rose Sent: Monday, June 17, 2019 5:13 PM To: 'Nathan Preheim' Cc: Chris Johnson Subject: Latest acceptance criteria draft and next check-in Hello Nathan: Latest draft of the acceptance criteria are attached. All outstanding / discussion items are flagged with comments. Note the confidential watermark – please mark any documents that you consider confidential as such moving forward; it will help clarify that this is sensitive info for our team as we engage other partners, and streamline responding to public records requests in the future. Chris and I aren’t available during the regular check-in time next Monday. Can we reschedule for next Tuesday at noon? We can also do this Wednesday at noon if you’d like to check in sooner and keep the process moving. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Eliot Rose nathan@sidewalklabs.com Accepted: Invitation: Portland Metro / Replica - AC Review @ Thu Jun 27, 2019 1pm - 1:30pm (CDT) (eliot.rose@oregonmetro.gov) From: To: Cc: Subject: Date: Attachments: Eliot Rose Nathan Preheim Chris Johnson RE: Declined: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) Monday, June 24, 2019 4:37:00 PM Portland - Acceptance Criteria v11.docx Great! Let’s do Thursday @11.   Latest draft of the acceptance criteria attached. I still have a little homework to do on the transit demographic criteria before the call, but it’s getting close!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, June 24, 2019 7:40 AM To: Eliot Rose Subject: Re: Declined: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) Hi Eliot! That works for me. Name the time and we'll get it booked. Thanks much, Nathan On Fri, Jun 21, 2019 at 7:09 PM Eliot Rose wrote: Hello Nathan:   Chris and I are unavailable during this time. We’ll need to reschedule – how does Thursday morning look for you?   -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Eliot Rose nathan@sidewalklabs.com Chris Johnson Declined: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) Hello Nathan: Chris and I are unavailable during this time. We’ll need to reschedule – how does Thursday morning look for you? From: To: Subject: Eliot Rose nathan@sidewalklabs.com Accepted: Invitation: Portland Replica // Acceptance Criteria Finalization @ Wed Jun 19, 2019 2pm - 2:45pm (CDT) (eliot.rose@oregonmetro.gov) From: To: Subject: Date: Eliot Rose Nathan Preheim RE: slides/demo and a public records request Tuesday, June 18, 2019 9:07:00 AM Awesome, thanks!   Best time to catch me and Chris is going to be tomorrow at noon – he’s out Thursday/Friday. We could also chat on the shoulders of today’s data collection conversation (before is preferable to after, but both are available) if that’s not going to be too much Portland for you.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 18, 2019 9:02 AM To: Eliot Rose Subject: Re: slides/demo and a public records request Hi Eliot! These folks have been added too. Totally agree on finding some middle ground on releasing acceptance criteria. Anything we can do to build confidence and dispel misperceptions is important. Nick is out this week and I'd have to confer with him on what is suitable for release. I will indeed set up another discussion for later this week on criteria. What works best for you and Chris? Let me know and I'll get that discussion scheduled. Thanks, Nathan On Mon, Jun 17, 2019 at 12:32 PM Eliot Rose wrote: Thanks Nathan! We won’t disclose the draft acceptance criteria document that I sent. I understand that the testing process is unique to Replica and is part of your business model. At the same time, I wouldn’t be surprised if we continue to get inquiries like this, and I do think that sharing some information about the acceptance criteria would help people understand why this is such a unique opportunity for our region, because the testing process gives us confidence that this is a responsible investment in accurate, anonymized data.  Is there any information on the acceptance criteria that you would be comfortable with sharing publicly?   I’m planning to send a revised version of the acceptance criteria later today with comments. I think that we’ll want to discuss the scooter data and ped data before we finalize. Would you like to try and set up something later in the week for that, or can we tackle it tomorrow / hold off til next Monday’s check-in?   Looking forward to the data collection kickoff tomorrow. In addition to adding Mike Gilligan from TriMet, can you please add the following folks from Portland: Michael.Kerr@portlandoregon.gov   joshua.lynch@portlandoregon.gov   Michael.McDonald@portlandoregon.gov Aubrey.Lindstrom@portlandoregon.gov Happy to forward the invite to them instead if you’d prefer.   -Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Sunday, June 16, 2019 3:23 PM To: Eliot Rose Subject: Re: slides/demo and a public records request Hi Eliot, We heard back from legal. They consider the acceptance criteria document proprietary, as it's entirely unique to measuring Replica quality and releasing it has the potential to cause business harm. So I'd ask that you not share that document. We don't have any issues with sharing the contract. Also, here's a high-level deck that you can use to jumpstart the conversation with the modelers group. Thanks, Nathan On Wed, Jun 12, 2019 at 6:27 PM Eliot Rose wrote: Thanks! I appreciate it.   One more thing to add to your list: Metro received a public records request from the local reporter who wrote the Geekwire story on Portland’s Replica pilot that ran recently. She is requesting the agreement between Metro and Replica and the acceptance criteria.   Per our agreement, I’m providing you with the versions of those documents that we intend to disclose. Can you please let me know if Replica considers any of this information confidential? If so, we’d appreciate it if you can provide any redactions by next Tuesday COB so that we can respond to the request in a timely fashion.   Thanks again, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, June 12, 2019 9:35 AM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! Honestly, this should only take me minutes. I've got something that's pretty good as a starting point. It's just a matter of subtracting slides that aren't applicable. I'll add this to my Friday task list. You'll have it by EOD Friday. Thanks, Nathan On Tue, Jun 11, 2019 at 5:45 PM Eliot Rose wrote: OK, thanks! I’m happy to cobble together something from existing slides if it saves you effort.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 11, 2019 3:25 PM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! I can throw something together for you. Give me a coupe days to prepare a 5-6 slide overview. And yes, you can certainly demo KC Replica. More soon! Thanks, Nathan On Tue, Jun 11, 2019 at 5:02 PM Eliot Rose wrote: Hello Nathan: I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there. I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Attachments: Eliot Rose Nathan Preheim Chris Johnson Latest acceptance criteria draft and next check-in Monday, June 17, 2019 5:12:00 PM Portland - Acceptance Criteria v10.docx Hello Nathan:   Latest draft of the acceptance criteria are attached. All outstanding / discussion items are flagged with comments. Note the confidential watermark – please mark any documents that you consider confidential as such moving forward; it will help clarify that this is sensitive info for our team as we engage other partners, and streamline responding to public records requests in the future.   Chris and I aren’t available during the regular check-in time next Monday. Can we reschedule for next Tuesday at noon? We can also do this Wednesday at noon if you’d like to check in sooner and keep the process moving.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Nathan Preheim RE: slides/demo and a public records request Monday, June 17, 2019 10:32:00 AM Thanks Nathan! We won’t disclose the draft acceptance criteria document that I sent. I understand that the testing process is unique to Replica and is part of your business model. At the same time, I wouldn’t be surprised if we continue to get inquiries like this, and I do think that sharing some information about the acceptance criteria would help people understand why this is such a unique opportunity for our region, because the testing process gives us confidence that this is a responsible investment in accurate, anonymized data.  Is there any information on the acceptance criteria that you would be comfortable with sharing publicly?   I’m planning to send a revised version of the acceptance criteria later today with comments. I think that we’ll want to discuss the scooter data and ped data before we finalize. Would you like to try and set up something later in the week for that, or can we tackle it tomorrow / hold off til next Monday’s check-in?   Looking forward to the data collection kickoff tomorrow. In addition to adding Mike Gilligan from TriMet, can you please add the following folks from Portland: Michael.Kerr@portlandoregon.gov   joshua.lynch@portlandoregon.gov   Michael.McDonald@portlandoregon.gov Aubrey.Lindstrom@portlandoregon.gov Happy to forward the invite to them instead if you’d prefer.   -Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Sunday, June 16, 2019 3:23 PM To: Eliot Rose Subject: Re: slides/demo and a public records request Hi Eliot, We heard back from legal. They consider the acceptance criteria document proprietary, as it's entirely unique to measuring Replica quality and releasing it has the potential to cause business harm. So I'd ask that you not share that document. We don't have any issues with sharing the contract. Also, here's a high-level deck that you can use to jumpstart the conversation with the modelers group. Thanks, Nathan On Wed, Jun 12, 2019 at 6:27 PM Eliot Rose wrote: Thanks! I appreciate it.   One more thing to add to your list: Metro received a public records request from the local reporter who wrote the Geekwire story on Portland’s Replica pilot that ran recently. She is requesting the agreement between Metro and Replica and the acceptance criteria.   Per our agreement, I’m providing you with the versions of those documents that we intend to disclose. Can you please let me know if Replica considers any of this information confidential? If so, we’d appreciate it if you can provide any redactions by next Tuesday COB so that we can respond to the request in a timely fashion.   Thanks again, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, June 12, 2019 9:35 AM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! Honestly, this should only take me minutes. I've got something that's pretty good as a starting point. It's just a matter of subtracting slides that aren't applicable. I'll add this to my Friday task list. You'll have it by EOD Friday. Thanks, Nathan On Tue, Jun 11, 2019 at 5:45 PM Eliot Rose wrote: OK, thanks! I’m happy to cobble together something from existing slides if it saves you effort.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 11, 2019 3:25 PM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! I can throw something together for you. Give me a coupe days to prepare a 5-6 slide overview. And yes, you can certainly demo KC Replica. More soon! Thanks, Nathan On Tue, Jun 11, 2019 at 5:02 PM Eliot Rose wrote: Hello Nathan: I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there. I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim RE: slides/demo and a public records request Friday, June 14, 2019 3:47:00 PM Hello Nathan:   Just wanted to confirm that you saw the heads-up about the public records request that I sent on Wednesday. I want to make sure that you get a chance to review before we respond.   Also, I’ve been rounding up names from TriMet and Portland to add to the invite for the data collection meeting on Tuesday. Still getting names from Portland, but for TriMet you can add Mike Gilligan - GilligaM@trimet.org   I’ve been following up on the questions from our last check-in and revising the acceptance criteria. I may not have a version to share until right before the call, but know that we’re getting closer – it seems like we will have some solid scooter data to include in the evaluation, but we’re not sure that our ped data is going to meet some of the needs that we discussed on the last call.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, June 12, 2019 4:27 PM To: 'Nathan Preheim' Subject: slides/demo and a public records request Thanks! I appreciate it.   One more thing to add to your list: Metro received a public records request from the local reporter who wrote the Geekwire story on Portland’s Replica pilot that ran recently. She is requesting the agreement between Metro and Replica and the acceptance criteria.   Per our agreement, I’m providing you with the versions of those documents that we intend to disclose. Can you please let me know if Replica considers any of this information confidential? If so, we’d appreciate it if you can provide any redactions by next Tuesday COB so that we can respond to the request in a timely fashion.   Thanks again, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, June 12, 2019 9:35 AM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! Honestly, this should only take me minutes. I've got something that's pretty good as a starting point. It's just a matter of subtracting slides that aren't applicable. I'll add this to my Friday task list. You'll have it by EOD Friday. Thanks, Nathan On Tue, Jun 11, 2019 at 5:45 PM Eliot Rose wrote: OK, thanks! I’m happy to cobble together something from existing slides if it saves you effort.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 11, 2019 3:25 PM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! I can throw something together for you. Give me a coupe days to prepare a 5-6 slide overview. And yes, you can certainly demo KC Replica. More soon! Thanks, Nathan On Tue, Jun 11, 2019 at 5:02 PM Eliot Rose wrote: Hello Nathan: I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there. I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Attachments: Eliot Rose Nathan Preheim slides/demo and a public records request Wednesday, June 12, 2019 4:27:00 PM 935980 Replica Agreement_ReplicaSigned.pdf Portland - Acceptance Criteria v8 notes.docx Thanks! I appreciate it.   One more thing to add to your list: Metro received a public records request from the local reporter who wrote the Geekwire story on Portland’s Replica pilot that ran recently. She is requesting the agreement between Metro and Replica and the acceptance criteria.   Per our agreement, I’m providing you with the versions of those documents that we intend to disclose. Can you please let me know if Replica considers any of this information confidential? If so, we’d appreciate it if you can provide any redactions by next Tuesday COB so that we can respond to the request in a timely fashion.   Thanks again, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, June 12, 2019 9:35 AM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! Honestly, this should only take me minutes. I've got something that's pretty good as a starting point. It's just a matter of subtracting slides that aren't applicable. I'll add this to my Friday task list. You'll have it by EOD Friday. Thanks, Nathan On Tue, Jun 11, 2019 at 5:45 PM Eliot Rose wrote: OK, thanks! I’m happy to cobble together something from existing slides if it saves you effort.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 11, 2019 3:25 PM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! I can throw something together for you. Give me a coupe days to prepare a 5-6 slide overview. And yes, you can certainly demo KC Replica. More soon! Thanks, Nathan On Tue, Jun 11, 2019 at 5:02 PM Eliot Rose wrote: Hello Nathan: I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there. I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim RE: slides/demo for Oregon modelers" presentation? Tuesday, June 11, 2019 3:45:00 PM OK, thanks! I’m happy to cobble together something from existing slides if it saves you effort.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 11, 2019 3:25 PM To: Eliot Rose Subject: Re: slides/demo for Oregon modelers' presentation? Hi Eliot! I can throw something together for you. Give me a coupe days to prepare a 5-6 slide overview. And yes, you can certainly demo KC Replica. More soon! Thanks, Nathan On Tue, Jun 11, 2019 at 5:02 PM Eliot Rose wrote: Hello Nathan: I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there. I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim slides/demo for Oregon modelers" presentation? Tuesday, June 11, 2019 3:02:00 PM Hello Nathan:   I’m pinch-hitting for Chris at a meeting of the Oregon Modelers’ group next week, discussing Metro’s work with Replica as part of a roundtable on innovative public agency data projects. It should be a good opportunity to get our peers interested in this work and considering how they might innovate in the future. Do you have a Replica explainer deck that I could borrow from when presenting? And would it be OK if I showed a quick demo of the KC version of the tool that we have access to? I can present materials during the webinar without sharing them with the group if you’d rather that the files not get out there.   I understand if you’d rather not share this stuff too widely, and I’m happy to come up with my own slides if you have concerns.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Eliot Rose nathan@sidewalklabs.com Accepted: Invitation: Portland Replica // Data Collection Kickoff @ Tue Jun 18, 2019 5:30pm - 6:15pm (CDT) (eliot.rose@oregonmetro.gov) From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson; Joe Broach; Maribeth Todd; Luis Murillo Replica data collection kickoff meeting Monday, June 10, 2019 5:56:00 PM Hello Nathan:   I’m cc’ing the folks at Metro who should be included in the Replica data collection meeting. It’s a little hard to find windows in the upcoming week. Monday between 10 and 1 looks best, but Thursday @ 3:30-4 could work for everyone and most could stick around til 4:30, and we may be able to find a time that works for most but not all of us on Friday afternoon. (All times Pacific.)   I’ll get you data collection contacts from TriMet and PBOT shortly!   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Attachments: Eliot Rose Nathan Preheim Chris Johnson RE: revised acceptance criteria Monday, June 10, 2019 12:19:00 PM Portland - Acceptance Criteria v8.docx Looking forward to chatting soon, Nathan! Slightly updated version of the criteria attached; no significant changes other than two new criteria that are flagged with a comment. We hope to spend today’s call reviewing the comments that we’ve flagged for you.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, June 05, 2019 4:48 PM To: 'Nathan Preheim' Cc: Chris Johnson Subject: revised acceptance criteria   Hello Nathan:   Hope the family vacation is going well! Updated acceptance criteria attached. Not final yet, but we’re getting close! There are several comments in there for you that we can discuss on Monday.   Til then, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Attachments: Eliot Rose Nathan Preheim Chris Johnson revised acceptance criteria Wednesday, June 05, 2019 4:48:00 PM Portland - Acceptance Criteria v7.docx Hello Nathan:   Hope the family vacation is going well! Updated acceptance criteria attached. Not final yet, but we’re getting close! There are several comments in there for you that we can discuss on Monday.   Til then, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson RE: Reschedule today"s call Tuesday, June 04, 2019 12:55:00 PM Yes, please spend time with your family! That probably works better for Chris anyway given all his recent travel.   We’re collecting lots of feedback on the acceptance criteria and will have a revised draft to share before we next talk.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, June 04, 2019 12:45 PM To: Eliot Rose Subject: Re: Reschedule today's call Hey Elio! I'm actually OOO this week on a family vacation. Would it be alright if we pushed this until next week? I'm certainly available if need be. Thanks, Nathan On Mon, Jun 3, 2019 at 10:17 AM Eliot Rose wrote: Great! Can you move the appt. since you own it? Eliot Rose Technology Strategist Metro eliot.rose@oregonmetro.gov   On Jun 3, 2019, at 8:13 AM, Nathan Preheim wrote: Sounds good. Thanks On Mon, Jun 3, 2019 at 9:51 AM Eliot Rose wrote: Hello Nathan: Chris is back from his travels but at a conference through Wednesday @ noon. Can we reschedule today’s check-in call to 1:30 on Wednesday? Getting plenty of good feedback on the acceptance criteria, so we’ll have some progress to discuss there. Eliot Rose Technology Strategist Metro eliot.rose@oregonmetro.gov   -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim Re: Reschedule today"s call Monday, June 03, 2019 8:17:50 AM Great! Can you move the appt. since you own it?  Eliot Rose Technology Strategist Metro eliot.rose@oregonmetro.gov On Jun 3, 2019, at 8:13 AM, Nathan Preheim wrote: Sounds good.  Thanks  On Mon, Jun 3, 2019 at 9:51 AM Eliot Rose wrote: Hello Nathan:  Chris is back from his travels but at a conference through Wednesday @ noon. Can we reschedule today’s check-in call to 1:30 on Wednesday? Getting plenty of good feedback on the acceptance criteria, so we’ll have some progress to discuss there.  Eliot Rose Technology Strategist Metro eliot.rose@oregonmetro.gov -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson RE: Portland Replica // Status Update Thursday, May 30, 2019 10:13:00 AM Sounds good, Nathan! I’ll await your call on when would be best to reschedule that visit.     The reporter who wrote that for GeekWire was very dedicated. She definitely dug deeper on the facts and the relationship to other public agency business than any of the other reporters who have contacted us on this topic. I expect some follow-ups to come.   Acceptance criteria are out to everyone for feedback. We’ll get this moving and relay any questions we hear when we check in on Monday, and shoot to get them finalized ASAP.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Thursday, May 30, 2019 9:08 AM To: Eliot Rose Cc: Chris Johnson Subject: Re: Portland Replica // Status Update Eliot! Great, thanks for the heads up. In talking with Nick I think it might be most advantageous to delay that trip a bit - and reschedule a tad closer to launch. That will likely be more fruitful. In the meantime we can go full bore on data collection so that we get this model up and running as quickly as possible. I can interface with anyone and everyone that will be sourcing data virtually. Also, I enjoyed the recent article in GeekWire. We received a ton of favorable feedback about it and I was surprised at the breadth of reach that article had! Talk to you next week! Thanks, Nathan On Wed, May 29, 2019 at 9:22 AM Eliot Rose wrote: Thanks Nathan! This all makes sense. I just spoke with Chris. He’s going to take another pass at the acceptance criteria and then we’re going to circulate to Portland and TriMet for their sign-off and try to get any feedback from them by our call next Monday so that we can discuss on our call - we especially need our partners to weigh in on the transit and scooter-related criteria. When I send those out I’ll also direct people to the data collection resources in the folder so that they can prepare, and mention the timing of your visit. Chris is available 6/11-6/13 so you can book your trip.   Happy to have a call if there’s anything else to discuss!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, May 29, 2019 8:23 AM To: Eliot Rose; Chris Johnson Subject: Portland Replica // Status Update Eliot, Thanks for the call yesterday. Couple key takeaways… First, it’s imperative we split the acceptance criteria flow from the data collection flow. These are independent activities. I’m confident that Chris will soon review and “accept” the proposed terms that we recently delivered. There might be a slight modification on a couple attributes, but we’re keen to nail this down. Second, let’s get moving on data collection. This is typically the most intensive effort of the workflow as there are multiple agencies involved and lots of people that will need to be engaged. I’ve created a document that identifies all of the required attributes for both traffic and transit data. Complete details can be found in the Ground Truth Data Formatting document within the 3. DATA folder. You and I have discussed data collection on a cursory level - time to dig in and start that effort in earnest. There are template emails that can be used (Data Collection Template) to assist with the data requests. You can management that process, or I can step in and work directly with the data partners. Expediency is key here. I’d like to have all data collected within the next two weeks. We’ve lost a little traction over the past couple of weeks and I’d like to regain momentum. Please let me know if you have questions about next steps. Happy to jump on a quick call anytime Thursday to discuss. Thanks, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim; Chris Johnson RE: Portland Replica // Status Update Wednesday, May 29, 2019 9:22:00 AM Thanks Nathan! This all makes sense. I just spoke with Chris. He’s going to take another pass at the acceptance criteria and then we’re going to circulate to Portland and TriMet for their sign-off and try to get any feedback from them by our call next Monday so that we can discuss on our call - we especially need our partners to weigh in on the transit and scooter-related criteria. When I send those out I’ll also direct people to the data collection resources in the folder so that they can prepare, and mention the timing of your visit. Chris is available 6/11-6/13 so you can book your trip.   Happy to have a call if there’s anything else to discuss!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, May 29, 2019 8:23 AM To: Eliot Rose; Chris Johnson Subject: Portland Replica // Status Update Eliot, Thanks for the call yesterday. Couple key takeaways… First, it’s imperative we split the acceptance criteria flow from the data collection flow. These are independent activities. I’m confident that Chris will soon review and “accept” the proposed terms that we recently delivered. There might be a slight modification on a couple attributes, but we’re keen to nail this down. Second, let’s get moving on data collection. This is typically the most intensive effort of the workflow as there are multiple agencies involved and lots of people that will need to be engaged. I’ve created a document that identifies all of the required attributes for both traffic and transit data. Complete details can be found in the Ground Truth Data Formatting document within the 3. DATA folder. You and I have discussed data collection on a cursory level - time to dig in and start that effort in earnest. There are template emails that can be used (Data Collection Template) to assist with the data requests. You can management that process, or I can step in and work directly with the data partners. Expediency is key here. I’d like to have all data collected within the next two weeks. We’ve lost a little traction over the past couple of weeks and I’d like to regain momentum. Please let me know if you have questions about next steps. Happy to jump on a quick call anytime Thursday to discuss. Thanks, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 402-595-0055 nathan Sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Eliot Rose nathan@sidewalklabs.com Accepted: Updated invitation: Metro + Replica // Weekly Sync @ Tue May 28, 2019 11am - 11:30am (CDT) (eliot.rose@oregonmetro.gov) From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson RE: Today"s check-in Friday, May 24, 2019 6:34:00 AM Hello Nathan:   Chris will be working in the mornings next week, so can we move our check in to next Tuesday sometime between 9-10 PT? Wednesday 9-10:30 and 11-12 also work.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, May 23, 2019 7:38 AM To: 'Nathan Preheim' Subject: RE: Today's check-in Thanks for the response, Nathan! I’m sorry I’ve been slow to get back to you. Talking with Chris about how best to keep this moving – a lot of this is in his court and he’s got some time away from the office scheduled in the coming weeks. I’ll try and get you an update by tomorrow.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, May 20, 2019 2:29 PM To: Eliot Rose Cc: Chris Johnson Subject: Re: Today's check-in Hi Eliot! No problem. The two big open issues are nailing down acceptance criteria and gearing up data collection. Also, I'd love your feedback on this OD matrix sample report. https://docs.google.com/spreadsheets/d/1r_JQWgB_v8mlvLeOG79KoD3E0LMyRgU4PkIBplS_0A/edit#gid=1932389190 We're going to be enhancing our data download capabilities in the near future and would love to know if these data points are useful. Aside from that, nothing pressing. Keen to keep moving forward. I'll shift next Monday's meeting to Tuesday if that's alright with you. Let me know if you have any feedback on the acceptance criteria in the interim. Thanks, Nathan On Mon, May 20, 2019 at 1:22 PM Eliot Rose wrote: Hello Nathan: Chris and I both have conflicts during today's check-in call. Would you like to reschedule for later in the week, or should we just skip this week since we checked in Friday (we should also reschedule next Monday's call since it's Memorial Day). I'll be more help scheduling stuff once I'm not in transit and back on wifi. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Today"s check-in Thursday, May 23, 2019 7:37:00 AM Thanks for the response, Nathan! I’m sorry I’ve been slow to get back to you. Talking with Chris about how best to keep this moving – a lot of this is in his court and he’s got some time away from the office scheduled in the coming weeks. I’ll try and get you an update by tomorrow.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, May 20, 2019 2:29 PM To: Eliot Rose Cc: Chris Johnson Subject: Re: Today's check-in Hi Eliot! No problem. The two big open issues are nailing down acceptance criteria and gearing up data collection. Also, I'd love your feedback on this OD matrix sample report. https://docs.google.com/spreadsheets/d/1r_JQWgB_v8mlvLeOG79KoD3E0LMyRgU4PkIBplS_0A/edit#gid=1932389190 We're going to be enhancing our data download capabilities in the near future and would love to know if these data points are useful. Aside from that, nothing pressing. Keen to keep moving forward. I'll shift next Monday's meeting to Tuesday if that's alright with you. Let me know if you have any feedback on the acceptance criteria in the interim. Thanks, Nathan On Mon, May 20, 2019 at 1:22 PM Eliot Rose wrote: Hello Nathan: Chris and I both have conflicts during today's check-in call. Would you like to reschedule for later in the week, or should we just skip this week since we checked in Friday (we should also reschedule next Monday's call since it's Memorial Day). I'll be more help scheduling stuff once I'm not in transit and back on wifi. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Eliot Rose nathan@sidewalklabs.com Accepted: Updated invitation with note: Metro + Replica // Weekly Sync @ Tue May 28, 2019 3pm - 3:30pm (CDT) (eliot.rose@oregonmetro.gov) From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson RE: Portland Metro Replica check-ins Tuesday, May 14, 2019 4:16:00 PM Thanks Nathan! Passing this along to Chris as well so that he can view your issue tracker (Chris, see the link below).   The 11th-13th sounds good. If you can, please hold off on booking til the end of the week; I’ve asked TriMet and Portland to let me know if there are any critical absences during the 10th-19th window we discussed.     Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, May 14, 2019 11:10 AM To: Eliot Rose Subject: Re: Portland Metro Replica check-ins Hi Eliot! 1:30pm PT recurring sounds great. Look for an invite for that date/time soon. I'll also send you a quick invite for a touch base on Wednesday. Look for that soon as well. Targeting June 11-13 for a trip. Ideally, this is an opportune time to meet with all agencies and work thru the Establishing Trust section of the requirements document, identify preferred use cases at the agency level, department level, and individual level, conduct observational research by the Replica product team (extremely valuable to ensure our product is guided by voice-of-customer), and deep dive on data collection. I'm working to develop a checklist to ensure a productive and meaningful trip. That's something we can discuss on tomorrow's call. Also, I'm using this document to track open issues. Again, something we'll use to guide our weekly status calls. https://docs.google.com/spreadsheets/d/1l-QKd_weXsRl5QdafsnQvAnszOn2aq_pdvSyoSA4CQ/edit#gid=1817381524 Thanks, Nathan On Mon, May 13, 2019 at 3:12 PM Eliot Rose wrote: Hello Nathan: Thanks for taking the time to chat with me on Friday. I talked with Chris about finding a time for us to set up a regular weekly check-in; does Mondays at 1:30 PT work for you? If that doesn’t work, Mondays and Tuesdays are generally pretty good days for us. Given that it’s already Monday; if you’d like to check in this week we can do tomorrow or Wednesday 11-1. As far as visiting here – June 10-19 is the window that works best for Chris and me; I’m going to reach out to Portland and TriMet PMs and see about their availability during that time. Please let me know if there are dates in that range that don’t work for you. Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Eliot Rose nathan@sidewalklabs.com Accepted: Invitation: Metro + Replica // Weekly Sync @ Weekly from 3:30pm to 4pm on Monday (CDT) (eliot.rose@oregonmetro.gov) From: To: Subject: Eliot Rose nathan@sidewalklabs.com Accepted: Invitation: Metro // Replica - Quick Sync @ Wed May 15, 2019 1pm - 1:30pm (CDT) (eliot.rose@oregonmetro.gov) From: To: Subject: Date: Eliot Rose Nathan Preheim Portland Metro Replica check-ins Monday, May 13, 2019 3:12:00 PM Hello Nathan:   Thanks for taking the time to chat with me on Friday. I talked with Chris about finding a time for us to set up a regular weekly check-in; does Mondays at 1:30 PT work for you? If that doesn’t work, Mondays and Tuesdays are generally pretty good days for us.   Given that it’s already Monday; if you’d like to check in this week we can do tomorrow or Wednesday 11-1.   As far as visiting here – June 10-19 is the window that works best for Chris and me; I’m going to reach out to Portland and TriMet PMs and see about their availability during that time. Please let me know if there are dates in that range that don’t work for you.     Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Eliot Rose nathan@sidewalklabs.com Accepted: Invitation: Metro // Replica - Validation Discussion @ Fri May 10, 2019 1:30pm - 2pm (CDT) (eliot.rose@oregonmetro.gov) From: To: Subject: Attachments: Eliot Rose on behalf of nathan@sidewalklabs.com Chris Johnson; Eliot Rose FW: Invitation: Metro // Replica - Validation Discussion @ Fri May 10, 2019 1:30pm - 2pm (CDT) (eliot.rose@oregonmetro.gov) invite.ics -----Original Appointment----From: nathan@sidewalklabs.com [mailto:nathan@sidewalklabs.com] Sent: Thursday, May 09, 2019 6:07 PM To: nathan@sidewalklabs.com; Eliot Rose Subject: Invitation: Metro // Replica - Validation Discussion @ Fri May 10, 2019 1:30pm - 2pm (CDT) (eliot.rose@oregonmetro.gov) When: Friday, May 10, 2019 11:30 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Google Hangout You have been invited to the following event. more details » Metro // Replica - Validation Discussion When  Fri May 10, 2019 1:30pm – 2pm Central Time - Chicago   Where  Google Hangout (map )   Joining info  meet.google.com/kod-ujqp-zbd    Or dial: +1 440-549-4584  PIN: 527853#  More phone numbers     Calendar  eliot.rose@oregonmetro.gov   Who  •  nathan@sidewalklabs.com   - organizer   •  eliot.rose@oregonmetro.gov     Several things to cover, if we're hyper efficient. :) - Updated acceptance criteria - Validation data hold-back - Data collection kickoff - Portland visit logistics - Recurring status call - Issue List within Google Drive Going (eliot.rose@oregonmetro.gov )?   Yes   - Maybe   - No      more options »   Invitation from Google Calendar You are receiving this courtesy email at the account eliot.rose@oregonmetro.gov   because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn More .    From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Requirements Meeting Recap and Updated Acceptance Criteria Thursday, May 09, 2019 9:56:00 AM How’s tomorrow between 10-12 Pacific look for you?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Thursday, May 09, 2019 8:57 AM To: Eliot Rose Subject: Re: Requirements Meeting Recap and Updated Acceptance Criteria Hi Eliot! Have time for a 15-20 minute chat about this soon? Perhaps this afternoon or tomorrow? I'll need perspective on who has what, and better perspective on coverage. That should give us insight into what you want to share and what you want to hold back. Let me know what works best. Thanks, Nathan On Tue, May 7, 2019 at 1:50 PM Eliot Rose wrote: Thanks for the follow-up, Nathan!   Chris and his team will work on getting you the bridge and major highway data you requested ASAP.   I appreciate the clarification on withholding data, and I’m looking forward to seeing your edits to the validation criteria. How do you envision documenting which of the validation data we intend to share vs. withhold? It seems like it would be useful to have that laid out since we’ll be working across 3 different agencies to do the validation.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, May 06, 2019 11:32 AM To: Eliot Rose Cc: Chris Johnson; Joe Broach; Maribeth Todd Subject: Re: Requirements Meeting Recap and Updated Acceptance Criteria Hi Eliot! In terms of clarification on data sharing, I think we're all on the same page. Yes, we are completely onboard with you all validating the final outputs. We don't use ground-truth for model creation, in fact we have already created an uncalibrated Replica for Portland. The ground-truth is used to calibrate (vs. validate), obviously meaning fit the model to the counts we see on the ground. It is typical, as you and Nick discussed, to hold back some of the ground-truth (so as it is consistent in time period and dimensionality) to validate for your own purposes. We certainly don't need everything shared, but getting 80% +/- of the available ground-truth as we start simulation runs for the purpose of fitting is pretty standard practice. Ideally, the 80% should stretch across all data sets (traffic, transit, TNCs, scooters, etc..). An example of this in practice, New York provided traffic counts for 8 or 9 major thoroughfares and held back 1-2 for the purposes of ensuring internally consistency. The counts, both provided and held back, were from the same season, collected in the same way, and baselined against Replica consistently. This is a practice we encourage for the sake of building trust in the model, particularly in places where ground-truth doesn't exist. That sound alright? In terms of that validation/ground truth data, it would be extremely helpful if you were able to provide the following inputs ASAP. There will absolutely be other ground-truth data requests, but these data sets are essential to start moving the Portland Replica model from uncalibrated to calibrated - and will really help us get a jump on that effort. Auto: Hourly counts for one (or more) weekday(s) within the date range of Fall of 2018 (September 1 thru November 30th). * Major bridges * At least one sensor on other major highways: I-5, I-205, I-84, 26, 217 - For ease of matching it’s best if they’re not at a major split/interchange Is that feasible? I've got a meeting scheduled with Alexei tomorrow and we'll be reviewing the acceptance criteria revisions before I send them over. Also, I just shared the Google Drive with everyone you suggested. Thanks, Nathan On Fri, May 3, 2019 at 5:53 PM Eliot Rose wrote: Thanks Nathan:   Looking forward to receiving your feedback on the draft acceptance criteria. The follow-ups that I noted from the meeting are below, anything that I missed? Also, can you please make sure that everyone who was at the meeting on our side has access to the Google Drive? Attendee list is below.   Before we dive into the follow-ups, we’d like to clarify your expectations for data sharing during the validation process. We intend to withhold some of the data that we use to validate Replica, both because there are privacy concerns involved (as you heard with Portland’s TNC data) and because we want to ensure that data is used solely for validation and not as an input to the model, which is a standard best practice for model validation. I had run that by Nick, and he seemed OK with it, so our contract reflects that intention, but on yesterday’s call it seemed like Alexei was operating off of the assumption that we were going to share all of the validation data with you. I understand that you need some validation data because you intend to validate Replica in parallel with us. Is your expectation that we’ll share all of the data we use for validation with you? How do we verify that you’re using data that we share with you solely for validation and not to train the model? I’m happy to set up a call to discuss.   My follow-ups from the meeting for Metro: · Begin compiling validation data so that we can add more detail to the acceptance criteria. Here are the data sets that I heard the most focus on making sure that we understand / consider adding to the process before we dive into acceptance testing: o Traffic counts o Bike/ped counts o Ridership data for other transit systems (C-Tran, Portland Streetcar) o Crosswalk of Metro employment zones x Census tracts o Bikeshare / scooter data o Demographic data on race / ethnicity · Send along the TriMet report about demographics of transit riders to see about including transit demographics in validation criteria · Schedule a follow-up with City of Portland re: TNC data (but let’s resolve the data sharing discussion above first)   Attendee list (I know that Peter and Stephanie, the last two folks listed don’t have access to the Drive folder):   Chris Johnson ; Maribeth Todd ; Kerr, Michael ; joshua.lynch@portlandoregon.gov; Martin, Kevin ; McHugh, Bibiana ; steelem@trimet.org; Lindstrom, Aubrey ; Joe Broach ; McDonald, Mike ; Gilligan, Mike ; ningsheng.zhou@portlandoregon.gov; Council Chamber Owen, Jeffrey ; Hesse, Eric ; Sherman, Jacob ; Bryan Nguyen ; Hurley, Peter Peter.T.Hurley@portlandoregon.gov; Lonsdale, Stephanie Stephanie.Lonsdale@portlandoregon.gov   Til soon, and have a great weekend! -Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Friday, May 03, 2019 11:30 AM To: Eliot Rose Subject: Requirements Meeting Recap and Updated Acceptance Criteria Hi Eliot! Thanks for coordinating details on your side to support the requirement kickoff call. Lots of useful dialog from all parties involved. We are updating the acceptance criteria based on feedback. I should have that back to you soon - early next week at the latest. The most germane next steps are acceptance criteria finalization and a fully executed contract. Sounds like both of these items will get resolved within the next week or so. Thanks again, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson; Joe Broach; Maribeth Todd RE: Requirements Meeting Recap and Updated Acceptance Criteria Tuesday, May 07, 2019 11:50:00 AM Thanks for the follow-up, Nathan!   Chris and his team will work on getting you the bridge and major highway data you requested ASAP.   I appreciate the clarification on withholding data, and I’m looking forward to seeing your edits to the validation criteria. How do you envision documenting which of the validation data we intend to share vs. withhold? It seems like it would be useful to have that laid out since we’ll be working across 3 different agencies to do the validation.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, May 06, 2019 11:32 AM To: Eliot Rose Cc: Chris Johnson; Joe Broach; Maribeth Todd Subject: Re: Requirements Meeting Recap and Updated Acceptance Criteria Hi Eliot! In terms of clarification on data sharing, I think we're all on the same page. Yes, we are completely onboard with you all validating the final outputs. We don't use ground-truth for model creation, in fact we have already created an uncalibrated Replica for Portland. The ground-truth is used to calibrate (vs. validate), obviously meaning fit the model to the counts we see on the ground. It is typical, as you and Nick discussed, to hold back some of the ground-truth (so as it is consistent in time period and dimensionality) to validate for your own purposes. We certainly don't need everything shared, but getting 80% +/- of the available ground-truth as we start simulation runs for the purpose of fitting is pretty standard practice. Ideally, the 80% should stretch across all data sets (traffic, transit, TNCs, scooters, etc..). An example of this in practice, New York provided traffic counts for 8 or 9 major thoroughfares and held back 1-2 for the purposes of ensuring internally consistency. The counts, both provided and held back, were from the same season, collected in the same way, and baselined against Replica consistently. This is a practice we encourage for the sake of building trust in the model, particularly in places where ground-truth doesn't exist. That sound alright? In terms of that validation/ground truth data, it would be extremely helpful if you were able to provide the following inputs ASAP. There will absolutely be other ground-truth data requests, but these data sets are essential to start moving the Portland Replica model from uncalibrated to calibrated - and will really help us get a jump on that effort. Auto: Hourly counts for one (or more) weekday(s) within the date range of Fall of 2018 (September 1 thru November 30th). * Major bridges * At least one sensor on other major highways: I-5, I-205, I-84, 26, 217 - For ease of matching it’s best if they’re not at a major split/interchange Is that feasible? I've got a meeting scheduled with Alexei tomorrow and we'll be reviewing the acceptance criteria revisions before I send them over. Also, I just shared the Google Drive with everyone you suggested. Thanks, Nathan On Fri, May 3, 2019 at 5:53 PM Eliot Rose wrote: Thanks Nathan:   Looking forward to receiving your feedback on the draft acceptance criteria. The follow-ups that I noted from the meeting are below, anything that I missed? Also, can you please make sure that everyone who was at the meeting on our side has access to the Google Drive? Attendee list is below.   Before we dive into the follow-ups, we’d like to clarify your expectations for data sharing during the validation process. We intend to withhold some of the data that we use to validate Replica, both because there are privacy concerns involved (as you heard with Portland’s TNC data) and because we want to ensure that data is used solely for validation and not as an input to the model, which is a standard best practice for model validation. I had run that by Nick, and he seemed OK with it, so our contract reflects that intention, but on yesterday’s call it seemed like Alexei was operating off of the assumption that we were going to share all of the validation data with you. I understand that you need some validation data because you intend to validate Replica in parallel with us. Is your expectation that we’ll share all of the data we use for validation with you? How do we verify that you’re using data that we share with you solely for validation and not to train the model? I’m happy to set up a call to discuss.   My follow-ups from the meeting for Metro: · Begin compiling validation data so that we can add more detail to the acceptance criteria. Here are the data sets that I heard the most focus on making sure that we understand / consider adding to the process before we dive into acceptance testing: o Traffic counts o Bike/ped counts o Ridership data for other transit systems (C-Tran, Portland Streetcar) o Crosswalk of Metro employment zones x Census tracts o Bikeshare / scooter data o Demographic data on race / ethnicity · Send along the TriMet report about demographics of transit riders to see about including transit demographics in validation criteria · Schedule a follow-up with City of Portland re: TNC data (but let’s resolve the data sharing discussion above first)   Attendee list (I know that Peter and Stephanie, the last two folks listed don’t have access to the Drive folder):   Chris Johnson ; Maribeth Todd ; Kerr, Michael ; joshua.lynch@portlandoregon.gov; Martin, Kevin ; McHugh, Bibiana ; steelem@trimet.org; Lindstrom, Aubrey ; Joe Broach ; McDonald, Mike ; Gilligan, Mike ; ningsheng.zhou@portlandoregon.gov; Council Chamber Owen, Jeffrey ; Hesse, Eric ; Sherman, Jacob ; Bryan Nguyen ; Hurley, Peter Peter.T.Hurley@portlandoregon.gov; Lonsdale, Stephanie Stephanie.Lonsdale@portlandoregon.gov   Til soon, and have a great weekend! -Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Friday, May 03, 2019 11:30 AM To: Eliot Rose Subject: Requirements Meeting Recap and Updated Acceptance Criteria Hi Eliot! Thanks for coordinating details on your side to support the requirement kickoff call. Lots of useful dialog from all parties involved. We are updating the acceptance criteria based on feedback. I should have that back to you soon - early next week at the latest. The most germane next steps are acceptance criteria finalization and a fully executed contract. Sounds like both of these items will get resolved within the next week or so. Thanks again, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson; Joe Broach; Maribeth Todd RE: Requirements Meeting Recap and Updated Acceptance Criteria Friday, May 03, 2019 3:53:00 PM Thanks Nathan:   Looking forward to receiving your feedback on the draft acceptance criteria. The follow-ups that I noted from the meeting are below, anything that I missed? Also, can you please make sure that everyone who was at the meeting on our side has access to the Google Drive? Attendee list is below.   Before we dive into the follow-ups, we’d like to clarify your expectations for data sharing during the validation process. We intend to withhold some of the data that we use to validate Replica, both because there are privacy concerns involved (as you heard with Portland’s TNC data) and because we want to ensure that data is used solely for validation and not as an input to the model, which is a standard best practice for model validation. I had run that by Nick, and he seemed OK with it, so our contract reflects that intention, but on yesterday’s call it seemed like Alexei was operating off of the assumption that we were going to share all of the validation data with you. I understand that you need some validation data because you intend to validate Replica in parallel with us. Is your expectation that we’ll share all of the data we use for validation with you? How do we verify that you’re using data that we share with you solely for validation and not to train the model? I’m happy to set up a call to discuss.   My follow-ups from the meeting for Metro: · Begin compiling validation data so that we can add more detail to the acceptance criteria. Here are the data sets that I heard the most focus on making sure that we understand / consider adding to the process before we dive into acceptance testing: o Traffic counts o Bike/ped counts o Ridership data for other transit systems (C-Tran, Portland Streetcar) o Crosswalk of Metro employment zones x Census tracts o Bikeshare / scooter data o Demographic data on race / ethnicity · Send along the TriMet report about demographics of transit riders to see about including transit demographics in validation criteria · Schedule a follow-up with City of Portland re: TNC data (but let’s resolve the data sharing discussion above first)   Attendee list (I know that Peter and Stephanie, the last two folks listed don’t have access to the Drive folder):   Chris Johnson ; Maribeth Todd ; Kerr, Michael ; joshua.lynch@portlandoregon.gov; Martin, Kevin ; McHugh, Bibiana ; steelem@trimet.org; Lindstrom, Aubrey ; Joe Broach ; McDonald, Mike ; Gilligan, Mike ; ningsheng.zhou@portlandoregon.gov; Council Chamber Owen, Jeffrey ; Hesse, Eric ; Sherman, Jacob ; Bryan Nguyen ; Hurley, Peter Peter.T.Hurley@portlandoregon.gov; Lonsdale, Stephanie Stephanie.Lonsdale@portlandoregon.gov   Til soon, and have a great weekend! -Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Friday, May 03, 2019 11:30 AM To: Eliot Rose Subject: Requirements Meeting Recap and Updated Acceptance Criteria Hi Eliot! Thanks for coordinating details on your side to support the requirement kickoff call. Lots of useful dialog from all parties involved. We are updating the acceptance criteria based on feedback. I should have that back to you soon - early next week at the latest. The most germane next steps are acceptance criteria finalization and a fully executed contract. Sounds like both of these items will get resolved within the next week or so. Thanks again, Nathan -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Portland Metro Replica kickoff Tuesday, April 30, 2019 3:11:00 PM Great, talk soon!   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, April 30, 2019 2:08 PM To: Eliot Rose Subject: Re: Portland Metro Replica kickoff Eliot! Perfect. Just sent you a calendar invite for 10:30am pacific. Talk to you tomorrow! -Nathan On Tue, Apr 30, 2019 at 4:05 PM Eliot Rose wrote: I can do tomorrow any time between 10:30 and 12:30 Pacific. What works best for you?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, April 30, 2019 11:55 AM To: Eliot Rose Subject: Re: Portland Metro Replica kickoff Eliot! Do you have 10-15 minutes to chat today or tomorrow? I can give you a deeper preview of what to expect on Thursday. On Tue, Apr 30, 2019 at 1:07 PM Eliot Rose wrote: PS – is there anything that you need me to prep for this meeting, or parts of the agenda that you would like my help with? I got the impression from what you sent that you’re planning to lead it, but please let me know if there’s anything that I can prepare to help support you.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, April 30, 2019 7:56 AM To: Nathan Preheim Cc: Chris Johnson Subject: Re: Portland Metro Replica kickoff Here you go! Looking forward to the discussion. -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Portland Metro Replica kickoff Tuesday, April 30, 2019 2:04:00 PM I can do tomorrow any time between 10:30 and 12:30 Pacific. What works best for you?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, April 30, 2019 11:55 AM To: Eliot Rose Subject: Re: Portland Metro Replica kickoff Eliot! Do you have 10-15 minutes to chat today or tomorrow? I can give you a deeper preview of what to expect on Thursday. On Tue, Apr 30, 2019 at 1:07 PM Eliot Rose wrote: PS – is there anything that you need me to prep for this meeting, or parts of the agenda that you would like my help with? I got the impression from what you sent that you’re planning to lead it, but please let me know if there’s anything that I can prepare to help support you.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, April 30, 2019 7:56 AM To: Nathan Preheim Cc: Chris Johnson Subject: Re: Portland Metro Replica kickoff Here you go! Looking forward to the discussion. -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Portland Metro Replica kickoff Tuesday, April 30, 2019 11:06:00 AM PS – is there anything that you need me to prep for this meeting, or parts of the agenda that you would like my help with? I got the impression from what you sent that you’re planning to lead it, but please let me know if there’s anything that I can prepare to help support you.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, April 30, 2019 7:56 AM To: Nathan Preheim Cc: Chris Johnson Subject: Re: Portland Metro Replica kickoff Here you go! Looking forward to the discussion. From: To: Cc: Subject: Date: Attachments: Eliot Rose Nathan Preheim Chris Johnson Re: Portland Metro Replica kickoff Tuesday, April 30, 2019 7:56:14 AM DRAFT Replica Acceptance Criteria v3.xlsx ATT00001.htm Here you go! Looking forward to the discussion.  From: To: Cc: Bcc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson 5days@followupthen.com RE: Portland Metro Replica kickoff Wednesday, April 24, 2019 4:21:00 PM Great; we’re making sure we get the latest edits from our partners; I’ll have something to share with you early next week.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, April 24, 2019 4:01 PM To: Eliot Rose Cc: Chris Johnson Subject: Re: Portland Metro Replica kickoff Eliot! Yes please. I’ll update the documents with your suggestions. Also, we’ll have members from our engineering team on the call to review and assess your preferences. Thanks much, Nathan On Wed, Apr 24, 2019 at 3:58 PM Eliot Rose wrote: Thanks Nathan! I added the meeting info and the materials to the invite. We’ve already met a couple of times to start drafting acceptance criteria; would it be helpful if we shared the draft with you before we meet?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, April 24, 2019 9:55 AM To: Eliot Rose Subject: Re: Portland Metro Replica kickoff Hello Eliot! Ok, some logistical items in preparation of next weeks meeting. 1. I just created a Google Hangout for this event. Feel free to update the agenda and include these details... I'll have my camera on so you'll be able to see me, and also follow along with the materials. Assuming you'll have internet access in the conference room. Meeting ID meet.google.com/mxy-gwxs-dep Phone Numbers ‎(‪US)‪+1 507-607-0110 Event will include a link to view full list of phone numbers for all supported countries. PIN: ‪658 102#   2. I've attached two documents - the Requirements Document and the Presentation. Please start to review the Requirements Document   We will spend the most amount of time working through the Data Quality sections. This is where we define acceptance criteria for your Replica model.   Sound good?   Thanks! -Nathan   On Tue, Apr 23, 2019 at 3:44 PM Eliot Rose wrote: Hello Nathan:   I got us the best meeting room I could for the kickoff; it’s got a good projector and teleconferencing system but no built-in videoconferencing. It sounded like you were planning to share a presentation; did you want to set up an online meeting for that, or should I do it on our end? And is videoconferencing essential? With the current setup we’d be able to see you if you’re sharing your camera, but with a little workaround we should be able to set up a camera that you can use to see us – I just can’t guarantee you’ll have a good view of everyone.   Looking forward to it! Also, please lte me know when you’ve added everyone to the Google Drive folder and I’ll add the link to that to the meeting invite.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Tuesday, April 23, 2019 3:37 PM To: Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse; ningsheng.zhou@portlandoregon.gov Subject: Portland Metro Replica kickoff When: Thursday, May 02, 2019 10:30 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro Council Chamber / call-in (see invite)     We’re looking forward to hosting the Portland / Replica kickoff next week. We were able to find a meeting room for the time that works best for everyone. We’ll have a call-in option, but please plan on attending in person if you can. I’ll be updating this invitation with call-in information and a link to the online meeting closer to the date.   Metro is at 600 NE Grand. Sign in at the front desk, then walk up the stairs. Turn around and walk toward the north side of the building when you get to the top of the stairs and the Council Chamber will be on your left.   Call-in information to come…   -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson RE: Portland Metro Replica kickoff Wednesday, April 24, 2019 3:58:00 PM Thanks Nathan! I added the meeting info and the materials to the invite. We’ve already met a couple of times to start drafting acceptance criteria; would it be helpful if we shared the draft with you before we meet?   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Wednesday, April 24, 2019 9:55 AM To: Eliot Rose Subject: Re: Portland Metro Replica kickoff Hello Eliot! Ok, some logistical items in preparation of next weeks meeting. 1. I just created a Google Hangout for this event. Feel free to update the agenda and include these details... I'll have my camera on so you'll be able to see me, and also follow along with the materials. Assuming you'll have internet access in the conference room. Meeting ID meet.google.com/mxy-gwxs-dep Phone Numbers ‎(‪US)‪+1 507-607-0110 Event will include a link to view full list of phone numbers for all supported countries. PIN: ‪658 102#   2. I've attached two documents - the Requirements Document and the Presentation. Please start to review the Requirements Document   We will spend the most amount of time working through the Data Quality sections. This is where we define acceptance criteria for your Replica model.   Sound good?   Thanks! -Nathan   On Tue, Apr 23, 2019 at 3:44 PM Eliot Rose wrote: Hello Nathan:   I got us the best meeting room I could for the kickoff; it’s got a good projector and teleconferencing system but no built-in videoconferencing. It sounded like you were planning to share a presentation; did you want to set up an online meeting for that, or should I do it on our end? And is videoconferencing essential? With the current setup we’d be able to see you if you’re sharing your camera, but with a little workaround we should be able to set up a camera that you can use to see us – I just can’t guarantee you’ll have a good view of everyone.   Looking forward to it! Also, please lte me know when you’ve added everyone to the Google Drive folder and I’ll add the link to that to the meeting invite.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Tuesday, April 23, 2019 3:37 PM To: Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse; ningsheng.zhou@portlandoregon.gov Subject: Portland Metro Replica kickoff When: Thursday, May 02, 2019 10:30 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro Council Chamber / call-in (see invite)     We’re looking forward to hosting the Portland / Replica kickoff next week. We were able to find a meeting room for the time that works best for everyone. We’ll have a call-in option, but please plan on attending in person if you can. I’ll be updating this invitation with call-in information and a link to the online meeting closer to the date.   Metro is at 600 NE Grand. Sign in at the front desk, then walk up the stairs. Turn around and walk toward the north side of the building when you get to the top of the stairs and the Council Chamber will be on your left.   Call-in information to come…   -- Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Subject: Date: Eliot Rose Nathan Preheim RE: Portland Metro Replica kickoff Tuesday, April 23, 2019 3:44:00 PM Hello Nathan: I got us the best meeting room I could for the kickoff; it’s got a good projector and teleconferencing system but no built-in videoconferencing. It sounded like you were planning to share a presentation; did you want to set up an online meeting for that, or should I do it on our end? And is videoconferencing essential? With the current setup we’d be able to see you if you’re sharing your camera, but with a little workaround we should be able to set up a camera that you can use to see us – I just can’t guarantee you’ll have a good view of everyone. Looking forward to it! Also, please lte me know when you’ve added everyone to the Google Drive folder and I’ll add the link to that to the meeting invite. Thanks, Eliot Eliot Rose Technology Strategist Metro -----Original Appointment----From: Eliot Rose Sent: Tuesday, April 23, 2019 3:37 PM To: Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse; ningsheng.zhou@portlandoregon.gov Subject: Portland Metro Replica kickoff When: Thursday, May 02, 2019 10:30 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro Council Chamber / call-in (see invite) We’re looking forward to hosting the Portland / Replica kickoff next week. We were able to find a meeting room for the time that works best for everyone. We’ll have a call-in option, but please plan on attending in person if you can. I’ll be updating this invitation with call-in information and a link to the online meeting closer to the date. Metro is at 600 NE Grand. Sign in at the front desk, then walk up the stairs. Turn around and walk toward the north side of the building when you get to the top of the stairs and the Council Chamber will be on your left. Call-in information to come… From: To: Subject: Importance: Eliot Rose Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse; ningsheng.zhou@portlandoregon.gov Canceled: Portland Metro Replica Kickoff High Hello all: Please hold this time for the Replica kickoff. I had to go with the 2nd best time because of a lack of available conference rooms, and I apologize for those that aren’t able to make it. I’m still scrounging for another room in our primary slot, and I’ll send a placeholder for that alternate time in case we find a venue. Hope to finalize this soon; thanks for your patience! -Eliot From: To: Subject: Date: Eliot Rose nathan@sidewalklabs.com RE: Accepted: [ALTERNATE] Portland Metro Replica kickoff @ Thu May 2, 2019 12:30pm - 2pm (CDT) (Eliot Rose) Friday, April 19, 2019 3:06:00 PM Hello Nathan: If you respond to the invite you’ll see all the folks I sent it to – those are the ones you should share your Google Drive folder with. Let me know if you need me to send you a list. Thanks, and looking forward to the kickoff! Eliot Rose Technology Strategist Metro -----Original Appointment----From: Google Calendar [mailto:calendar-notification@google.com] On Behalf Of nathan@sidewalklabs.com Sent: Thursday, April 18, 2019 6:18 PM To: Eliot Rose Subject: Accepted: [ALTERNATE] Portland Metro Replica kickoff @ Thu May 2, 2019 12:30pm 2pm (CDT) (Eliot Rose) When: Thursday, May 02, 2019 10:30 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: TBD nathan@sidewalklabs.com has accepted this invitation. [ALTERNATE] Portland Metro Replica kickoff When         Thu May 2, 2019 12:30pm – 2pm Central Time - Chicago         Where         TBD (map)         Calendar         Eliot Rose         Who         •         Eliot Rose - organizer         •         nathan@sidewalklabs.com - creator         •         Lindstrom, Aubrey         •         joshua.lynch@portlandoregon.gov         •         Pamela Blackhorse         •         steelem@trimet.org         •         Chris Johnson         •         Gilligan, Mike         •         Kerr, Michael         •         McHugh, Bibiana         •         McDonald, Mike         •         Joe Broach         •         Martin, Kevin         •         Maribeth Todd                 Hello all: This is an alternate date for the Replica kickoff; it was the time that worked best for all but we’re still seeking a meeting room. I’ll cancel this or the other invite and get a final time/location by the end of the week. -Eliot         Invitation from Google Calendar You are receiving this courtesy email at the account eliot.rose@oregonmetro.gov because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others         regardless of their own invitation status, or to modify your RSVP. Learn More.  << File: invite.ics >> From: To: Subject: Eliot Rose Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse [ALTERNATE] Portland Metro Replica kickoff Hello all: This is an alternate date for the Replica kickoff; it was the time that worked best for all but we’re still seeking a meeting room. I’ll cancel this or the other invite and get a final time/location by the end of the week. -Eliot From: To: Subject: Eliot Rose Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse Portland Metro Replica Kickoff Hello all: Please hold this time for the Replica kickoff. I had to go with the 2nd best time because of a lack of available conference rooms, and I apologize for those that aren’t able to make it. I’m still scrounging for another room in our primary slot, and I’ll send a placeholder for that alternate time in case we find a venue. Hope to finalize this soon; thanks for your patience! -Eliot From: To: Subject: Date: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike; Christine.Kendrick@portlandoregon.gov; Joe Broach; Nathan Preheim; Nick Bowden Portland Metro Replica kickoff meeting Tuesday, April 16, 2019 9:29:00 AM Hello all:   Please fill out this poll to help us schedule the Replica kickoff meeting – preferably by tomorrow COB if possible. We’ll have people from Metro, City of Portland, TriMet and Sidewalk Labs on the call.   I know that some of you are still trying to loop in a couple of other folks from your organization; feel free to forward the scheduling link to them. Sidewalk Labs has some materials to share, and we’ll make sure that everyone on the meeting invite gets access to those.   Here’s the agenda, from Sidewalk Labs:     Phases of Deployment (5 minutes) Requirements Kickoff (45 minutes)      Data Accuracy + Acceptance Criteria      Data Trust      Product Utility Data Collection (15 minutes) Next Steps (5 minutes)   Most of the conversation centers around capturing as much requirements information about your community as possible. As you know we need a host of inputs so that we can build up a representative model. The acceptance criteria (which we call Data Accuracy) is a big component to that, but there are other dimensions that we also explore on the call.    The call is a kickoff, and it lays the foundation for all of the work and effort required to start the deployment process.   -Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 15, 2019 11:40 AM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello everyone:   Just a reminder to please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the Replica kickoff ASAP. I plan to send out a scheduling poll at the end of the day.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, April 11, 2019 4:24 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Chris Johnson; Nick Bowden RE: [External email]Re: Revised Portland Metro Replica agreement Tuesday, April 16, 2019 9:15:00 AM Got it. Rather than trying to find a time on our end and then bouncing windows back to you, I’ll just include you and Nick on the poll that I’m about to send and you can forward to other members of your team as needed.   Also, what AV stuff do we need on our end? Most of our meeting rooms are equipped with a conference line and projector, but not as many of them have videoconferencing.   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, April 16, 2019 9:12 AM To: Eliot Rose Cc: Chris Johnson; Nick Bowden Subject: Re: [External email]Re: Revised Portland Metro Replica agreement Eliot! Let’s plan 90. Thanks! On Tue, Apr 16, 2019 at 10:51 AM Eliot Rose wrote: Thanks! I’ll send a poll to the group with some times for both the week of 4/22 and 4/29 and let you know a couple of windows that work. Are you looking for an hour or 90 mins.? The agenda you sent me had 70 minutes worth of material.   Phases of Deployment (5 minutes) Requirements Kickoff (45 minutes) Data Accuracy + Acceptance Criteria Data Trust Product Utility Data Collection (15 minutes) Next Steps (5 minutes)   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, April 16, 2019 7:03 AM To: Eliot Rose Cc: Nick Bowden; Chris Johnson Subject: Re: [External email]Re: Revised Portland Metro Replica agreement Hi Eliot! It's not something we typically do but yes, we can kickoff prior to signed contract. Only final details I need are a confirmation of participants and a couple day/times that work for the group. I'll then follow up with an invitation and kickoff resources. Thanks much! -Nathan On Thu, Apr 11, 2019 at 6:30 PM Eliot Rose wrote: Hello Nathan and Nick:   We had a call with the project team today; I’ll be getting you a complete list of names and emails for the kickoff by the end of the week and some scheduling options midweek next week. A couple scheduling questions: we’re working with our contracting folks to get this ready for sig ASAP (and I appreciate your quick responses to move that along), but there’s a chance it won’t be signed by the week of 4/22 when we hold the kickoff. Are you comfortable holding the kickoff before we get a final contract signed? Also, we’ve been working to draft our acceptance criteria. Would it be helpful to share those in advance of the kickoff, or should we hold off until we get a chance to discuss the process during the kickoff?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, April 08, 2019 2:15 PM To: Eliot Rose Cc: Nick Bowden; Chris Johnson Subject: [External email]Re: Revised Portland Metro Replica agreement Hi Eliot! The week of 4/22 looks good for a kickoff call. I too think we should start smaller. Let's focus on the 3 core contracting agencies - it's up to you all to define requirements. There will be plenty of opportunities to engage the other players as we get into the data collection process. I have a host of online materials to share when I send out the calendar invite. Who specifically, name and email addresses, should I include? Looking forward to working with you! Thanks, Nathan On Wed, Apr 3, 2019 at 5:40 PM Eliot Rose wrote: Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April —as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Nick Bowden; Chris Johnson RE: [External email]Re: Revised Portland Metro Replica agreement Tuesday, April 16, 2019 8:51:00 AM Thanks! I’ll send a poll to the group with some times for both the week of 4/22 and 4/29 and let you know a couple of windows that work. Are you looking for an hour or 90 mins.? The agenda you sent me had 70 minutes worth of material.   Phases of Deployment (5 minutes) Requirements Kickoff (45 minutes)      Data Accuracy + Acceptance Criteria      Data Trust      Product Utility Data Collection (15 minutes) Next Steps (5 minutes)   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Tuesday, April 16, 2019 7:03 AM To: Eliot Rose Cc: Nick Bowden; Chris Johnson Subject: Re: [External email]Re: Revised Portland Metro Replica agreement   Hi Eliot!   It's not something we typically do but yes, we can kickoff prior to signed contract.     Only final details I need are a confirmation of participants and a couple day/times that work for the group. I'll then follow up with an invitation and kickoff resources.   Thanks much! -Nathan     On Thu, Apr 11, 2019 at 6:30 PM Eliot Rose wrote: Hello Nathan and Nick:   We had a call with the project team today; I’ll be getting you a complete list of names and emails for the kickoff by the end of the week and some scheduling options midweek next week. A couple scheduling questions: we’re working with our contracting folks to get this ready for sig ASAP (and I appreciate your quick responses to move that along), but there’s a chance it won’t be signed by the week of 4/22 when we hold the kickoff. Are you comfortable holding the kickoff before we get a final contract signed? Also, we’ve been working to draft our acceptance criteria. Would it be helpful to share those in advance of the kickoff, or should we hold off until we get a chance to discuss the process during the kickoff?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, April 08, 2019 2:15 PM To: Eliot Rose Cc: Nick Bowden; Chris Johnson Subject: [External email]Re: Revised Portland Metro Replica agreement   Hi Eliot!   The week of 4/22 looks good for a kickoff call. I too think we should start smaller. Let's focus on the 3 core contracting agencies - it's up to you all to define requirements. There will be plenty of opportunities to engage the other players as we get into the data collection process.   I have a host of online materials to share when I send out the calendar invite. Who specifically, name and email addresses, should I include?   Looking forward to working with you!   Thanks, Nathan     On Wed, Apr 3, 2019 at 5:40 PM Eliot Rose wrote: Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 102:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement   Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week.   Thanks. Nick     On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline.  Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign.   We are done! Thanks! Nick   On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   Yes, that addition is okay.    Awesome on legal name.   Thanks. Nick     On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.      Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation.  We added a cap on the indemnification. Hopefully this gets us to the finish line.    Thanks. Nick       On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement   Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. -          Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? -          On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   HI Eliot   Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so.   Thanks. Nick     On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5.   On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.    Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call.   On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by mid-week at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place     -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com   -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com   replica.sidewalkabs.com From: To: Cc: Subject: Date: Eliot Rose Nathan Preheim Nick Bowden; Chris Johnson RE: [External email]Re: Revised Portland Metro Replica agreement Thursday, April 11, 2019 4:30:00 PM Hello Nathan and Nick:   We had a call with the project team today; I’ll be getting you a complete list of names and emails for the kickoff by the end of the week and some scheduling options midweek next week. A couple scheduling questions: we’re working with our contracting folks to get this ready for sig ASAP (and I appreciate your quick responses to move that along), but there’s a chance it won’t be signed by the week of 4/22 when we hold the kickoff. Are you comfortable holding the kickoff before we get a final contract signed? Also, we’ve been working to draft our acceptance criteria. Would it be helpful to share those in advance of the kickoff, or should we hold off until we get a chance to discuss the process during the kickoff?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, April 08, 2019 2:15 PM To: Eliot Rose Cc: Nick Bowden; Chris Johnson Subject: [External email]Re: Revised Portland Metro Replica agreement Hi Eliot! The week of 4/22 looks good for a kickoff call. I too think we should start smaller. Let's focus on the 3 core contracting agencies - it's up to you all to define requirements. There will be plenty of opportunities to engage the other players as we get into the data collection process. I have a host of online materials to share when I send out the calendar invite. Who specifically, name and email addresses, should I include? Looking forward to working with you! Thanks, Nathan On Wed, Apr 3, 2019 at 5:40 PM Eliot Rose wrote: Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Michelle Bellia; Chris Johnson RE: Revised Portland Metro Replica agreement Wednesday, April 10, 2019 10:02:00 AM Thanks Nick! That makes sense, and we checked in with our contracting folks and it sounds like it makes sense on their end.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, April 10, 2019 7:28 AM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement We, and the agreement, generally treat the signature date as the start of the contract. The license term and payments are tied to acceptance date. So, contract starts at signature, license term (12 months) starts at acceptance. Everywhere we have worked, we both sign contract now and then we have used email or a one paragraph letter from you to indicate acceptance has been met. Does that make sense? Nick On Tue, Apr 9, 2019 at 6:12 PM Eliot Rose wrote: Next question:   Our contracts folks are used to working on contracts with specific beginning and end dates, and were asking about putting dates on this agreement (e.g., 4/15/19-12/31/20). But in rereading 11.1, it says that the contract starts when we accept the data. Should we put a start date on the contract, or add that at acceptance?   This also raised a question about signatures – are you expecting us to sign the contract now, or once we accept the data?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, April 09, 2019 9:03 AM To: Eliot Rose Cc: Nathan Preheim; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement No need to keep those sections, sorry missed that in the first round of emails. Once I get confirmation, we will route for signature. Thanks. Nick On Mon, Apr 8, 2019 at 3:45 PM Eliot Rose wrote: Thanks Nick:   30 days sounds good. I added that into the attached draft. Please let me check in with our contracts folks to make sure that they’ve completed their review before you route for sig. I’ll keep you posted when I hear back from them. Also, still waiting for a response from you on whether you want to leave in the sections marked [reserved] (2 and 12.1).   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, April 08, 2019 9:20 AM To: Eliot Rose; Nathan Preheim Cc: Lisa Hefty; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We typically have used 30 days as a reasonable amount of time for you all to determine if acceptance criteria has been met. It usually is much shorter than that, but it provides enough time for you all to review. If that sounds okay, we can add on our end and then route for signature. I will let Nathan respond on the particulars for the kick-off meeting. Let me know if contract is good to go per above and we can get signed on our end. Thanks. Nick On Fri, Apr 5, 2019 at 6:07 PM Eliot Rose wrote: Hello Nick:   Formatted contract attached. Other than changing the formatting we added a missing “shall” to 13.1. Only add’l remaining question (other than the ones below) is whether you want to leave in the sections marked [reserved] (2 and 12.1).   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, April 03, 2019 3:41 PM To: 'Nick Bowden' Cc: 'Nathan Preheim'; Chris Johnson Subject: RE: Revised Portland Metro Replica agreement Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April —as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Michelle Bellia; Chris Johnson RE: Revised Portland Metro Replica agreement Tuesday, April 09, 2019 4:12:00 PM Next question:   Our contracts folks are used to working on contracts with specific beginning and end dates, and were asking about putting dates on this agreement (e.g., 4/15/19-12/31/20). But in rereading 11.1, it says that the contract starts when we accept the data. Should we put a start date on the contract, or add that at acceptance?   This also raised a question about signatures – are you expecting us to sign the contract now, or once we accept the data?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, April 09, 2019 9:03 AM To: Eliot Rose Cc: Nathan Preheim; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement No need to keep those sections, sorry missed that in the first round of emails. Once I get confirmation, we will route for signature. Thanks. Nick On Mon, Apr 8, 2019 at 3:45 PM Eliot Rose wrote: Thanks Nick:   30 days sounds good. I added that into the attached draft. Please let me check in with our contracts folks to make sure that they’ve completed their review before you route for sig. I’ll keep you posted when I hear back from them. Also, still waiting for a response from you on whether you want to leave in the sections marked [reserved] (2 and 12.1).   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, April 08, 2019 9:20 AM To: Eliot Rose; Nathan Preheim Cc: Lisa Hefty; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We typically have used 30 days as a reasonable amount of time for you all to determine if acceptance criteria has been met. It usually is much shorter than that, but it provides enough time for you all to review. If that sounds okay, we can add on our end and then route for signature. I will let Nathan respond on the particulars for the kick-off meeting. Let me know if contract is good to go per above and we can get signed on our end. Thanks. Nick On Fri, Apr 5, 2019 at 6:07 PM Eliot Rose wrote: Hello Nick:   Formatted contract attached. Other than changing the formatting we added a missing “shall” to 13.1. Only add’l remaining question (other than the ones below) is whether you want to leave in the sections marked [reserved] (2 and 12.1).   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, April 03, 2019 3:41 PM To: 'Nick Bowden' Cc: 'Nathan Preheim'; Chris Johnson Subject: RE: Revised Portland Metro Replica agreement Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 102:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Nathan Preheim; Michelle Bellia; Chris Johnson RE: Revised Portland Metro Replica agreement Tuesday, April 09, 2019 9:46:00 AM Great, thanks! I corrected that in the draft that I just sent on to our contracting folks. I’ll let you know ASAP when I hear from them.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, April 09, 2019 9:03 AM To: Eliot Rose Cc: Nathan Preheim; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement No need to keep those sections, sorry missed that in the first round of emails. Once I get confirmation, we will route for signature. Thanks. Nick On Mon, Apr 8, 2019 at 3:45 PM Eliot Rose wrote: Thanks Nick:   30 days sounds good. I added that into the attached draft. Please let me check in with our contracts folks to make sure that they’ve completed their review before you route for sig. I’ll keep you posted when I hear back from them. Also, still waiting for a response from you on whether you want to leave in the sections marked [reserved] (2 and 12.1).   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, April 08, 2019 9:20 AM To: Eliot Rose; Nathan Preheim Cc: Lisa Hefty; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We typically have used 30 days as a reasonable amount of time for you all to determine if acceptance criteria has been met. It usually is much shorter than that, but it provides enough time for you all to review. If that sounds okay, we can add on our end and then route for signature. I will let Nathan respond on the particulars for the kick-off meeting. Let me know if contract is good to go per above and we can get signed on our end. Thanks. Nick On Fri, Apr 5, 2019 at 6:07 PM Eliot Rose wrote: Hello Nick:   Formatted contract attached. Other than changing the formatting we added a missing “shall” to 13.1. Only add’l remaining question (other than the ones below) is whether you want to leave in the sections marked [reserved] (2 and 12.1).   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, April 03, 2019 3:41 PM To: 'Nick Bowden' Cc: 'Nathan Preheim'; Chris Johnson Subject: RE: Revised Portland Metro Replica agreement Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 102:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot - Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Bcc: Subject: Date: Eliot Rose Nathan Preheim Nick Bowden; Chris Johnson 2days@followupthen.com RE: [External email]Re: Revised Portland Metro Replica agreement Monday, April 08, 2019 6:49:00 PM Thanks Nathan! Keeping it small sounds good. Invite list as I have it for now is below, but I’ll get you a final list and some times after the validation team meets on Thursday early afternoon.   'Kerr, Michael' ; Chris Johnson ; Robert Kirkman ; 'Martin, Kevin' ; 'Kerr, Michael' ; 'McHugh, Tim' ; 'Gilligan, Mike' , eliot.rose@oregonmetro.gov   Eliot Rose Technology Strategist Metro   From: Nathan Preheim [mailto:nathan@sidewalklabs.com] Sent: Monday, April 08, 2019 2:15 PM To: Eliot Rose Cc: Nick Bowden; Chris Johnson Subject: [External email]Re: Revised Portland Metro Replica agreement Hi Eliot! The week of 4/22 looks good for a kickoff call. I too think we should start smaller. Let's focus on the 3 core contracting agencies - it's up to you all to define requirements. There will be plenty of opportunities to engage the other players as we get into the data collection process. I have a host of online materials to share when I send out the calendar invite. Who specifically, name and email addresses, should I include? Looking forward to working with you! Thanks, Nathan On Wed, Apr 3, 2019 at 5:40 PM Eliot Rose wrote: Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place -Head of Customer Operations, Replica 575 Market Street, 4th Floor San Francisco, CA 94105 c 402-595-0055 e nathan@sidewalklabs.com replica.sidewalkabs.com From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden; Nathan Preheim Michelle Bellia; Chris Johnson RE: Revised Portland Metro Replica agreement Monday, April 08, 2019 1:45:00 PM SaaS Portland Metro final draft 20190408.docx Thanks Nick:   30 days sounds good. I added that into the attached draft. Please let me check in with our contracts folks to make sure that they’ve completed their review before you route for sig. I’ll keep you posted when I hear back from them. Also, still waiting for a response from you on whether you want to leave in the sections marked [reserved] (2 and 12.1).   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, April 08, 2019 9:20 AM To: Eliot Rose; Nathan Preheim Cc: Lisa Hefty; Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We typically have used 30 days as a reasonable amount of time for you all to determine if acceptance criteria has been met. It usually is much shorter than that, but it provides enough time for you all to review. If that sounds okay, we can add on our end and then route for signature. I will let Nathan respond on the particulars for the kick-off meeting. Let me know if contract is good to go per above and we can get signed on our end. Thanks. Nick On Fri, Apr 5, 2019 at 6:07 PM Eliot Rose wrote: Hello Nick:   Formatted contract attached. Other than changing the formatting we added a missing “shall” to 13.1. Only add’l remaining question (other than the ones below) is whether you want to leave in the sections marked [reserved] (2 and 12.1).   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, April 03, 2019 3:41 PM To: 'Nick Bowden' Cc: 'Nathan Preheim'; Chris Johnson Subject: RE: Revised Portland Metro Replica agreement Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Lisa Hefty; Michelle Bellia; Chris Johnson RE: Revised Portland Metro Replica agreement Friday, April 05, 2019 4:07:00 PM SaaS Portland Metro final draft 20190405.docx Hello Nick:   Formatted contract attached. Other than changing the formatting we added a missing “shall” to 13.1. Only add’l remaining question (other than the ones below) is whether you want to leave in the sections marked [reserved] (2 and 12.1).   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, April 03, 2019 3:41 PM To: 'Nick Bowden' Cc: 'Nathan Preheim'; Chris Johnson Subject: RE: Revised Portland Metro Replica agreement Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Nathan Preheim; Chris Johnson RE: Revised Portland Metro Replica agreement Wednesday, April 03, 2019 3:40:00 PM SaaSPortland Metro 3 22 2019 Metro edit.docx Hello Nick and Nathan:   Thanks for your patience. Can we schedule a kickoff for the week of 4/22? We’d like to have the signed contract in place before kickoff. Monday after 11, Tuesday all day, and Wednesday 10-2:30 work for us. If you let us know a couple of times within those windows that work for you we’ll circulate to the broader group on our end to find the one that works the best.   Can you tell us a little more about who you think should be at this meeting on our end? We have three agencies that are funding this pilot and doing the validation, and a much broader group of around 30 partners who are interested in accessing this tool. Most of that broader group are not in a position to provide any data to help build Replica, but a few might have some useful ground truth data to contribute. We’re inclined to keep the kickoff limited to the three core partners because it’ll make for a more efficient conversation, but we could invite a broader group of you think that’ll produce a better outcome.   I also want to make sure that we’re smoothing the path to contract signature. Attached is the latest draft, which we consider to be substantively final. In order to get it to a signable state, we’re planning to clean up the formatting on our end before passing a draft back to you. We’d also like to fill in the one remaining blank that we see in Exhibit A-1 – what number should go there? “Metro will have a period of __ days from the later of the delivery of the Services or the addition of acceptance criteria to this Agreement (“Acceptance Period” and the date of acceptance, the “Acceptance Date”), to confirm that the Services meet the foregoing acceptance criteria.” Once we format and fill in that blank, we’ll pass a version along to you for signature before we countersign. Does that sound good?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 01, 2019 1:33 PM To: 'Nick Bowden' Cc: Nathan Preheim Subject: RE: Revised Portland Metro Replica agreement Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Nathan Preheim RE: Revised Portland Metro Replica agreement Monday, April 01, 2019 1:32:00 PM Thanks Nick:   As predicted, we’re still moving contract signatures along and it’s always an interesting process. I should have an update on timing for you tomorrow, at which point I’ll also circle back w/ Nathan re: scheduling the kickoff. Thanks for your patience.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 31, 2019 2:52 PM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Circling back on this. I am traveling this week out of country, but will have access to email. If all is good, we can facilitate signature through email and set kick off call for next week. Thanks. Nick On Tue, Mar 26, 2019 at 12:30 PM Eliot Rose wrote: Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline. Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden Re: Revised Portland Metro Replica agreement Tuesday, March 26, 2019 10:30:20 AM Thanks Nick! I'll be back in the office tomorrow and Thursday and give you an update on timeline.  Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Mar 25, 2019, at 8:48 AM, Nick Bowden wrote: This is good to go. If you want to accept and process for signature we can then sign. We are done! Thanks! Nick On Fri, Mar 22, 2019 at 6:52 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement   Yes, that addition is okay.    Awesome on legal name.   Thanks. Nick     On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.      Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation.  We added a cap on the indemnification. Hopefully this gets us to the finish line.    Thanks. Nick       On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement   Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. -          Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? -          On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   HI Eliot   Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so.   Thanks. Nick     On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement   Hi Eliot   We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5.   On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards.    Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call.   On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Hi There   Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side.   Thanks. Nick       On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement   Thanks, Eliot. I should have something back by mid-week at the latest.   Nick     On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Michelle Bellia; Chris Johnson RE: Revised Portland Metro Replica agreement Friday, March 22, 2019 4:52:00 PM SaaSPortland Metro 3 22 2019 Metro edit.docx Hello Nick:   Attached are our revisions to the contract with changes tracked. Other than the addition discussed below, we made a few clarifying changes to 4.5 and 10.2, but no substantive ones. Hopefully this version is good to go, but I’ll wait for you to confirm that before we take it forward for signature. I believe this needs some additional formatting too just to clean things up; happy to let that happen on your end if you prefer.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot - Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Revised Portland Metro Replica agreement Wednesday, March 20, 2019 5:16:00 PM Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 12:01 PM To: Eliot Rose Cc: Michelle Bellia; Chris Johnson Subject: Re: Revised Portland Metro Replica agreement Yes, that addition is okay. Awesome on legal name. Thanks. Nick On Wed, Mar 20, 2019 at 1:56 PM Eliot Rose wrote: Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Michelle Bellia; Chris Johnson RE: Revised Portland Metro Replica agreement Wednesday, March 20, 2019 11:56:00 AM Thanks Nick! I’ll run the changes by our attorneys.   One thing that jumped out at me in 4.5(k) – I thought that we had agreed to make an exception that allows Metro to attempt to ID people within the data during the validation phase so that we can verify that the data adequately protects people’s privacy. Are you OK with the italicized addition below?   (k) reverse engineer, or obtain any other data, services or other technology permitting use of the Services or Content (and actually use such data, services or other technology to) make any identification of persons, personalize data or otherwise attempt to identify or personalize data subjects or obtain personally identifiable data or otherwise attempt to invade the privacy of another, except for the purposes of ensuring that Services and Content adequately safeguard residents’ privacy during validation testing.    Also, Metro’s legal name is Metro. Catchy, huh?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 20, 2019 9:47 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi Eliot Here is an updated version. A couple of items: We added some additional language to 4.5. One thing worth pointing out, 4.4 explicitly covers our responsibilities as it relates to de / re identification, whereas 4.5 is related to Metro's obligation. We added a cap on the indemnification. Hopefully this gets us to the finish line. Thanks. Nick On Fri, Mar 15, 2019 at 2:31 PM Eliot Rose wrote: Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Revised Portland Metro Replica agreement Friday, March 15, 2019 12:31:00 PM Hello Nick:   I think we’re on track to get the contract signed the 1st week in April—as long as you can respond to the questions below soon so that we can get the final draft in front of our attorneys and begin the signing process. I know that’s a week off your timeline, but I also confirmed that we have some flexibility to roll over our payment to the next FY if we don’t make it calibrated delivery by the end of June. I still think it’s a good idea to shoot for that.   This is a little far afield, but for the purposes of long-term planning we were wondering whether once you calibrate Replica for us you can go back and produce historical estimates. For instance, if we’re interested in continuing the relationship after this year’s Replica test, but we aren’t able to get the funding together til 2021, could you still give us 2020 outputs at that point so that we’d have a continuous quarterly dataset?   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, March 13, 2019 11:10 AM To: 'Nick Bowden' Cc: Chris Johnson; Robert Kirkman; Michelle Bellia Subject: RE: Revised Portland Metro Replica agreement Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other - potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Chris Johnson; Robert Kirkman; Michelle Bellia RE: Revised Portland Metro Replica agreement Wednesday, March 13, 2019 11:10:00 AM SaaSPortland Metro 3 13 2019 comparison.docx Hello Nick:   Meeting with the team today to discuss this. I think that we’re extremely close to a final draft and expect the conversation will focused on next steps and the timeline for getting it signed on our end.   I just reviewed the contract against my recent notes and emails (attached) and only have two outstanding questions for you. If you can get me some text to add to the contract to help address these it’ll help move this through. Can SWL provide any language further detailing what it means to “make identifications of persons” in 4.5k? We’ve been trying to be clear about this because it’s at the heart of the privacy discussion surrounding this project. For example, in 4.4 we inserted text about protecting privacy meaning that it’s impossible to derive individuals’ home/work location or demographic characteristics from the travel data included in the Output, or vice versa. Does that cover the ways in which Replica could be used to violate privacy, or are there other potential uses that would constitute identifying someone? On 10.2, you mentioned ““we would at least like to have a cap (maximum being the value of the contract as an example)” – I don’t see that language reflected in here.      There’s also a note in there from your attorneys asking you to confirm quarterly payment is OK; assuming that we can let that one slide since we’ve been discussing it all along.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 13, 2019 9:54 AM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement HI Eliot Just wanted to circle back on this and see if you have any questions. Ideally, to stay on schedule, we should try and get this wrapped up in the next week or so. Thanks. Nick On Thu, Mar 7, 2019 at 3:13 PM Eliot Rose wrote: Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Bcc: Subject: Date: Eliot Rose Nick Bowden Chris Johnson; Michelle Bellia; Robert Kirkman 4days@followupthen.com RE: Revised Portland Metro Replica agreement Thursday, March 07, 2019 3:12:00 PM Thanks Nick! I agree that we don’t need to schedule a call right now. I’ll check in with the team and let you know if we have any further questions early next week.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, March 06, 2019 6:15 PM To: Eliot Rose Cc: Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Revised Portland Metro Replica agreement Hi Eliot We are fine on 13.2. On 10.2, in order to be okay with the clause, we would at least like to have a cap (maximum being the value of the contract as an example). Also related to 10.2, we would prefer a carve out on indemnification for any data specifically provided by Metro or used for 4.5. On 4.5, I think some of the convo is getting lost in translation, and that's my fault. The process of building Replica's, and therefore the Replica outputs, protect individual privacy, whereby it's mathematically impossible to re-identify an individual within Replica. Which is different than taking Replica outputs, and then combining them with several other data sets in a specific and nefarious to identify a particular person. We can't conceive of a scenario, with any combination of outside data, whereby this is possible, however, the intent of the clause from our end is to ensure that Replica isn't being used as a mechanism for attempting to specifically target a single individual. From our perspective, there is a difference between a legal safeguard (terms preventing people from doing such a thing) and whether or not said thing is possible (in the case not). For the purposes you have identified, I think the sentence you have added is fine starting with "except for the purposes of..", can be included so you all feel confident in the safeguards. Does that make sense? Happy to discuss if needed, although if we are on same page, may not need a call. On Wed, Mar 6, 2019 at 9:02 AM Eliot Rose wrote: Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Michelle Bellia; Chris Johnson; Robert Kirkman RE: Revised Portland Metro Replica agreement Wednesday, March 06, 2019 7:02:00 AM SaaSPortland Metro 3 6 2019 comparison.docx Thanks Nick! It does look close. I’m attaching a compare version of the two documents that shows your most recent changes vs. the version that we sent to help others follow the edits.   Are you available for a quick call on Monday between 9-3 PT? And do you want to further discuss the comments on 10.2 and 13.2, which are addressed to you, not us? We’d like to talk about how we verify that Replica protects privacy during the validation process – I added our comments back in on 4.5k for discussion – and briefly touch on data ownership.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, March 05, 2019 6:21 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Hi There Unfortunately not able to attend the summit. But, we do have a turn. We are very close. See last few comments from our side. Thanks. Nick On Mon, Mar 4, 2019 at 11:35 AM Eliot Rose wrote: Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Revised Portland Metro Replica agreement Monday, March 04, 2019 9:35:00 AM Thanks! I’m traveling to the Shared Use Mobility Summit this week (any chance I’ll see you there?), but I’ll move this forward with my team as soon as I receive your comments.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, March 03, 2019 11:46 AM To: Eliot Rose Subject: Re: Revised Portland Metro Replica agreement Thanks, Eliot. I should have something back by mid-week at the latest. Nick On Thu, Feb 28, 2019 at 7:47 PM Eliot Rose wrote: Hello Nick and Ben: Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments. I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits. Thanks, Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden; Benjamin Y. Bloom Michelle Bellia; Chris Johnson Revised Portland Metro Replica agreement Thursday, February 28, 2019 5:47:00 PM SaaSPortland Comparison Metro 20190228.docx Hello Nick and Ben:   Attached is our latest round of revisions to the contract based on the input from our partners. All changes since the last version are tracked and the substantive ones are flagged with comments.   I’ve discussed most of these changes with Nick and it sounded like he was OK with them in spirit but wanted to see the specific text changes. Please let us know if you’d like to set up a call to discuss or would prefer to send back another round of edits.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Eliot Rose kevin@sidewalklabs.com Accepted: Invitation: Replica Portland feature sync @ Mon Mar 4, 2019 9am - 10am (PST) (eliot.rose@oregonmetro.gov) From: To: Cc: Subject: Date: Eliot Rose Kevin Grennan Kerr, Michael; Nick Bowden RE: Replica: Follow Up Thursday, February 28, 2019 5:02:00 PM Sorry, I just had a conflict come up. Can we do 9 PT?   Eliot Rose Technology Strategist Metro   From: Kevin Grennan [mailto:kevin@sidewalklabs.com] Sent: Thursday, February 28, 2019 5:00 PM To: Eliot Rose Cc: Kerr, Michael; Nick Bowden Subject: Re: Replica: Follow Up Monday at 10am works for me. Looking forward to hearing about your needs and ensuring we can meet them. Kevin On Thu, Feb 28, 2019 at 4:56 PM Eliot Rose wrote: Time is pretty tight for me next week because I’m headed to the SUMC summit Tuesday afternoon. Can we do Monday @10? Eliot Rose Technology Strategist Metro From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Thursday, February 28, 2019 11:41 AM To: Kevin Grennan; Eliot Rose Cc: Nick Bowden Subject: RE: Replica: Follow Up Kevin, thanks for connecting and apologies for the delayed response. Seeing how it’s already Thursday, how about we look to early next week to connect. I have open windows on Monday and Tuesday – how about you? Eliot? Very eager to get on the phone as we have a legitimate need to get ahead of real concern that it sounds like you all are already aware of. Best, Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue, Suite 800 Portland, OR 97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-8236868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg From: Kevin Grennan Sent: Thursday, February 21, 2019 7:00 PM To: Kerr, Michael ; Eliot Rose Cc: Nick Bowden Subject: Re: Replica: Follow Up Hi Michael & Eliot, I'd love to learn more about your use cases and how we can ensure Replica is ready to meet your needs. It would be also great to get your feedback on some of the things we're planning so we can make sure we're prioritizing the right features. I'm traveling next Monday to Wednesday but available all day next Thursday and Friday afternoon. Let me know if you're free to talk either of those days. Thanks, Kevin On Thu, Feb 21, 2019 at 2:44 PM Nick Bowden wrote: Hi Guys Appreciate the conversation today. Attaching Kevin from our team who leads all of our design and user experience efforts. As I mentioned, we would love to dive a little deeper into possible solutions related to data download -- maintaining value of fidelity without creating unintended risk. I'll let Kevin coordinate with you guys to find a time for a feedback session. Thanks again. Nick From: To: Cc: Subject: Date: Eliot Rose Kerr, Michael; Kevin Grennan Nick Bowden RE: Replica: Follow Up Thursday, February 28, 2019 4:56:00 PM Time is pretty tight for me next week because I’m headed to the SUMC summit Tuesday afternoon. Can we do Monday @10?   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Thursday, February 28, 2019 11:41 AM To: Kevin Grennan; Eliot Rose Cc: Nick Bowden Subject: RE: Replica: Follow Up   Kevin, thanks for connecting and apologies for the delayed response.  Seeing how it’s already Thursday, how about we look to early next week to connect.  I have open windows on Monday and Tuesday – how about you?  Eliot?       Very eager to get on the phone as we have a legitimate need to get ahead of real concern that it sounds like you all are already aware of.    Best,     Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Kevin Grennan Sent: Thursday, February 21, 2019 7:00 PM To: Kerr, Michael ; Eliot Rose Cc: Nick Bowden Subject: Re: Replica: Follow Up   Hi Michael & Eliot,    I'd love to learn more about your use cases and how we can ensure Replica is ready to meet your needs. It would be also great to get your feedback on some of the things we're planning so we can make sure we're prioritizing the right features. I'm traveling next Monday to Wednesday but available all day next Thursday and Friday afternoon. Let me know if you're free to talk either of those days.    Thanks, Kevin   On Thu, Feb 21, 2019 at 2:44 PM Nick Bowden wrote: Hi Guys   Appreciate the conversation today. Attaching Kevin from our team who leads all of our design and user experience efforts. As I mentioned, we would love to dive a little deeper into possible solutions related to data download -- maintaining value of fidelity without creating unintended risk. I'll let Kevin coordinate with you guys to find a time for a feedback session.   Thanks again. Nick   From: To: Subject: Date: Eliot Rose Nick Bowden RE: Portland Replica update Tuesday, February 19, 2019 10:03:00 AM Thanks for setting this up. I think that it’s going to help our team better understand the data. It’s a lot easier for us to understand the context when looking at a US city.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 19, 2019 7:33 AM To: Eliot Rose Subject: Re: Portland Replica update Hi Eliot The list of folks you provided all have access to Replica Kansas City. Visit here: https://replicaexplorer.sidewalklabs.com/ Just use the email you provided to sign up. Happy to answer any questions for people if needed. Thanks. nick On Thu, Feb 14, 2019 at 6:10 PM Eliot Rose wrote: Thursday at 10 PT is confirmed. I’ll send an invite. That will probably be just me and Michael Kerr, who’s managing the project from the Portland side. I think that his main question will be about whether it’s possible to manage permissions so that the download function is turned off for some users and open to others. But it’ll be good for you two to be able to hear from him directly about the privacy concerns that the city has more generally.   Here’s the list of folks who’s like access to the KC tool.   Eliot Rose Eliot.Rose@oregonmetro.gov Chris Johnson Chris.Johnson@oregonmetro.gov   Robert Kirkman Robert.Kirkman@oregonmetro.gov McHugh, Tim McHughT@trimet.org Gilligan, Mike GilligaM@trimet.org Kerr, Michael Michael.Kerr@portlandoregon.gov Lindstrom, Aubrey Aubrey.Lindstrom@portlandoregon.gov Nayame, Jacqueline Jacqueline.Nayame@portlandoregon.gov McDonald, Mike Michael.McDonald@portlandoregon.gov McEwen, Kirk Kirk.McEwen@portlandoregon.gov Zhou, Ningsheng Ningsheng.Zhou@portlandoregon.gov Hesse, Eric Eric.Hesse@portlandoregon.gov Hurley, Peter Peter.T.Hurley@portlandoregon.gov Marx, Michelle Michelle.Marx@portlandoregon.gov Geller, Roger Roger.Geller@portlandoregon.gov Bertelsen, April April.Bertelsen@portlandoregon.gov Pearce, Art Art.Pearce@portlandoregon.gov   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, February 14, 2019 12:24 PM To: Eliot Rose Subject: Re: Portland Replica update Any email works. On Thu, Feb 14, 2019 at 12:46 PM Eliot Rose wrote: Thanks! I’ll confirm that time with the team. And yes, it would be helpful to see KC’s build. I’ll get you emails shortly—do they need to be Google-enabled emails like last time?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, February 14, 2019 10:29 AM To: Eliot Rose Subject: Re: Portland Replica update Happy to join. How about next thursday, 10am PST? You guys should have access to the product, but all I need is email addresses for people and I can provide access to KC. Thanks. Nick On Thu, Feb 14, 2019 at 11:59 AM Eliot Rose wrote: I’ve been talking with Portland more about their concerns about what data they’ll be able to access. I think that it would help them to understand in more detail how the query/download function within the Replica tool works. Would you be up for joining in a half-hour call next week with them to discuss? If so, please let me know some windows that work for you – Wednesday, Thursday AM, and Friday AM are pretty good on our end. Also, if it’s possible to share a link to an updated version of the tool that they can view in advance of the call that would help as well.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, February 13, 2019 8:09 AM To: Eliot Rose Subject: Re: Portland Replica update Yes, we would provide the data table outside of the tool / product itself. Either through an API or a dump of the data table into a cloud bucket you all maintain. I am happy to make an intro to Jean-Louis if you believe it will help. I think it's important to note that he doesn't have any expertise as it relates to the contextual use or communication of Replica. His expertise is simply in testing privacy and security frameworks within applications. If there are particular questions about privacy that remain, I am positively certain we can provide the necessary information / documentation required. Let me know how you would like to proceed. Thanks. Nick On Tue, Feb 12, 2019 at 12:09 PM Eliot Rose wrote: Thanks Nick:   Appreciate all of the information; I’ll forward it on to members of the group.   I suspect that we’ll end up requesting that Metro have access to the entire data table, but not the other partners in the group. We haven’t discussed the method by which that would be shared, but I was assuming from the conversation that it would be provided separately from the online tool. Is that correct?   Would you be open to us reaching out to Jean-Louis Tambay to ask him for some more of the background on his findings? I understand why you can’t share his report, but I want to make sure that we do our due diligence on privacy. Staff continue to get a lot of questions about SWL/Replica from the public, and it might help for us to be able to run those questions by him.   Thanks, Eliot   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 12, 2019 8:45 AM To: Eliot Rose Subject: Re: Portland Replica update Hi Eliot Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions: 1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level. Let me know if you want to chat by phone. Thanks. Nick On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick: Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call. I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract. Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: · Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? · You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. · There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria? · How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts. Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Eliot Rose Nick Bowden; Kerr, Michael; Martin, Kevin Chris Johnson; Robert Kirkman Portland Replica data This call is to clarify the level of detail that will be available through Replica data, both in the online tool and via download/query. Mike and I will be on for sure; Robb, Chris and Kevin are welcome to join if available. From: To: Subject: Date: Eliot Rose Nick Bowden RE: Portland Replica update Thursday, February 14, 2019 4:10:00 PM Thursday at 10 PT is confirmed. I’ll send an invite. That will probably be just me and Michael Kerr, who’s managing the project from the Portland side. I think that his main question will be about whether it’s possible to manage permissions so that the download function is turned off for some users and open to others. But it’ll be good for you two to be able to hear from him directly about the privacy concerns that the city has more generally.   Here’s the list of folks who’s like access to the KC tool.   Eliot Rose Eliot.Rose@oregonmetro.gov Chris Johnson Chris.Johnson@oregonmetro.gov   Robert Kirkman Robert.Kirkman@oregonmetro.gov McHugh, Tim McHughT@trimet.org Gilligan, Mike GilligaM@trimet.org Kerr, Michael Michael.Kerr@portlandoregon.gov Lindstrom, Aubrey Aubrey.Lindstrom@portlandoregon.gov Nayame, Jacqueline Jacqueline.Nayame@portlandoregon.gov McDonald, Mike Michael.McDonald@portlandoregon.gov McEwen, Kirk Kirk.McEwen@portlandoregon.gov Zhou, Ningsheng Ningsheng.Zhou@portlandoregon.gov Hesse, Eric Eric.Hesse@portlandoregon.gov Hurley, Peter Peter.T.Hurley@portlandoregon.gov Marx, Michelle Michelle.Marx@portlandoregon.gov Geller, Roger Roger.Geller@portlandoregon.gov Bertelsen, April April.Bertelsen@portlandoregon.gov Pearce, Art Art.Pearce@portlandoregon.gov   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, February 14, 2019 12:24 PM To: Eliot Rose Subject: Re: Portland Replica update Any email works. On Thu, Feb 14, 2019 at 12:46 PM Eliot Rose wrote: Thanks! I’ll confirm that time with the team. And yes, it would be helpful to see KC’s build. I’ll get you emails shortly—do they need to be Google-enabled emails like last time?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, February 14, 2019 10:29 AM To: Eliot Rose Subject: Re: Portland Replica update Happy to join. How about next thursday, 10am PST? You guys should have access to the product, but all I need is email addresses for people and I can provide access to KC. Thanks. Nick On Thu, Feb 14, 2019 at 11:59 AM Eliot Rose wrote: I’ve been talking with Portland more about their concerns about what data they’ll be able to access. I think that it would help them to understand in more detail how the query/download function within the Replica tool works. Would you be up for joining in a half-hour call next week with them to discuss? If so, please let me know some windows that work for you – Wednesday, Thursday AM, and Friday AM are pretty good on our end. Also, if it’s possible to share a link to an updated version of the tool that they can view in advance of the call that would help as well.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, February 13, 2019 8:09 AM To: Eliot Rose Subject: Re: Portland Replica update Yes, we would provide the data table outside of the tool / product itself. Either through an API or a dump of the data table into a cloud bucket you all maintain. I am happy to make an intro to Jean-Louis if you believe it will help. I think it's important to note that he doesn't have any expertise as it relates to the contextual use or communication of Replica. His expertise is simply in testing privacy and security frameworks within applications. If there are particular questions about privacy that remain, I am positively certain we can provide the necessary information / documentation required. Let me know how you would like to proceed. Thanks. Nick On Tue, Feb 12, 2019 at 12:09 PM Eliot Rose wrote: Thanks Nick:   Appreciate all of the information; I’ll forward it on to members of the group.   I suspect that we’ll end up requesting that Metro have access to the entire data table, but not the other partners in the group. We haven’t discussed the method by which that would be shared, but I was assuming from the conversation that it would be provided separately from the online tool. Is that correct?   Would you be open to us reaching out to Jean-Louis Tambay to ask him for some more of the background on his findings? I understand why you can’t share his report, but I want to make sure that we do our due diligence on privacy. Staff continue to get a lot of questions about SWL/Replica from the public, and it might help for us to be able to run those questions by him.   Thanks, Eliot   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 12, 2019 8:45 AM To: Eliot Rose Subject: Re: Portland Replica update Hi Eliot Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions: 1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level. Let me know if you want to chat by phone. Thanks. Nick On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick: Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call. I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract. Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: · Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? · You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. · There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria? · How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts. Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Portland Replica update Thursday, February 14, 2019 10:46:00 AM Thanks! I’ll confirm that time with the team. And yes, it would be helpful to see KC’s build. I’ll get you emails shortly—do they need to be Google-enabled emails like last time?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, February 14, 2019 10:29 AM To: Eliot Rose Subject: Re: Portland Replica update Happy to join. How about next thursday, 10am PST? You guys should have access to the product, but all I need is email addresses for people and I can provide access to KC. Thanks. Nick On Thu, Feb 14, 2019 at 11:59 AM Eliot Rose wrote: I’ve been talking with Portland more about their concerns about what data they’ll be able to access. I think that it would help them to understand in more detail how the query/download function within the Replica tool works. Would you be up for joining in a half-hour call next week with them to discuss? If so, please let me know some windows that work for you – Wednesday, Thursday AM, and Friday AM are pretty good on our end. Also, if it’s possible to share a link to an updated version of the tool that they can view in advance of the call that would help as well.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, February 13, 2019 8:09 AM To: Eliot Rose Subject: Re: Portland Replica update Yes, we would provide the data table outside of the tool / product itself. Either through an API or a dump of the data table into a cloud bucket you all maintain. I am happy to make an intro to Jean-Louis if you believe it will help. I think it's important to note that he doesn't have any expertise as it relates to the contextual use or communication of Replica. His expertise is simply in testing privacy and security frameworks within applications. If there are particular questions about privacy that remain, I am positively certain we can provide the necessary information / documentation required. Let me know how you would like to proceed. Thanks. Nick On Tue, Feb 12, 2019 at 12:09 PM Eliot Rose wrote: Thanks Nick:   Appreciate all of the information; I’ll forward it on to members of the group.   I suspect that we’ll end up requesting that Metro have access to the entire data table, but not the other partners in the group. We haven’t discussed the method by which that would be shared, but I was assuming from the conversation that it would be provided separately from the online tool. Is that correct?   Would you be open to us reaching out to Jean-Louis Tambay to ask him for some more of the background on his findings? I understand why you can’t share his report, but I want to make sure that we do our due diligence on privacy. Staff continue to get a lot of questions about SWL/Replica from the public, and it might help for us to be able to run those questions by him.   Thanks, Eliot   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 12, 2019 8:45 AM To: Eliot Rose Subject: Re: Portland Replica update Hi Eliot Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions: 1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level. Let me know if you want to chat by phone. Thanks. Nick On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick: Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call. I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract. Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: · Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? · You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. · There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria? · How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts. Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Portland Replica update Thursday, February 14, 2019 9:59:00 AM I’ve been talking with Portland more about their concerns about what data they’ll be able to access. I think that it would help them to understand in more detail how the query/download function within the Replica tool works. Would you be up for joining in a half-hour call next week with them to discuss? If so, please let me know some windows that work for you – Wednesday, Thursday AM, and Friday AM are pretty good on our end. Also, if it’s possible to share a link to an updated version of the tool that they can view in advance of the call that would help as well.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, February 13, 2019 8:09 AM To: Eliot Rose Subject: Re: Portland Replica update Yes, we would provide the data table outside of the tool / product itself. Either through an API or a dump of the data table into a cloud bucket you all maintain. I am happy to make an intro to Jean-Louis if you believe it will help. I think it's important to note that he doesn't have any expertise as it relates to the contextual use or communication of Replica. His expertise is simply in testing privacy and security frameworks within applications. If there are particular questions about privacy that remain, I am positively certain we can provide the necessary information / documentation required. Let me know how you would like to proceed. Thanks. Nick On Tue, Feb 12, 2019 at 12:09 PM Eliot Rose wrote: Thanks Nick:   Appreciate all of the information; I’ll forward it on to members of the group.   I suspect that we’ll end up requesting that Metro have access to the entire data table, but not the other partners in the group. We haven’t discussed the method by which that would be shared, but I was assuming from the conversation that it would be provided separately from the online tool. Is that correct?   Would you be open to us reaching out to Jean-Louis Tambay to ask him for some more of the background on his findings? I understand why you can’t share his report, but I want to make sure that we do our due diligence on privacy. Staff continue to get a lot of questions about SWL/Replica from the public, and it might help for us to be able to run those questions by him.   Thanks, Eliot   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 12, 2019 8:45 AM To: Eliot Rose Subject: Re: Portland Replica update Hi Eliot Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions: 1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level. Let me know if you want to chat by phone. Thanks. Nick On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick: Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call. I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract. Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: · Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? · You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. · There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria? · How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts. Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Portland Replica update Wednesday, February 13, 2019 8:13:00 AM Thanks! I’m talking w/ Portland about whether it would help to have a conversation with Jean-Louis since they’re the ones who are hearing the biggest concerns from residents right now. I’ll let you know.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, February 13, 2019 8:09 AM To: Eliot Rose Subject: Re: Portland Replica update Yes, we would provide the data table outside of the tool / product itself. Either through an API or a dump of the data table into a cloud bucket you all maintain. I am happy to make an intro to Jean-Louis if you believe it will help. I think it's important to note that he doesn't have any expertise as it relates to the contextual use or communication of Replica. His expertise is simply in testing privacy and security frameworks within applications. If there are particular questions about privacy that remain, I am positively certain we can provide the necessary information / documentation required. Let me know how you would like to proceed. Thanks. Nick On Tue, Feb 12, 2019 at 12:09 PM Eliot Rose wrote: Thanks Nick:   Appreciate all of the information; I’ll forward it on to members of the group.   I suspect that we’ll end up requesting that Metro have access to the entire data table, but not the other partners in the group. We haven’t discussed the method by which that would be shared, but I was assuming from the conversation that it would be provided separately from the online tool. Is that correct?   Would you be open to us reaching out to Jean-Louis Tambay to ask him for some more of the background on his findings? I understand why you can’t share his report, but I want to make sure that we do our due diligence on privacy. Staff continue to get a lot of questions about SWL/Replica from the public, and it might help for us to be able to run those questions by him.   Thanks, Eliot   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.     Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 12, 2019 8:45 AM To: Eliot Rose Subject: Re: Portland Replica update Hi Eliot Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions: 1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies. 2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us. 3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes. 4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level. Let me know if you want to chat by phone. Thanks. Nick On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick: Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call. I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract. Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: · Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? · You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. · There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria? · How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts. Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Portland Replica update Tuesday, February 12, 2019 10:09:00 AM Thanks Nick:   Appreciate all of the information; I’ll forward it on to members of the group.   I suspect that we’ll end up requesting that Metro have access to the entire data table, but not the other partners in the group. We haven’t discussed the method by which that would be shared, but I was assuming from the conversation that it would be provided separately from the online tool. Is that correct?   Would you be open to us reaching out to Jean-Louis Tambay to ask him for some more of the background on his findings? I understand why you can’t share his report, but I want to make sure that we do our due diligence on privacy. Staff continue to get a lot of questions about SWL/Replica from the public, and it might help for us to be able to run those questions by him.   Thanks, Eliot   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies.    2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us.    3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes.    4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.      Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, February 12, 2019 8:45 AM To: Eliot Rose Subject: Re: Portland Replica update   Hi Eliot   Thank you for the update. Look forward to receiving the last turn of the agreement from you all. In terms of your questions:   1. On the first question, end users can export desegregated trips based on the queries they build. It's all synthetic of course, but that functionality exist for everyone. We were planning to provide you all with complete access to the entire data table, which is not something we would do for other agencies.    2. We can't share the entire report, only in that it is a comprehensive security and privacy report which a substantial amount of proprietary and sensitive information related to our architecture, algorithms, and data processing. The author of the report is Jean-Louis Tambay, a renowned privacy expert based in Canada, who works fulltime for STATCAN, and does independent privacy reviews like he did for us.    3. Of course on the validation data. As long as we get / have access to the required data needed for calibration, you all holding some back for validation is great. Many regions don't collect enough ground-truth to truly have a control group of sorts for validation purposes.    4. Intersection crossings are really tough to measure in that, they aren't typically consider a network link, say like a sidewalk. For the most part, they are entirely unavailable in both OpenStreetMap or google. If you have ground-truth specific to crossings, we can certainly attempt to do something, but I don't want to set any unrealistic expectations around our ability to do or subsequent accuracy at the crossing level.    Let me know if you want to chat by phone.   Thanks. Nick     On Thu, Feb 7, 2019 at 6:01 PM Eliot Rose wrote: Hello Nick:   Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call.  I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract.   Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: ·         Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? ·         You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. ·         There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria?      ·         How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts.   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Nick Bowden Portland Replica update Thursday, February 07, 2019 4:01:00 PM Hello Nick:   Quick status update from the Portland region: the IGA is signed and we held calls today to go over the contract and acceptance criteria with Portland and TriMet. They have until 2/15 to complete their review of the contract and then we hope to reconcile any changes and get them back to you by 2/22, at which point it’ll be hopefully close to final. Based on what I heard today, I expect further refinements from them with respect to the issues that we’ve already been working on with you – clarifying use and ownership of the Replica Output and protecting PII (we continue to hear concerns from our residents about privacy) – but I didn’t hear any significant new concerns on the call.  I’m also hoping we’ll be able to make good headway on getting you some draft acceptance criteria by 2/22 so that we have those ready to insert into the contract.   Here are some quick questions to help us move forward – let me know if you’d like to chat by phone: · Our understanding of data access under this agreement is that Metro and all our partner agencies will have access to the Replica web tool, which enables users to query and download aggregate TAZ-level data, and that SWL will provide Metro (but not our partners) with the disaggregate output at the trip-person-activity level. Are we on the same page there? · You added an excerpt from a privacy audit of Replica to the Data Model and Schema document. Are you willing to tell us who wrote that and/or share the complete report? It would help instill some confidence among people that are concerned about privacy. · There’s some data that we’d like to use in the validation process that’s not publicly available, which creates some barriers to sharing it with SWL. We know that you intend to validate the data in parallel. Are you open to us withholding some of the data that we use in the validation process and then sharing it if/when we determine that Replica fails the associated validation criteria?      · How detailed is the pedestrian data in Replica – can we use it to look at intersection crossings? We have a lot of data on intersection crossings, but that tends to be more complicated than traditional ped data that are based on mid-block counts.   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Nick Bowden RE: Metro review of Replica agreement Tuesday, January 29, 2019 5:28:00 PM Hello Nick:   Portland should have an IGA signed tomorrow, and once they do we’ll countersign and get them the contract for review. The public continues to raise concerns about privacy, especially in front of Portland, so they’ll be looking closely at those aspects of the contract.   I think that it’s good to shoot for late February. Probably best to start w/ Portland, TriMet, and Metro, and then follow up w/ the three counties. I’ll reach out to folks about scheduling when we kickoff the contract review.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Tuesday, January 29, 2019 12:17 PM To: Eliot Rose Subject: Re: Metro review of Replica agreement Hi Eliot Just wanted to check in and see how the review process is going? On another note, one thing we have been doing to ease the onboarding process is starting conversations with partner organizations early in the process. Intent of these is to give them a little introduction to the product and also learn about the projects they are working on prior to launching. Is it possible to look at time mid to late February for us to visit and talk with 3-6 of the key agencies involved? Typically try and do an hour per agency over a 2 day period. Thanks. Nick On Fri, Jan 11, 2019 at 5:40 PM Eliot Rose wrote: Thanks Nick! Looking forward to reviewing this. As for the agreement, we’re close to our final review and I don’t expect any changes that are going to be surprising to you. I’ll let you know if any come up or if we have questions about the documentation, but otherwise I’m inclined to let our partners review the current version once they sign the IGA before we send it back to you to minimize back and forth. That could take up to 3 weeks because it’ll take another week to get the IGA signed and then they have up to two weeks for review. I’ll try to keep the process moving and notify you ahead of time if there are significant questions that come up.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, January 11, 2019 2:23 PM To: Eliot Rose Cc: Benjamin Y. Bloom; Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Metro review of Replica agreement Hi Eliot (and team) Thanks for taking time to speak this week. As promised, I'm attaching a new version of the data disclosure document that includes our vendor audit process and a summary for the third party privacy audit we had conducted on Replica. I believe you all are going to turn back one final redline for the agreement and then we should be good to go. Please let me know if you have any questions. Thanks everyone. Have a great weekend! Nick On Mon, Jan 7, 2019 at 10:48 AM Eliot Rose wrote: Thanks for the heads-up, Nick. It looks like our team could do 10:30 or 11:30 as well if that works better for you. Since we only have a half hour it’d be helpful if everyone’s able to join in for as much of the call as possible.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 8:23 AM To: Eliot Rose; Benjamin Y. Bloom Subject: Re: Metro review of Replica agreement Hi Eliot Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed. Sorry for the last minute change (typing from plane now). Talk soon. Nick On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments —seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) - Timeline for signing the contract - Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Hi Eliot Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below. Attaching a few items here: 1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation. We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created. Thanks. As always, please let me know if you have questions. Nick On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread. We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items. In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them. We can provide the Replica data schema and all related metadata as well. We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know. Happy Holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Thanks for sending this Eliot. There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions. On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this. I am hoping I will be on the call but wanted to send this just in case. Thanks. Nick On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. · How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. · How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. · Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments —but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Benjamin Y. Bloom; Chris Johnson; Michelle Bellia; Robert Kirkman RE: Metro review of Replica agreement Friday, January 11, 2019 3:40:00 PM Thanks Nick! Looking forward to reviewing this. As for the agreement, we’re close to our final review and I don’t expect any changes that are going to be surprising to you. I’ll let you know if any come up or if we have questions about the documentation, but otherwise I’m inclined to let our partners review the current version once they sign the IGA before we send it back to you to minimize back and forth. That could take up to 3 weeks because it’ll take another week to get the IGA signed and then they have up to two weeks for review. I’ll try to keep the process moving and notify you ahead of time if there are significant questions that come up.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, January 11, 2019 2:23 PM To: Eliot Rose Cc: Benjamin Y. Bloom; Chris Johnson; Michelle Bellia; Robert Kirkman Subject: Re: Metro review of Replica agreement Hi Eliot (and team) Thanks for taking time to speak this week. As promised, I'm attaching a new version of the data disclosure document that includes our vendor audit process and a summary for the third party privacy audit we had conducted on Replica. I believe you all are going to turn back one final redline for the agreement and then we should be good to go. Please let me know if you have any questions. Thanks everyone. Have a great weekend! Nick On Mon, Jan 7, 2019 at 10:48 AM Eliot Rose wrote: Thanks for the heads-up, Nick. It looks like our team could do 10:30 or 11:30 as well if that works better for you. Since we only have a half hour it’d be helpful if everyone’s able to join in for as much of the call as possible.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 8:23 AM To: Eliot Rose; Benjamin Y. Bloom Subject: Re: Metro review of Replica agreement Hi Eliot Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed. Sorry for the last minute change (typing from plane now). Talk soon. Nick On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments— seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) - Timeline for signing the contract - Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Hi Eliot Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below. Attaching a few items here: 1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation. We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created. Thanks. As always, please let me know if you have questions. Nick On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread. We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items. In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them. We can provide the Replica data schema and all related metadata as well. We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know. Happy Holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Thanks for sending this Eliot. There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions. On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this. I am hoping I will be on the call but wanted to send this just in case. Thanks. Nick On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. · How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. · How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. · Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments— but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Subject: Attachments: Eliot Rose on behalf of bowden@sidewalklabs.com Michelle Bellia; Chris Johnson; Robert Kirkman; Eliot Rose; bbloom@mindengross.com; valerie.eisen@sidewalklabs.com; alexei@sidewalklabs.com FW: Updated invitation: Metro + Sidewalk Labs (Contract) @ Mon Jan 7, 2019 12:45pm - 1:15pm (CST) (eliot.rose@oregonmetro.gov) invite.ics Bumping up the time of this a bit. Let’s plan on gathering in Chris’ office. -----Original Appointment----From: bowden@sidewalklabs.com [mailto:bowden@sidewalklabs.com] Sent: Thursday, January 03, 2019 7:25 AM To: bowden@sidewalklabs.com; Eliot Rose; bbloom@mindengross.com; valerie.eisen@sidewalklabs.com; alexei@sidewalklabs.com Subject: Updated invitation: Metro + Sidewalk Labs (Contract) @ Mon Jan 7, 2019 12:45pm - 1:15pm (CST) (eliot.rose@oregonmetro.gov) When: Monday, January 07, 2019 10:45 AM-11:15 AM (UTC-08:00) Pacific Time (US & Canada). Where: USE HANGOUT CALL-IN NUMBER This event has been changed. more details » Metro + Sidewalk Labs (Contract) When  Changed: Mon Jan 7, 2019 12:45pm – 1:15pm Central Time - Chicago   Where  USE HANGOUT CALL-IN NUMBER (map )   Joining info  meet.google.com/urw-ewqm-tye    Or dial: +1 513-401-7245  PIN: 140385#  More phone numbers     Calendar  eliot.rose@oregonmetro.gov   Who  •  bowden@sidewalklabs.com   - organizer   •  eliot.rose@oregonmetro.gov   •  bbloom@mindengross.com   •  valerie.eisen@sidewalklabs.com   •  alexei@sidewalklabs.com     Going (eliot.rose@oregonmetro.gov )?   Yes   - Maybe   - No      more options »   Invitation from Google Calendar You are receiving this courtesy email at the account eliot.rose@oregonmetro.gov   because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to modify your RSVP response. Learn More .    From: To: Subject: Eliot Rose bowden@sidewalklabs.com Accepted: Updated invitation: Metro + Sidewalk Labs (Contract) @ Mon Jan 7, 2019 12:45pm - 1:15pm (CST) (eliot.rose@oregonmetro.gov) From: To: Subject: Eliot Rose bowden@sidewalklabs.com Accepted: Updated invitation: Metro + Sidewalk Labs (Contract) @ Mon Jan 7, 2019 12:30pm - 1pm (CST) (eliot.rose@oregonmetro.gov) From: To: Cc: Subject: Date: Eliot Rose Nick Bowden; Benjamin Y. Bloom Chris Johnson; Michelle Bellia; Robert Kirkman RE: Metro review of Replica agreement Monday, January 07, 2019 8:48:00 AM Thanks for the heads-up, Nick. It looks like our team could do 10:30 or 11:30 as well if that works better for you. Since we only have a half hour it’d be helpful if everyone’s able to join in for as much of the call as possible.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, January 07, 2019 8:23 AM To: Eliot Rose; Benjamin Y. Bloom Subject: Re: Metro review of Replica agreement Hi Eliot Looking forward to our call today. I am traveling today and was supposed to arrive in SF long before our meeting started. However, delays have changed my schedule quite a bit. I am still planning to be on the call, from the airport, for at least the first 15 minutes. Ben and Alexei will be on the call from our team as well. Ben, to handle contract questions and Alexei to discuss data and method if needed. Sorry for the last minute change (typing from plane now). Talk soon. Nick On Fri, Jan 4, 2019 at 11:52 AM Eliot Rose wrote: Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments— seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) - Timeline for signing the contract - Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Hi Eliot Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below. Attaching a few items here: 1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation. We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created. Thanks. As always, please let me know if you have questions. Nick On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread. We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items. In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them. We can provide the Replica data schema and all related metadata as well. We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know. Happy Holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Thanks for sending this Eliot. There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions. On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this. I am hoping I will be on the call but wanted to send this just in case. Thanks. Nick On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. · How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. · How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. · Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson RE: Metro review of Replica agreement Friday, January 04, 2019 9:52:00 AM SaaSPortland Comparison Metro 20190104.docx Hello all:   Looking forward to talking on Monday morning. I’m attaching our latest edits and comments— seems like we’re getting close. For some reason this document is still flattening all tracked changes on our end; apologies for that. Fortunately they’re easier to keep straight now that there are fewer of them—all SWL and Metro changes to the agreement since the previous Metro version are tracked, and all comments are preserved, but I’ve highlighted the new comments and proposed changes to the agreement from our end that we’d like to discuss on the call. When we chat, we’d like to discuss: Remaining changes to the contract (primarily the comments re: the definitions and TOU; it seems like we can address the rest via email) Timeline for signing the contract Remaining questions re: documentation and methodology (time permitting)   Have a great weekend, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 11:08 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Hi Eliot Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below. Attaching a few items here: 1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation. We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created. Thanks. As always, please let me know if you have questions. Nick On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread. We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items. In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them. We can provide the Replica data schema and all related metadata as well. We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know. Happy Holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Thanks for sending this Eliot. There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions. On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this. I am hoping I will be on the call but wanted to send this just in case. Thanks. Nick On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. · How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. · How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. · Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Michelle Bellia; Valerie Eisen; Clifford Higgins RE: Replica agreement - public records request Friday, January 04, 2019 7:49:00 AM Thanks Nick—that sounds good.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, January 03, 2019 2:59 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen; Clifford Higgins Subject: Re: Replica agreement - public records request Sorry for not clarifying. It is fine if you turn over the term sheet with the pricing formula redacted. The rest of the information in the term sheet is about timing and product outputs, both of which are contained in the email thread and / or are available publicly. Is this an okay solution? Thanks. Nick On Thu, Jan 3, 2019 at 4:46 PM Eliot Rose wrote: Thanks again for the quick response, Nick. I didn’t see an answer from you about the attachments in the PDF, but based on what you’re telling me about how SWL views trade secrets we plan to disclose the term sheet as redacted in the attached version. The request specifically mentions signed agreements between us and SWL, so there’s a high likelihood that the requestor would petition the DA to challenge the withholding the term sheet, and we won’t be able to defend withholding it. We plan to provide the emails with your redactions incorporated and other attachments omitted.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, January 03, 2019 8:08 AM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen; Clifford Higgins Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails_Final.pdf Thanks, Eliot. I've attached an updated redaction request PDF. We do believe these requests fall under the broad definition of either trade secrets or confidentiality as defined by Oregon Law. Broadly speaking, we are the only provider of a Software-as-a-Service, fully calibrated travel-demand model in the United States at this time, which has resulted and will likely continue to result in competing firms looking to uncover as much about our product and business model as possible. Specifically; 1. Our business model of working with MPOs to provide services to all cities and counties, using a per capita pricing model, is the first of its kind and something, particularly the business plan and pricing formula things we consider to be trade secrets. We don't object to the total price paid (as is shown in the term sheet and contract), but the distribution model and pricing formula is something we consider a trade secret at this time. 2. References to other customers, their specific outputs, use cases, acceptance reports, and the like we consider to be trade secrets as well. 3. Our technical process and procedure of combining several disparate data sets, using a synthetic population generator, and then assigning that population movements through an activity generator we consider to be a significant technical trade secret. I suspect the difference with many other data vendors is that most, if not all, have a single source of data and do aggregation as a primary means of privacy-protections. Additionally, the specific sources of this data and how / when it is used is something we consider to be a significant trade secret. 4. Non-public documents that detail the above process or provide more than readily available documentation about our process, methodology, or technical procedures we also consider to fall under the trade secret definition. 5. We also consider our customer release schedule to be confidential. 6. Finally, because the final contract is not being released, any specific reference of clauses or documents associated specifically with the final contract have been redacted. In this second review, I've removed redactions to anything that would be considered common knowledge about our product, process, or customers. Moving forward, happy to flag these items in communications. Hope this helps. Happy to discuss if needed. Thanks. Nick On Wed, Jan 2, 2019 at 7:35 PM Eliot Rose wrote: Hello Nick:   Thanks for turning around your review of our response to the public records request so quickly. I reviewed your suggested redactions and discussed with our attorneys’ office today. I’d encourage you to take one more pass at this before tomorrow (Thursday) COB and make sure that all the redactions you’re requesting are defensible given Oregon law’s definitions of confidential and trade secret (below), and remove those that aren’t. Also, I saw that many of the attachments linked to in the PDF are highlighted—are these attachments that you’re requesting we remove? Happy to hop on a call to discuss any of this tomorrow; I’m available from 8-10 or from 12:30-2:30.   If the requestor pushes back and asks to see the redacted information their request will go before a DA, who may question the redactions as a whole and order us to release a version with all redactions removed if he/she feels they are not well-based in the law. I understand the intent behind many of your proposed redactions given what you’ve told me about how Sidewalk views trade secrets, but some of them seem questionable, especially requesting that we withhold the signed term sheet. Our signed agreements generally tend to be a matter of public record, and this redaction seems likely to be challenged by the requestor. If you’re concerned about the pricing information becoming public record we could just redact the Cost and Cost Certainty sections of the term sheet. The redactions to the text of my emails and the request that we withhold the methodology article that you shared with me, which has been published in a peer-reviewed journal, also seem questionable (assuming that your highlights mean that you want us to withhold that attachment).   Unfortunately, we’ve also got to ask you to flag all associated text that recurs throughout the response. It would also be helpful if you can provide us with a more detailed explanation about why the redactions that you are requesting are necessary – with some of the more established data products we work with high-level descriptions of the methodology and cost, like some of what you’ve flagged for redaction, aren’t always trade secret. Is there something about the unique nature of Replica that makes them a trade secret in this case?   I checked in with the City of Portland today, and I learned that they were also hit with a public records request. Because of the timing of the request—after they approved the IGA, but before they had reviewed the draft agreement—they complied in full, which means that the requestor likely has access to anything that you’ve shared directly with the broader group of agencies that were involved in the conversation, which includes the data sample that you shared with us. In the future can you please flag any documents that you consider to be confidential or trade secret up front?   Thanks, Eliot   Confidential:   192.355(4) Information submitted to a public body in confidence and not otherwise required by law to be submitted, where such information should reasonably be considered confidential, the public body has obliged itself in good faith not to disclose the information, and when the public interest would suffer by the disclosure. Trade secret:   192.345(2) [U]nless the public interest requires disclosure in the particular instance, [t]rade secrets. “Trade secrets,” as used in this section, may include, but are not limited to, any formula, plan, pattern, process, tool, mechanism, compound, procedure, production data, or compilation of information which is not patented, which is known only to certain individuals within an organization and which is used in a business it conducts, having actual or potential commercial value, and which gives its user an opportunity to obtain a business advantage over competitors who do not know or use it.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Michelle Bellia; Valerie Eisen; Clifford Higgins RE: Replica agreement - public records request Thursday, January 03, 2019 2:46:00 PM Replica Portland Term Sheet_Metro signed redacted.pdf Thanks again for the quick response, Nick. I didn’t see an answer from you about the attachments in the PDF, but based on what you’re telling me about how SWL views trade secrets we plan to disclose the term sheet as redacted in the attached version. The request specifically mentions signed agreements between us and SWL, so there’s a high likelihood that the requestor would petition the DA to challenge the withholding the term sheet, and we won’t be able to defend withholding it. We plan to provide the emails with your redactions incorporated and other attachments omitted.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, January 03, 2019 8:08 AM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen; Clifford Higgins Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails_Final.pdf Thanks, Eliot. I've attached an updated redaction request PDF. We do believe these requests fall under the broad definition of either trade secrets or confidentiality as defined by Oregon Law. Broadly speaking, we are the only provider of a Software-as-a-Service, fully calibrated travel-demand model in the United States at this time, which has resulted and will likely continue to result in competing firms looking to uncover as much about our product and business model as possible. Specifically; 1. Our business model of working with MPOs to provide services to all cities and counties, using a per capita pricing model, is the first of its kind and something, particularly the business plan and pricing formula things we consider to be trade secrets. We don't object to the total price paid (as is shown in the term sheet and contract), but the distribution model and pricing formula is something we consider a trade secret at this time. 2. References to other customers, their specific outputs, use cases, acceptance reports, and the like we consider to be trade secrets as well. 3. Our technical process and procedure of combining several disparate data sets, using a synthetic population generator, and then assigning that population movements through an activity generator we consider to be a significant technical trade secret. I suspect the difference with many other data vendors is that most, if not all, have a single source of data and do aggregation as a primary means of privacy-protections. Additionally, the specific sources of this data and how / when it is used is something we consider to be a significant trade secret. 4. Non-public documents that detail the above process or provide more than readily available documentation about our process, methodology, or technical procedures we also consider to fall under the trade secret definition. 5. We also consider our customer release schedule to be confidential. 6. Finally, because the final contract is not being released, any specific reference of clauses or documents associated specifically with the final contract have been redacted. In this second review, I've removed redactions to anything that would be considered common knowledge about our product, process, or customers. Moving forward, happy to flag these items in communications. Hope this helps. Happy to discuss if needed. Thanks. Nick On Wed, Jan 2, 2019 at 7:35 PM Eliot Rose wrote: Hello Nick:   Thanks for turning around your review of our response to the public records request so quickly. I reviewed your suggested redactions and discussed with our attorneys’ office today. I’d encourage you to take one more pass at this before tomorrow (Thursday) COB and make sure that all the redactions you’re requesting are defensible given Oregon law’s definitions of confidential and trade secret (below), and remove those that aren’t. Also, I saw that many of the attachments linked to in the PDF are highlighted—are these attachments that you’re requesting we remove? Happy to hop on a call to discuss any of this tomorrow; I’m available from 8-10 or from 12:302:30.   If the requestor pushes back and asks to see the redacted information their request will go before a DA, who may question the redactions as a whole and order us to release a version with all redactions removed if he/she feels they are not well-based in the law. I understand the intent behind many of your proposed redactions given what you’ve told me about how Sidewalk views trade secrets, but some of them seem questionable, especially requesting that we withhold the signed term sheet. Our signed agreements generally tend to be a matter of public record, and this redaction seems likely to be challenged by the requestor. If you’re concerned about the pricing information becoming public record we could just redact the Cost and Cost Certainty sections of the term sheet. The redactions to the text of my emails and the request that we withhold the methodology article that you shared with me, which has been published in a peer-reviewed journal, also seem questionable (assuming that your highlights mean that you want us to withhold that attachment).   Unfortunately, we’ve also got to ask you to flag all associated text that recurs throughout the response. It would also be helpful if you can provide us with a more detailed explanation about why the redactions that you are requesting are necessary – with some of the more established data products we work with high-level descriptions of the methodology and cost, like some of what you’ve flagged for redaction, aren’t always trade secret. Is there something about the unique nature of Replica that makes them a trade secret in this case?   I checked in with the City of Portland today, and I learned that they were also hit with a public records request. Because of the timing of the request—after they approved the IGA, but before they had reviewed the draft agreement—they complied in full, which means that the requestor likely has access to anything that you’ve shared directly with the broader group of agencies that were involved in the conversation, which includes the data sample that you shared with us. In the future can you please flag any documents that you consider to be confidential or trade secret up front?   Thanks, Eliot   Confidential:   192.355(4) Information submitted to a public body in confidence and not otherwise required by law to be submitted, where such information should reasonably be considered confidential, the public body has obliged itself in good faith not to disclose the information, and when the public interest would suffer by the disclosure. Trade secret:   192.345(2) [U]nless the public interest requires disclosure in the particular instance, [t]rade secrets. “Trade secrets,” as used in this section, may include, but are not limited to, any formula, plan, pattern, process, tool, mechanism, compound, procedure, production data, or compilation of information which is not patented, which is known only to certain individuals within an organization and which is used in a business it conducts, having actual or potential commercial value, and which gives its user an opportunity to obtain a business advantage over competitors who do not know or use it.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot - We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland OR 97232?2736 503?797?1825 Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Replica agreement - public records request Thursday, January 03, 2019 7:46:00 AM Great, thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, January 03, 2019 7:23 AM To: Eliot Rose Subject: Re: Replica agreement - public records request Let's plan on 11am PST Monday to have the discussion to wrap up the contract. I will send an invite now to you, feel free to share with whomever on your team that will attend. Thanks. Nick On Wed, Jan 2, 2019 at 7:47 PM Eliot Rose wrote: Great! We’re available Monday 11-1, Wednesday 9:30-10:30 and 12-1:30, and Thursday 11-2. Glad to know that the data are getting close to ready to go.   I looked over the contract with our attorneys and I think that’s in pretty good shape, just a few minor issues to discuss. I’ll send an updated versions with those issues flagged tomorrow. I’m also having our technical folks review the documentation you sent so we can let you know if we have any remaining questions on that.   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 1:18 PM To: Eliot Rose Subject: Re: Replica agreement - public records request I think we can just focus the call on the contract and try to close the loop on the outstanding points. The FOIA process is pretty much dictated, so no need to discuss until you would like to. Understand on the term sheet, no worries. Thanks again for all your help on these items. Nick On Wed, Jan 2, 2019 at 1:08 PM Eliot Rose wrote: Thanks! Reviewing everything now and then talking with our legal at 2:30. Definitely going to check in on the term sheet – I understand your concerns but as I understand it that’s a signed agreement and therefore subject to disclosure.   Sending some times for us to talk later today once I get to check in with the team. Is your plan for the call to just loop in the attorneys and focus on the agreement, or should we plan to cover any outstanding issues re: the public records request and documentation as well?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 10:53 AM To: Eliot Rose Subject: Re: Replica agreement - public records request Sorry about that! Just shared. On Wed, Jan 2, 2019 at 12:47 PM Eliot Rose wrote: Hello Nick:   I just tried to open this up and it looks like you shared it with my Metro acct., which isn’t Google-enabled. Can you please share with me at   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Eliot Rose bowden@sidewalklabs.com Accepted: Invitation: Metro + Sidewalk Labs (Contract) @ Mon Jan 7, 2019 1pm - 1:30pm (CST) (eliot.rose@oregonmetro.gov) From: To: Subject: Attachments: Eliot Rose on behalf of bowden@sidewalklabs.com Robert Kirkman; Chris Johnson; Michelle Bellia; Eliot Rose; bbloom@mindengross.com; valerie.eisen@sidewalklabs.com FW: Invitation: Metro + Sidewalk Labs (Contract) @ Mon Jan 7, 2019 1pm - 1:30pm (CST) (eliot.rose@oregonmetro.gov) invite.ics -----Original Appointment----From: bowden@sidewalklabs.com   [mailto:bowden@sidewalklabs.com] Sent: Thursday, January 03, 2019 7:25 AM To: bowden@sidewalklabs.com ; Eliot Rose; bbloom@mindengross.com ; valerie.eisen@sidewalklabs.com Subject: Invitation: Metro + Sidewalk Labs (Contract) @ Mon Jan 7, 2019 1pm - 1:30pm (CST) (eliot.rose@oregonmetro.gov ) When: Monday, January 07, 2019 11:00 AM-11:30 AM (UTC-08:00) Pacific Time (US & Canada). Where: USE HANGOUT CALL-IN NUMBER more details » Metro + Sidewalk Labs (Contract) When  Mon Jan 7, 2019 1pm – 1:30pm Central Time - Chicago   Where  USE HANGOUT CALL-IN NUMBER (map )   Joining info  meet.google.com/urw-ewqm-tye    Or dial: +1 513-401-7245  PIN: 140385#  More phone numbers     Calendar  eliot.rose@oregonmetro.gov   Who  •  bowden@sidewalklabs.com   - organizer   •  eliot.rose@oregonmetro.gov   •  bbloom@mindengross.com   •  valerie.eisen@sidewalklabs.com     Going (eliot.rose@oregonmetro.gov )?   Yes   - Maybe   - No      more options »   Invitation from Google Calendar You are receiving this courtesy email at the account eliot.rose@oregonmetro.gov   because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to modify your RSVP response. Learn More .    From: To: Subject: Date: Eliot Rose Nick Bowden RE: Replica agreement - public records request Wednesday, January 02, 2019 6:10:00 PM Thanks for this update! I want to get the contract signed ASAP as well. I’m hoping that we can stay to that timeline, but we will need to allow for Portland and TriMet to review the contract (at least, a close-to-final version), which they requested in the IGA that we developed with them. We’ve been keeping them updated so I don’t anticipate any major changes. We should discuss timeline when we chat as well.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 1:21 PM To: Eliot Rose Subject: Re: Replica agreement - public records request Just saw this thread. On deployment, we actually just finished an uncalibrated version of Portland last week, which is really cool. Once we get contract hammered out, we can immediately turn to acceptance criteria and validation plan. We will also need to get moving pretty quickly on collecting ground-truth data, but we have a checklist for that process. I think if you and I make it a goal to try and get contract finalized and signed by end of next week, we will still be on track for late March calibrated deployment. On Wed, Jan 2, 2019 at 1:30 PM Eliot Rose wrote: …and I should have added – thanks to you and your team for your patience and thoroughness about reviewing this. I’m checking to see if we have ways of our end of ensuring that all your intended redactions are carried over in the various quoted text.   Also, as I revisit these emails and our discussion of timelines, I’m noting that we’re behind the Q4 2018 validation texting, but it seems from the more recent discussions that we can start on that soon in 2019. Can you confirm that? I know that y’all have a lot of other work in progress.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, January 02, 2019 11:08 AM To: 'Nick Bowden' Subject: RE: Replica agreement - public records request Thanks! Reviewing everything now and then talking with our legal at 2:30. Definitely going to check in on the term sheet – I understand your concerns but as I understand it that’s a signed agreement and therefore subject to disclosure.   Sending some times for us to talk later today once I get to check in with the team. Is your plan for the call to just loop in the attorneys and focus on the agreement, or should we plan to cover any outstanding issues re: the public records request and documentation as well?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 10:53 AM To: Eliot Rose Subject: Re: Replica agreement - public records request Sorry about that! Just shared. On Wed, Jan 2, 2019 at 12:47 PM Eliot Rose wrote: Hello Nick:   I just tried to open this up and it looks like you shared it with my Metro acct., which isn’t Google-enabled. Can you please share with me at   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot - Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 Metro Making a great place From: To: Bcc: Subject: Date: Eliot Rose Nick Bowden 2days@followupthen.com RE: Replica agreement - public records request Wednesday, January 02, 2019 5:47:00 PM Great! We’re available Monday 11-1, Wednesday 9:30-10:30 and 12-1:30, and Thursday 11-2. Glad to know that the data are getting close to ready to go.   I looked over the contract with our attorneys and I think that’s in pretty good shape, just a few minor issues to discuss. I’ll send an updated versions with those issues flagged tomorrow. I’m also having our technical folks review the documentation you sent so we can let you know if we have any remaining questions on that.   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 1:18 PM To: Eliot Rose Subject: Re: Replica agreement - public records request I think we can just focus the call on the contract and try to close the loop on the outstanding points. The FOIA process is pretty much dictated, so no need to discuss until you would like to. Understand on the term sheet, no worries. Thanks again for all your help on these items. Nick On Wed, Jan 2, 2019 at 1:08 PM Eliot Rose wrote: Thanks! Reviewing everything now and then talking with our legal at 2:30. Definitely going to check in on the term sheet – I understand your concerns but as I understand it that’s a signed agreement and therefore subject to disclosure.   Sending some times for us to talk later today once I get to check in with the team. Is your plan for the call to just loop in the attorneys and focus on the agreement, or should we plan to cover any outstanding issues re: the public records request and documentation as well?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 10:53 AM To: Eliot Rose Subject: Re: Replica agreement - public records request Sorry about that! Just shared. On Wed, Jan 2, 2019 at 12:47 PM Eliot Rose wrote: Hello Nick:   I just tried to open this up and it looks like you shared it with my Metro acct., which isn’t Google-enabled. Can you please share with me at ?   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Michelle Bellia; Valerie Eisen; Clifford Higgins RE: Replica agreement - public records request Wednesday, January 02, 2019 5:35:00 PM Hello Nick:   Thanks for turning around your review of our response to the public records request so quickly. I reviewed your suggested redactions and discussed with our attorneys’ office today. I’d encourage you to take one more pass at this before tomorrow (Thursday) COB and make sure that all the redactions you’re requesting are defensible given Oregon law’s definitions of confidential and trade secret (below), and remove those that aren’t. Also, I saw that many of the attachments linked to in the PDF are highlighted—are these attachments that you’re requesting we remove? Happy to hop on a call to discuss any of this tomorrow; I’m available from 8-10 or from 12:30-2:30.   If the requestor pushes back and asks to see the redacted information their request will go before a DA, who may question the redactions as a whole and order us to release a version with all redactions removed if he/she feels they are not well-based in the law. I understand the intent behind many of your proposed redactions given what you’ve told me about how Sidewalk views trade secrets, but some of them seem questionable, especially requesting that we withhold the signed term sheet. Our signed agreements generally tend to be a matter of public record, and this redaction seems likely to be challenged by the requestor. If you’re concerned about the pricing information becoming public record we could just redact the Cost and Cost Certainty sections of the term sheet. The redactions to the text of my emails and the request that we withhold the methodology article that you shared with me, which has been published in a peer-reviewed journal, also seem questionable (assuming that your highlights mean that you want us to withhold that attachment).   Unfortunately, we’ve also got to ask you to flag all associated text that recurs throughout the response. It would also be helpful if you can provide us with a more detailed explanation about why the redactions that you are requesting are necessary – with some of the more established data products we work with high-level descriptions of the methodology and cost, like some of what you’ve flagged for redaction, aren’t always trade secret. Is there something about the unique nature of Replica that makes them a trade secret in this case?   I checked in with the City of Portland today, and I learned that they were also hit with a public records request. Because of the timing of the request—after they approved the IGA, but before they had reviewed the draft agreement—they complied in full, which means that the requestor likely has access to anything that you’ve shared directly with the broader group of agencies that were involved in the conversation, which includes the data sample that you shared with us. In the future can you please flag any documents that you consider to be confidential or trade secret up front?   Thanks, Eliot   Confidential:   192.355(4) Information submitted to a public body in confidence and not otherwise required by law to be submitted, where such information should reasonably be considered confidential, the public body has obliged itself in good faith not to disclose the information, and when the public interest would suffer by the disclosure. Trade secret:   192.345(2) [U]nless the public interest requires disclosure in the particular instance, [t]rade secrets. “Trade secrets,” as used in this section, may include, but are not limited to, any formula, plan, pattern, process, tool, mechanism, compound, procedure, production data, or compilation of information which is not patented, which is known only to certain individuals within an organization and which is used in a business it conducts, having actual or potential commercial value, and which gives its user an opportunity to obtain a business advantage over competitors who do not know or use it.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Replica agreement - public records request Wednesday, January 02, 2019 11:30:00 AM …and I should have added – thanks to you and your team for your patience and thoroughness about reviewing this. I’m checking to see if we have ways of our end of ensuring that all your intended redactions are carried over in the various quoted text.   Also, as I revisit these emails and our discussion of timelines, I’m noting that we’re behind the Q4 2018 validation texting, but it seems from the more recent discussions that we can start on that soon in 2019. Can you confirm that? I know that y’all have a lot of other work in progress.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, January 02, 2019 11:08 AM To: 'Nick Bowden' Subject: RE: Replica agreement - public records request Thanks! Reviewing everything now and then talking with our legal at 2:30. Definitely going to check in on the term sheet – I understand your concerns but as I understand it that’s a signed agreement and therefore subject to disclosure.   Sending some times for us to talk later today once I get to check in with the team. Is your plan for the call to just loop in the attorneys and focus on the agreement, or should we plan to cover any outstanding issues re: the public records request and documentation as well?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 10:53 AM To: Eliot Rose Subject: Re: Replica agreement - public records request Sorry about that! Just shared. On Wed, Jan 2, 2019 at 12:47 PM Eliot Rose wrote: Hello Nick:   I just tried to open this up and it looks like you shared it with my Metro acct., which isn’t Googleenabled. Can you please share with me at ?   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Subject: Date: Eliot Rose Nick Bowden RE: Replica agreement - public records request Wednesday, January 02, 2019 11:08:00 AM Thanks! Reviewing everything now and then talking with our legal at 2:30. Definitely going to check in on the term sheet – I understand your concerns but as I understand it that’s a signed agreement and therefore subject to disclosure.   Sending some times for us to talk later today once I get to check in with the team. Is your plan for the call to just loop in the attorneys and focus on the agreement, or should we plan to cover any outstanding issues re: the public records request and documentation as well?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, January 02, 2019 10:53 AM To: Eliot Rose Subject: Re: Replica agreement - public records request Sorry about that! Just shared. On Wed, Jan 2, 2019 at 12:47 PM Eliot Rose wrote: Hello Nick:   I just tried to open this up and it looks like you shared it with my Metro acct., which isn’t Googleenabled. Can you please share with me at ?   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503?797?1825 Metro Making a great place From: To: Subject: Date: Importance: Eliot Rose Nick Bowden RE: Replica agreement - public records request Wednesday, January 02, 2019 10:47:00 AM High Hello Nick:   I just tried to open this up and it looks like you shared it with my Metro acct., which isn’t Googleenabled. Can you please share with me at ?   Thanks!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, December 31, 2018 1:11 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request SWL_Review - Eliot Rose Portland Emails 398-979... Hi Eliot We had a chance to do a full review of the correspondence and our redactions are attached in the PDF as strikethroughs. Obviously, per FOIA law, all of the redactions are items we consider to be proprietary and if released could cause business harm. In terms of attachments, we do consider our term sheet (which disclosed pricing methodology) and the acceptance criteria proprietary. Post-deployment, you are free to publish the acceptance criteria of course with the data. As you know from seeing the emails, there are a ton of threads and repeats of the same threads. So, we tried to make sure we consistently struck out the same items, but certainly possible we might have missed something. Let us know if you have any questions / issues. Thanks again and sorry for all of the trouble! Happy new year! Nick On Fri, Dec 28, 2018 at 12:25 PM Eliot Rose wrote: Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Bcc: Subject: Date: Eliot Rose Nick Bowden Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson 1day@followupthen.com Re: Metro review of Replica agreement Monday, December 31, 2018 12:49:47 PM Thanks Nick! I'll send out some times to discuss shortly. Metro team, please review this, we'll discuss on Wednesday.  Eliot Rose Technology Strategist, Metro eliot.rose@oregonmetro.gov / 503.797.1825 On Dec 31, 2018, at 11:08 AM, Nick Bowden wrote: Hi Eliot Happy new year to everyone. It might make sense to schedule a short-call to finish the small number of remaining items on the contract later this week, see below. Attaching a few items here: 1. Reviewed contract, there are a very small number of outstanding comments / questions from our side, but we are close to the finish line. 2. Data Disclosure document. This document contains detailed information about data sources and use of said data in the Replica building process. NDAs prevent us from sharing specific vendor names, but this provides a great deal of detail about the general sources and the process. Most importantly, Replica does not handle, process, or store personally identifiable information at any point. Happy to discuss this in more detail over phone if needed. 3. One-page technical workflow. This provides insight into the actual workflow of creating a Replica. 4. Replica Schema. This document should provide better insight into the output metadata as a means of thinking about validation. We obviously consider #2, 3, and 4 confidential and proprietary but want to make sure you all have a complete understanding of how Replica is created.  Thanks. As always, please let me know if you have questions.  Nick On Fri, Dec 28, 2018 at 1:14 PM Eliot Rose wrote: Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread.   We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items.    In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them.  We can provide the Replica data schema and all related metadata as well.  We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know.   Happy Holidays! Nick     On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement   Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Thanks for sending this Eliot.    There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions.   On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this.   I am hoping I will be on the call but wanted to send this just in case.   Thanks. Nick     On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. ·         How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. ·         How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. ·         Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: ·         Export/download the complete dataset for all Replica builds that we receive during the contract term ·         Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term ·         Host Replica data for Metro staff and partner organizations to be able to access ·         Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period ·         Display Replica data or results of analyses based on Replica data on publicfacing websites, documents, and other public materials ·         Apply Replica data in analyses as we see fit, including: o   using Replica data as an input travel in our travel model o   producing maps displaying aggregate Replica data, either alone or in combination with other data sources ·         Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   I’ll send out an invite shortly they will include a line number. Thank you!   On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement   Any chance you all are available from 12-1230 PT on Friday?    On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement   Hi Eliot   Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes.    If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue.    Thanks. Nick       On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement   Thanks, Eliot.    You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one.    If not, we can compare against the original, but thought I would check before we do that.   Thanks. Nick   On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro <(4) SaaSPortland Comparison Metro 20181228.docx> From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson RE: Metro review of Replica agreement Friday, December 28, 2018 11:14:00 AM Thanks Nick! Happy to jump on a call to discuss if that’s helpful. I just set a meeting next week to for us review the documentation that you send, as well as any other open contracting issues.   I understand that you have NDAs in place, but the more specificity you can provide on data sources the better. It’s particularly important to clarify whether Google data was used to building Replica or not. We’ve been assuming that based on our conversations and emails, but in your interview w/ OPB you mentioned that Google data was not used in building Replica. That question will invariably come up as this moves toward approval.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, December 26, 2018 7:54 AM To: Eliot Rose Cc: Valerie Eisen; Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. As an FYI, Valerie is taking the lead on this for us, so removing Marie from the thread. We will be reviewing this week and if required, reach out to schedule a call to discuss potential remaining items. In terms of the additional request: We are happy to provide additional documentation on data sources and methodology. We do have NDAs in place with many of our data providers, which means we can provide a detailed description of the data used, how it's collected, our vendor auditing process, and other specifics, but cannot specifically name the vendors per our agreements with them. We can provide the Replica data schema and all related metadata as well. We will provide these by end of week. Thank you to everyone on your side as we work through this together. Between the public scrutiny and back-and-forth, I am really appreciative of everyone's time and effort. As always, if any additional questions arise, please let me know. Happy Holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Thanks for sending this Eliot. There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions. On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this. I am hoping I will be on the call but wanted to send this just in case. Thanks. Nick On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. · How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. · How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. · Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Michelle Bellia; Valerie Eisen RE: Replica agreement - public records request Friday, December 28, 2018 10:24:00 AM Complete PDF Eliot Rose Emails w attachments.pdf Hello Nick:   Please review the attached version instead. It’s similar to the version I sent yesterday except that it contains the attachments that were part of the emails as links within the PDF, as well as some additional deleted emails at the end that we realized that we had forgotten to include in the previous version.   Apologies for the multiple versions. I had hoped to get you more clarity on which of the attachments we intended to withhold before sending this, but with the folks who can advise on that out for the holidays that would mean giving you less time for review. Our attorney has told me that the draft versions of the agreement are not disclosable under Oregon public records law, but there are several attachments like the early versions of the contract template, acceptance criteria, etc. that are related to the contract but not drafts per se, and I’m not sure whether those are disclosable or not.     Please flag any attachments or text within the emails that you consider to be proprietary using highlighting, strikeout, or comments on the PDF, and send back to us by COB 1/3/19. Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, December 27, 2018 2:47 PM To: 'Nick Bowden' Cc: Michelle Bellia; Valerie Eisen Subject: RE: Replica agreement - public records request Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the emails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Michelle Bellia; Valerie Eisen RE: Replica agreement - public records request Thursday, December 27, 2018 2:47:00 PM REDACTED PDF Eliot Rose Emails.pdf Hello Nick:   Attached are the emails that we are planning to provide in response to the public records request. Can you please let us know if SWL considers any of this trade secret that you would request we withhold by 1/3? That will give us a couple working days to respond to your changes before we respond to the request on 1/7. This does not include attachments, and I’m still checking to see which of the attachments included in these emails our attorneys need to disclose. If there are attachments we intend to disclose I’ll follow up with those for your review, but it might not be til early next week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the emails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Valerie Eisen RE: Replica agreement - public records request Wednesday, December 26, 2018 7:58:00 AM Sounds good. We set an expectation with the reporter that we’d respond by 1/7, so we have a little more time. I think that you can expect our final response to the PRR for review in the early new year. I’m checking to see if I can get it to you sooner so that you can review in parallel with our legal and comms, but I’m still new to this process so I can’t guarantee anything.   -Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 21, 2018 6:37 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Thanks, Eliot! We appreciate it. I will plan to review quickly once received. Have a great weekend and happy holidays! Nick On Fri, Dec 21, 2018 at 1:48 PM Eliot Rose wrote: Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the emails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland OR 97232?2736 503?797?1825 Metro Making a great place From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden; Marie Hlavaty; Valerie Eisen Michelle Bellia; Robert Kirkman; Chris Johnson RE: Metro review of Replica agreement Friday, December 21, 2018 10:19:00 AM SaaSPortland Comparison Metro 20181220.docx Hello all:   Attached is our latest review of the agreement. We cleared up several of the open issues following our conversation with Marie and Valerie, and we flagged issues that are still open. We’re happy to have a call to discuss any of the changes or open issues early in the new year when everyone is back in the office.   Our changes in this version are mostly focused on resolving the conflicts between the usage restrictions and our intended use cases. You’ll also see some edits in there that address privacy – this pilot has been receiving increased scrutiny from the press and our residents following the NYT story about private uses of location data, and these changes will help address any privacy concerns that might come up as this moves forward.    Also, we would like to see more thorough documentation of the Replica data sources, methodology, and dataset prior to signing. The paper that Nick sent was a good general summary of the Replica travel modeling methodology, but I didn’t see any description of the demographic data that are used by Replica, nor of the specific travel and location input data that will be used in our build of the model. This will help us craft avalidation plan, and also make sure that we’re prepared to answer any questions that come up on our end about Replica’s data sources as the agreement is finalized. We’d also like to see documentation and metadata for the Replica data to inform our validation plan. Nick shared a data sample early on from Toronto, but it didn’t contain metadata on the attributes, field coding, etc. Chris, our modeling manager, is reviewing the paper and data sample more thoroughly and may have some additional details to request.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, November 30, 2018 2:53 PM To: 'Nick Bowden'; Marie Hlavaty; Valerie Eisen Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: RE: Metro review of Replica agreement Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Thanks for sending this Eliot. There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions. On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this. I am hoping I will be on the call but wanted to send this just in case. Thanks. Nick On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. · How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. · How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. · Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Michelle Bellia; Valerie Eisen RE: Replica agreement - public records request Friday, December 21, 2018 9:46:00 AM Thanks Nick! I think that we can accommodate your request to review the documents, but we might have to ask you for a pretty short turnaround given that we’re trying to respond to the request promptly while we’re short-staffed over the holidays. I’ll be in touch when I hear more, but it might not be a bad idea to plan on setting aside some time to review on Wednesday/Thursday if you can.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, December 20, 2018 4:03 PM To: Eliot Rose Cc: Michelle Bellia; Valerie Eisen Subject: Re: Replica agreement - public records request Hi Eliot Happy holidays to you all as well. Many of our partners have received FOIA recently. Typically, companies do have a chance to review correspondence to ensure there is nothing included that could be considered proprietary and / or cause the business harm in being released. If possible, we would love to review the correspondence you plan to provide. I know this adds another step for you all and we certainly understand if it’s not possible. There are generally a few things we consider to be proprietary: 1. Any correspondence or documents that disclose our technical process or specific elements of acceptance criteria 2. Any correspondence or documents that specifically refer to other customer use cases 3. Any correspondence or documents that refer specifically to our data vendors by name (not in the general sense) Thank you very much and apologize for the added work in this request. We are looking forward to finalizing the agreement and getting started. Thank you. Nick On Thu, Dec 20, 2018 at 5:34 PM Eliot Rose wrote: Hello SWL team: Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them. Please let us know if you have any questions or concerns. -Eliot Eliot Rose Technology Strategist Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov Metro Making a great place From: To: Cc: Subject: Date: Eliot Rose Nick Bowden; Marie Hlavaty; Valerie Eisen Michelle Bellia Replica agreement - public records request Thursday, December 20, 2018 3:34:00 PM Hello SWL team:   Happy Holidays! We’re preparing a revised draft of the agreement to send to you shortly. In the meantime, this project has been getting some attention, and we’ve received a public records request for my correspondence with SWL as well as any agreements between Metro and SWL. We are planning to disclose all of those emails and attachments, including the signed term sheet, but will withhold the drafts of the agreement that we’re in the process of developing, as well as removing any drafts of the agreement from the e-mails that we disclose where they are included as attachments. However, if the requester petitions for disclosure and the DA orders us to disclose the draft contracts, Metro will disclose them.   Please let us know if you have any questions or concerns. -Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736               503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Nick Bowden RE: Metro review of Replica agreement Friday, December 07, 2018 10:13:00 AM Correct!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, December 07, 2018 10:10 AM To: Eliot Rose Subject: Re: Metro review of Replica agreement That's totally fine. I assume the group is all public sector folks that will be involved and have access to Replica? On Fri, Dec 7, 2018 at 12:05 PM Eliot Rose wrote: I’ve been pulling together the data that we need and working with our partners who collect more detailed data sets. Any objection to me sharing the acceptance criteria doc with one of our regional data working groups?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:40 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Also, forget to attach two items. The first is the most up-to-date template for Acceptance Criteria. This is the format we are using with both NYC and Chicago. Second, I attached the recently approved calibration report for KC Season 1. This will give you an idea of how we report against the acceptance criteria each quarter. On Wed, Nov 28, 2018 at 11:22 AM Nick Bowden wrote: Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Subject: Date: Eliot Rose Nick Bowden RE: Metro review of Replica agreement Friday, December 07, 2018 10:05:00 AM I’ve been pulling together the data that we need and working with our partners who collect more detailed data sets. Any objection to me sharing the acceptance criteria doc with one of our regional data working groups?   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:40 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Also, forget to attach two items. The first is the most up-to-date template for Acceptance Criteria. This is the format we are using with both NYC and Chicago. Second, I attached the recently approved calibration report for KC Season 1. This will give you an idea of how we report against the acceptance criteria each quarter. On Wed, Nov 28, 2018 at 11:22 AM Nick Bowden wrote: Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Bcc: Subject: Date: Eliot Rose Nick Bowden; Marie Hlavaty; Valerie Eisen Michelle Bellia; Robert Kirkman; Chris Johnson 1week@followupthen.com RE: Metro review of Replica agreement Friday, November 30, 2018 2:53:00 PM Sorry to miss you today, Nick. The call was a little disjointed due to technical difficulties om both Metro and SWL’s end, but I think that we got to some more clarity on how we can work in exemptions for our desired use cases. We’re going to come up with an approach and run that by you, continue to deal with smaller changes via markup, and then circle back to discuss remaining open issues.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, November 30, 2018 10:49 AM To: Eliot Rose; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Thanks for sending this Eliot. There is a chance I may not be able to make the call today as I am in the middle of trying to get NYC Replica reports finalized and have been in and out of meetings all day. If for some reason I can't make it, Marie and Valeria can discuss and handle the use restrictions question and contract language side. They are both entirely up to speed on the discussions. On the question of other partner use. Each individual user (re: of agency) agrees to the terms of use upon account creation. The terms of use (which is attached) is a much lighter version of the contract, but does contemplate and restrict things like outside commercial use, etc.. It's a standard TOU for software products. Again, both Marie and Valerie can answer specific questions about this. I am hoping I will be on the call but wanted to send this just in case. Thanks. Nick On Fri, Nov 30, 2018 at 12:15 PM Eliot Rose wrote: Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. · How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. · How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. · Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen RE: Metro review of Replica agreement Friday, November 30, 2018 9:15:00 AM Thanks Nick! Looking forward to chatting with you. There’s plenty for us to discuss, but we think the best path forward is to focus on the big-picture questions below during the call. · How do we make sure that the agreement allows Metro to use the Replica data as we’ve discussed while maintaining the necessary restrictions on the data? We’ve created a list of use cases (below), and we’re happy to answer any questions about these. We’d also like to hear more about the rationale behind some of the restrictions in 4.4, particularly h, i, j, and n. · How are other pilot testers balancing providing access to their partner agencies to get a rich group of user testers with maintaining compliance with the agreement and terms of use? We need to be clear with our partners about the terms under which they will be able to access Replica. · Can you please help us understand SWL’s changes to its own original language in the agreement, for example: Do we no longer need the “Free Service” language because Metro is instead accessing services offered as a free trial? Do the subscription fees in schedule 1 take the place of the “Order Form”? In addition, we’re interested in seeing a draft of the terms of use, as well as more information about the structure and size of the Replica dataset. We don’t necessarily need to discuss that on the call, but I wanted to flag it for you because we’ll need that information to finalize the agreement.   Til soon, Eliot   Metro’s intended use cases: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 10:13 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement I’ll send out an invite shortly they will include a line number. Thank you! On Thu, Nov 29, 2018 at 1:01 PM Eliot Rose wrote: Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Subject: Eliot Rose bowden@sidewalklabs.com Accepted: Invitation: Replica + Portland @ Fri Nov 30, 2018 2pm - 2:30pm (CST) (eliot.rose@oregonmetro.gov) From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen RE: Metro review of Replica agreement Thursday, November 29, 2018 10:01:00 AM Yes, that works for us. Do you want to set up a line, or just call us directly? We can all gather in one room.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 29, 2018 4:52 AM To: Eliot Rose Cc: Chris Johnson; Marie Hlavaty; Michelle Bellia; Robert Kirkman; Valerie Eisen Subject: Re: Metro review of Replica agreement Any chance you all are available from 12-1230 PT on Friday? On Wed, Nov 28, 2018 at 12:58 PM Eliot Rose wrote: Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen RE: Metro review of Replica agreement Wednesday, November 28, 2018 10:58:00 AM Thanks Nick! Friday is pretty open for our team during working West Coast working hours. Want to do 11 PT?   I’ll review your changes later today and let you know tomorrow if there’s anything particular that we want to cover on the call. We’ll track any changes from here on out.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Wednesday, November 28, 2018 9:22 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson; Marie Hlavaty; Valerie Eisen Subject: Re: Metro review of Replica agreement Hi Eliot Attached is a redlined version of the agreement from our legal team (also attached). Some of the items are simple clean-up, others are suggestions, and there are a few open items for discussion. It likely makes sense to find a time for our respective legal teams to connect on the outstanding issues. Can we schedule something for tomorrow or Friday? I don't think it will take longer than 20 minutes. If you do make edits / revisions, would you please track changes as well. That will prevent getting into a version-control issue. Thanks. Nick On Mon, Nov 26, 2018 at 12:14 PM Eliot Rose wrote: I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Michelle Bellia; Robert Kirkman; Chris Johnson RE: Metro review of Replica agreement Monday, November 26, 2018 10:14:00 AM I don’t –it got pretty hectic with all of our overlapping changes and comments—but I can do a document compare too if it helps. Sorry!   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Sunday, November 25, 2018 11:57 AM To: Eliot Rose Cc: Michelle Bellia; Robert Kirkman; Chris Johnson Subject: Re: Metro review of Replica agreement Thanks, Eliot. You don't happen to have a redline version do you? It's difficult to see exactly what all has been modified without one. If not, we can compare against the original, but thought I would check before we do that. Thanks. Nick On Wed, Nov 21, 2018 at 2:40 PM Eliot Rose wrote: Hello Nick: Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.” You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions. You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t? We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria. Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review. Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Bowden Michelle Bellia; Robert Kirkman; Chris Johnson Metro review of Replica agreement Wednesday, November 21, 2018 12:40:00 PM Sidewalk Labs Service Subscription Agreement Replica Metro 20181121.docx Hello Nick:   Attached are our revisions to the Replica contract. We have completed a preliminary read / edit of the document, and attempted to change all references in the Agreement to “Metro” or “SWL.”   You and I had some conversations as we were developing this about the usage restrictions in section 4.4. We had discussed clarifying that Metro’s intended use cases are exempt from the restrictions in that section, but upon further review we decided it would provide more clarity to remove the restrictions in that section that we were concerned with rather than listing broad exemptions to a set of pretty broad restrictions.   You’ve also mentioned that some of the supporting documents included in the agreement (Order Forms, Terms of Use, Documentation) are no longer relevant. Can you please edit the agreement to either include those documents and the associated references where they’re still relevant, and to remove the references where they aren’t?   We also still need to draft acceptance criteria. If you can provide us with some more information about the structure of a quarterly Replica dataset (e.g., size, attributes, field structure) it will help us figure out our validation process and acceptance criteria.   Beyond that, unless Metro or SWL have further substantive changes to the body of the Agreement, the only changes we will make to finalize will be to do a comprehensive editing and formatting review.   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Nick Bowden Chris Johnson; Robert Kirkman RE: Replica Portland: Next Steps Friday, November 09, 2018 2:21:00 PM Thanks for reaching out, Nick.   We’ve got a close-to-final draft of the contract to share; it’s going through one more round of technical review and then we hope to send it middle of next week. In case it helps to have a headsup on what you’ll see in that draft:   Revising this has been a long process because the contract language, particularly the usage restrictions in section 4.4, prevents many of the uses of the Replica data that we’re most interested in and that we’ve previously discussed with you. Our main change is to clarify that the following use cases are exempted from the restrictions in 4.4: · Export/download the complete dataset for all Replica builds that we receive during the contract term · Archive and access (in perpetuity) the full dataset for all Replicas that we receive during the contract term · Host Replica data for Metro staff and partner organizations to be able to access · Publish and/or make publicly available (in perpetuity) data and/or analysis and information derived from the Replica data that was provided to us during the contract period · Display Replica data or results of analyses based on Replica data on public-facing websites, documents, and other public materials · Apply Replica data in analyses as we see fit, including: o using Replica data as an input travel in our travel model o producing maps displaying aggregate Replica data, either alone or in combination with other data sources · Create derivative data products (in perpetuity) i.e., model estimation datasets, from the Replicas that are provided to us during the contract period.   We are also concerned about the broad requirements to “use  commercially  reasonable  efforts  to  prevent  unauthorized  access to or use of Services and Content” (4.3d) and broad restrictions on building similar products in 4.4n. We’ll be looking to SWL to refine those to focus on the specific security cases and derivative products that you’re concerned with.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, November 08, 2018 12:12 PM To: Eliot Rose Subject: Re: Replica Portland: Next Steps Hi Eliot! Just wanted to check and see how this is coming along. Let me know if you have any questions. Thanks. Nick On Thu, Oct 25, 2018 at 12:31 PM Eliot Rose wrote: Thanks Nick! That approach sounds good to me. We’ll work with our attorney to outline those use cases—I don’t think that they will be a surprise to you based on our conversations so far, but we want to make sure that they’re covered under the agreement.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Thursday, October 25, 2018 6:37 AM To: Eliot Rose Cc: Chris Johnson Subject: Re: Replica Portland: Next Steps This reference is a vestige from an older version of the SaaS agreement that didn't get removed from the current version. It was intended to reference the MOU / Term sheet. It's easiest to ignore for now and we will redline out once we get your review. On the use cases you are anticipating, it's likely easiest to include those in your redline explicitly and we will accept the changes upon receipt. Does this work? Thanks. Nick On Wed, Oct 24, 2018 at 7:09 PM Eliot Rose wrote: PS – Can you please provide an example of the Order Forms and Documentation that are referenced in the contract? The language in Section 4.4 appears to prohibit some of the use cases that we’re interested in, but it also allows for exceptions as detailed in those two documents, so it will expedite our review if we can see them too.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, October 22, 2018 12:53 PM To: 'Nick Bowden' Cc: Chris Johnson Subject: RE: Replica Portland: Next Steps Thanks for checking in, Nick. We’ve done both a technical and legal review of the agreement, and want to take one more pass before we get a redlined version back to you. That should give us some time for back-and-forth before Portland and TriMet are ready to enter into an MOU with Metro, which should happen in early November, so that we can get the agreement signed.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, October 22, 2018 11:51 AM To: Eliot Rose Cc: Chris Johnson Subject: Re: Replica Portland: Next Steps Hi Eliot Touching base on the agreement. How are things coming along on your end? We are planning to start working on the uncalibrated Portland Replica sometime late next month. Thanks! Nick On Tue, Oct 2, 2018 at 4:48 PM Nick Bowden wrote: Hi Eliot Our legal has updated the contract (attached) to address some of the issues you notes. Section 7.2 now makes it explicit that you can download or export data at any time. Sections 4.4 and 4.6 are the places where restrictions are identified for data use. On any of these items, any suggestions that you have (not covered by the updates) are welcome. I want to make sure we are aligned with what you have identified. Thanks! Nick On Mon, Sep 24, 2018 at 6:42 PM Eliot Rose wrote: Hello Nick:   We’ve been reviewing the Replica SaaS contract that you sent over, and we have a couple of questions. · It’s important to our team that we be able to download and host the Replica data, and we’ve discussed that before with you and gotten the impression that we’re going to be able to do that through the pilot. The only place we see reference to downloading the data in the SaaS agreement is in the termination clause—is there any language about our ability to download data during the pilot elsewhere in the contract or its exhibits, like in the Order Form and Documentation (which weren’t included in full in the draft that you sent)? · Similarly, where can we find more information on restrictions on the access to the dataset? The contract refers to limitations on the number of users and on queries, but doesn’t detail what those limitations are. · Lastly, is there any information that you can share with us on the technical specs of the Replica database—approximate size for a region like ours, format, etc.? Our enterprise data manager, Robb, is cc’d here and can fill in some of the specifics if you need more information about what we’re looking to know.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, September 17, 2018 8:29 AM To: Eliot Rose Subject: Re: Replica Portland: Next Steps Acceptance Criteria Template attached. Some of the numbers can't actually be determined until we do a variance analysis on the ground truth data. As an example, if the natural variance in freeway loop detectors is 18% (actually number from Kansas City), then the threshold for that particular metric of course can't actually be below 18%. Let me know if you have any questions. In terms of to-do's / order of work: We should be able to work the SaaS agreement mark ups / acceptance criteria in parallel. The SaaS mark ups will likely come from legal, which acceptance agreement will come from you / others at Metro. Assuming we can get everything finalized in the next 45 days, we will start the uncalibrated build in early November-ish. The only thing we will need to do this is GTFS for all transit. Let's say early December is when we will start the data request from local agencies with a plan to have it collected by mid-Jan. Normalized by early feb. That gives us two months to do calibration runs, with final, fully calibrated Replica done by end of March. Thanks! Nick On Thu, Sep 13, 2018 at 11:10 PM Eliot Rose wrote: Thanks Nick! I’ll get you a data sample shortly. Looking forward to seeing the acceptance criteria.   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Monday, September 10, 2018 12:55 PM To: Eliot Rose Subject: Re: Replica Portland: Next Steps Hi Eliot Thanks for sending this and sorry for delayed response. Answers inline below: On Tue, Sep 4, 2018 at 5:57 PM Eliot Rose wrote: Hello Nick:   I’ve been moving forward getting an IGA in place on our side to fund the Replica pilot, but I haven’t gotten the chance to follow up on your requests yet. Here’s a quick update and some questions to help keep this moving. 1. Contract: I looked into it and we don’t have a standard SAS contract template, nor even a comparable contract that we feel like we can base this one on, so we should start from your sample contract. Attached is the latest version I have of that; is this the right one to be using? I've attached the most recent version, I *think it's the same as the one attached but did not go line by line. 2. Acceptance criteria: same question as above; is this the right version to be working off of? No, we have an updated version, it continues to get modified as learn new things with each effort. I am working to finish today, and then will send your way. 3. Sample data: working on this right now. Transit data is pretty straightforward, but in our region there’s no single regional source of traffic data—Metro compiles some, but for the complete set we’ll have to pull together data from 3 counties and the city of Portland. Do you want me to send you the different data sets as I access them so that you can think with us about how to best mash them up in this process, or just get you the regional dataset? Any one sample is fine for now. We will ultimately handled the normalizing or meshing, we don't require agencies to do any work, just send us what they have. I am just trying to get a feel for what's out there at this point. Anything you have is great.   Also, we continue to be interested in any documentation you’ve developed about the data sources and methodology used in Replica. You sent a research paper that covers the use of cell data well, but doesn’t discuss other sources in detail. I’m particularly interested in hearing more about how Replica incorporates Streetlight data, and how the end product differs from Streetlight. I have a sense of that from speaking to you, but some of the folks that are going to be approving the contract are well aware of Streetlight, and it will help to be able to articulate to them what the additional value add of Replica is. We are finalizing the data disclosure document for Kansas City right now and once complete I will send your way. We actually use a combination of streelight, safegraph, cell companies, consumer marketing data, google data, and census data. So, the first component of differentiation is in the composite of data across multiple sources which provides a much larger samples and makes de-biasing the data much easier. Secondly, we produce a full-scale model, that represents all trips vs. aggregate counts with small factoring, which is a pretty significant effort / difference. Thirdly, trip purpose (why people travel) but household characteristics are associated with every trip, which means who is traveling is equally important as how and why people travel. Finally, and most important in my opinion, is we calibrate Replica to world conditions to ensure it can be used in policy making vs. just a source of additional data.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, August 21, 2018 3:38 PM To: 'Nick Bowden'; Chris Johnson; Robert Kirkman Subject: RE: Replica Portland: Next Steps Hello Nick:   Sorry to be a little slow in responding to this; I was out on vacation for a long weekend. I think that you may have forgotten to attach the updated acceptance criteria. I’m attaching the most recent version I have for reference.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Nick Bowden [mailto:bowden@sidewalklabs.com] Sent: Friday, August 17, 2018 1:46 PM To: Eliot Rose; Chris Johnson; Robert Kirkman Subject: Replica Portland: Next Steps Good Afternoon Thanks for taking time to talk this week. We are very excited to get started. Here are the follow up items from our call: 1. Contract Form / Process: You guys are going to check on using your own standard agreement vs. using our standard agreement. Once determined, we can start the procurement / legal review process. 2. Acceptance Criteria Document: Updated Template Version Attached. It takes a little localization, but by and large, about 90% consistent across regions. 3. Data Collection Process: I've attached a data requirements checklist to the email as well. This provides a run down of all the data required across all participating agencies. Again, the gathering of this will be handled in a very programatic way, so don't fret the process at this point. 4. Sample Date: If you have a sample set of traffic and transit counts you can share in their raw format, that would be great. Please let me know if I forgot anything. Have a great weekend. Nick From: To: Subject: Date: Martin, Kevin Eliot Rose [External sender]Re: Replica - custom geographies Wednesday, October 30, 2019 11:02:25 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. ZGF thing was fine, but you did not miss much. My next couple of weeks are fun-filled. How about Wed 11/20? Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Wednesday, October 30, 2019 3:35 PM To: Martin, Kevin Subject: RE: Replica - custom geographies Oh, and thanks for the response on the ZGF event. I’m glad I was able to duck out… a couple fires to put out this afternoon. I hope it’s useful! Eliot Rose Technology Strategist Metro From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Wednesday, October 30, 2019 1:16 PM To: Eliot Rose Subject: [External sender]Re: Replica - custom geographies CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Bet you're going to get a big list. Hey, question for you. Can you get your hands on the Metro/PSU PORTAL IGA? Another question. Ship Ahoy soon? Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Kerr, Michael Sent: Wednesday, October 30, 2019 1:14 PM To: Martin, Kevin Cc: Eliot Rose ; Gilligan, Mike ; McHugh, Tim ; Joe Broach Subject: Re: Replica - custom geographies Have an inquiry out to a bunch of pbot planners (including Ning) with a request to get back to me ASAP. Believe Ning derives city specific TAZs from Metro TAZs, but am pretty sure he was fine with the latter. I’ll get back to everyone soon, Mike Sent from my iPhone  On Oct 30, 2019, at 3:29 PM, Martin, Kevin wrote: Mike, what geographies is Ning using for transportation modeling in PBOT? TAZs? Same TAZs as Metro (there are several versions, or used to be)? That's my only thought, that we'd want alignment there. Seems like we could use the census geographies to aggregate to other areas of interest, deal with any slight differences. Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Tuesday, October 29, 2019 5:50 PM To: Kerr, Michael ; Gilligan, Mike ; Martin, Kevin ; McHugh, Tim Cc: Joe Broach Subject: Replica - custom geographies Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Martin, Kevin Eliot Rose [External sender]Re: Replica - custom geographies Wednesday, October 30, 2019 1:15:14 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Bet you're going to get a big list. Hey, question for you. Can you get your hands on the Metro/PSU PORTAL IGA? Another question. Ship Ahoy soon? Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Kerr, Michael Sent: Wednesday, October 30, 2019 1:14 PM To: Martin, Kevin Cc: Eliot Rose ; Gilligan, Mike ; McHugh, Tim ; Joe Broach Subject: Re: Replica - custom geographies Have an inquiry out to a bunch of pbot planners (including Ning) with a request to get back to me ASAP. Believe Ning derives city specific TAZs from Metro TAZs, but am pretty sure he was fine with the latter. I’ll get back to everyone soon, Mike Sent from my iPhone  On Oct 30, 2019, at 3:29 PM, Martin, Kevin wrote: Mike, what geographies is Ning using for transportation modeling in PBOT? TAZs? Same TAZs as Metro (there are several versions, or used to be)? That's my only thought, that we'd want alignment there. Seems like we could use the census geographies to aggregate to other areas of interest, deal with any slight differences. Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Tuesday, October 29, 2019 5:50 PM To: Kerr, Michael ; Gilligan, Mike ; Martin, Kevin ; McHugh, Tim Cc: Joe Broach Subject: Replica - custom geographies Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Kerr, Michael Martin, Kevin Eliot Rose; Gilligan, Mike; McHugh, Tim; Joe Broach [External sender]Re: Replica - custom geographies Wednesday, October 30, 2019 1:13:18 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Have an inquiry out to a bunch of pbot planners (including Ning) with a request to get back to me ASAP. Believe Ning derives city specific TAZs from Metro TAZs, but am pretty sure he was fine with the latter. I’ll get back to everyone soon, Mike Sent from my iPhone On Oct 30, 2019, at 3:29 PM, Martin, Kevin wrote:  Mike, what geographies is Ning using for transportation modeling in PBOT? TAZs? Same TAZs as Metro (there are several versions, or used to be)? That's my only thought, that we'd want alignment there. Seems like we could use the census geographies to aggregate to other areas of interest, deal with any slight differences. Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Tuesday, October 29, 2019 5:50 PM To: Kerr, Michael ; Gilligan, Mike ; Martin, Kevin ; McHugh, Tim Cc: Joe Broach Subject: Replica - custom geographies Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Martin, Kevin Eliot Rose; Kerr, Michael; Gilligan, Mike; McHugh, Tim Joe Broach [External sender]Re: Replica - custom geographies Wednesday, October 30, 2019 12:28:52 PM CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Mike, what geographies is Ning using for transportation modeling in PBOT? TAZs? Same TAZs as Metro (there are several versions, or used to be)? That's my only thought, that we'd want alignment there. Seems like we could use the census geographies to aggregate to other areas of interest, deal with any slight differences. Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Tuesday, October 29, 2019 5:50 PM To: Kerr, Michael ; Gilligan, Mike ; Martin, Kevin ; McHugh, Tim Cc: Joe Broach Subject: Replica - custom geographies Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503?797?1825 Metro Making a great place From: Martin Kevin To: Rgse Subject: Tentative: Replica pilot: validation From: To: Subject: Martin, Kevin Kerr, Michael; Eliot Rose Sidewalk Labs discussion Mike and I are both dealing with kids and school closures, so let's do this by phone! Adding the Skype info. Eliot, let us know if that works for you. K -------------Looks like Mike is booked until 3:30. Eliot, can you make this work? Mike, mind grabbing a room? Or can we fit in your fancy new office? Digging that 13th floor.   Thanks! Kevin ......................................................................................................................................... Join online meeting Join by Phone 7209882090,,71623010# (Toll Number) 8666708282,,71623010# (Toll-Free Number) Find a local number Participant code: 71623010 Forgot your dial-in PIN? ......................................................................................................................................... From: To: Subject: Date: Martin, Kevin Eliot Rose Re: Replica pilot: validation Monday, February 04, 2019 3:27:50 PM Can’t wait! Kevin Martin (he/him) Smart City PDX City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com   From: Eliot Rose Sent: Monday, February 4, 2019 3:06 PM To: Martin, Kevin Subject: RE: Replica pilot: validation   Catch some air out there!   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Monday, February 04, 2019 2:58 PM To: Eliot Rose Subject: Tentative: Replica pilot: validation When: Thursday, February 07, 2019 11:00 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro: Chris' office / (605) 472-5418, Access code: 616836     I'll be out (Mt. Bachelor), but holding on my calendar to remind me to follow up with you all after the meeting. Thanks Eliot! Kevin   From: To: Subject: Martin, Kevin Eliot Rose Tentative: Replica pilot: validation I'll be out (Mt. Bachelor), but holding on my calendar to remind me to follow up with you all after the meeting. Thanks Eliot! Kevin From: To: Subject: Start: End: Location: Martin, Kevin Kerr, Michael; Eliot Rose Sidewalk Labs discussion Tuesday, February 05, 2019 3:30:00 PM Tuesday, February 05, 2019 4:30:00 PM Congress Center, 13th floor, Mike"s fancy new office Looks like Mike is booked until 3:30. Eliot, can you make this work? Mike, mind grabbing a room? Or can we fit in your fancy new office? Digging that 13th floor.   Thanks! Kevin From: To: Subject: Martin, Kevin Kerr, Michael; Eliot Rose Sidewalk Labs discussion Looks like Mike is booked until 3:30. Eliot, can you make this work? Mike, mind grabbing a room? Or can we fit in your fancy new office? Digging that 13th floor. Thanks! Kevin From: To: Subject: Date: Martin, Kevin Eliot Rose; Kerr, Michael Re: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Tuesday, January 29, 2019 12:53:59 AM Great, all good points below. How about later next Tuesday 2/5, 3 or 3:30pm? If that works, do you two have a preference of venue? Congress Center and Metro are both close to my ride home, so happy to visit either! kevin martin (he/him)  smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com From: Eliot Rose Sent: Monday, January 28, 2019 3:14:33 PM To: Kerr, Michael; Martin, Kevin Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Great—I love talking with you guys about this stuff. Next week is pretty open for me.   Agree that it’s not necessary to invite John, but it does seem like we need to make sure we have a plan to get a consistent message out to other PIOs and staff who might field questions before we get any other tough ones. I’m getting TriMet’s PIO up to speed this week, and y’all should feel free to forward the points/memo I’ve sent to any Smart Cities Team members or other CoP staff who might get questions. Please let me know if I can help there.   Can you also please ask CoP comms staff give Metro (myself or Nick Christensen, NickC@oregonmetro.gov) the heads-up if they get further inquiries? Thanks for being on the front lines!   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 1:52 PM To: Martin, Kevin; Eliot Rose Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Let’s get ourselves together and aligned first.  I can make time for this next week, no problem.       Agreed that Eliot’s points are dead on.  Now it’s a matter of educating and sharing concrete examples of how what we do today isn’t all that different.  To the consent issue… it’s a point we’ve made in interviews (that Cities are being held to task for an issue that needs to get resolved Federally), but we definitely need to speak up about how we (Cities) are stepping into the void and doing everything we can to protect people’s privacy.   Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Martin, Kevin Sent: Monday, January 28, 2019 1:45 PM To: Eliot Rose ; Kerr, Michael Subject: Re: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Well said.   I think we need to get in a room sometime in the next few weeks and dig in. Repeating myself, but it’s going to be critical that we’re all on the same page with how w address questions and concerns around SL, and — on the City side — how it overlaps with our Privacy Resolution. Clearly all of this has struck a media nerve.   Think we can find an hour next week or the week after? Eliot, I can check Mike’s schedule and float a couple of times? Mike, do we need someone Ike John Brady in that room, or should we talk first?   Thanks, Kevin   kevin martin (he/him) smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com   From: Eliot Rose Sent: Monday, January 28, 2019 12:58 PM To: Kerr, Michael; Martin, Kevin Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Thanks - I'll pass this around to folks on our end. I think that the response to the add'l concern that you/the article raised about reverseengineering is that we get pitched on other products derived from cell data all the time and that the method that SWL is using - simulating travel based on the data - is a more compelling way to protect privacy. It's what we've used for years to protect the confidentiality of our travel survey respondents. This article makes it sound like Replica is the first to offer any of this data and the first time we've considered any of these issues. Seems like we need to emphasize our experience here and how it's led us to work w/ Replica. The issue that the article (and Doug from WSJ) raises about whether the data that informs Replica has been obtained with consent is one I want to dig into further. I feel like that's a real concern, and that it's also out of the league of local governments to address. The way we discussed it with WSJ is that we hear concerns about privacy from our citizens, and at the same time we hear a very strong demand for us to use a data-driven approach to make the streets safer. This pilot with Replica is an opportunity to understand if those goals conflict and if so, how we can balnce those competing concerns. Eliot Rose Technology Strategist Metro -----Original Message----From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 8:48 AM To: Eliot Rose; Martin, Kevin Subject: FW: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones The story is evolving... https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replicacellphone-data/ This one is more in line with what I'd imagine the WSJ is writing and, while the focus is primarily on SL, it's making cities look as if they're blindly entering into these agreements. To add to that, this article raises a new (but not unpredictable) concern that if SL can disaggregate and synthesize, why couldn't someone else un-synthesize and re-aggregate. Look forward to piecing together a plan for proactively working through these concerns. Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg -----Original Message----From: Kerr, Michael Sent: Monday, January 28, 2019 6:33 AM To: Kerr, Michael Subject: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Sent from my iPhone From: To: Subject: Date: Martin, Kevin Eliot Rose; Kerr, Michael Re: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Monday, January 28, 2019 1:45:07 PM Well said. I think we need to get in a room sometime in the next few weeks and dig in. Repeating myself, but it’s going to be critical that we’re all on the same page with how w address questions and concerns around SL, and — on the City side — how it overlaps with our Privacy Resolution. Clearly all of this has struck a media nerve. Think we can find an hour next week or the week after? Eliot, I can check Mike’s schedule and float a couple of times? Mike, do we need someone Ike John Brady in that room, or should we talk first? Thanks, Kevin kevin martin (he/him) smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com   From: Eliot Rose Sent: Monday, January 28, 2019 12:58 PM To: Kerr, Michael; Martin, Kevin Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Thanks - I'll pass this around to folks on our end. I think that the response to the add'l concern that you/the article raised about reverse-engineering is that we get pitched on other products derived from cell data all the time and that the method that SWL is using - simulating travel based on the data - is a more compelling way to protect privacy. It's what we've used for years to protect the confidentiality of our travel survey respondents. This article makes it sound like Replica is the first to offer any of this data and the first time we've considered any of these issues. Seems like we need to emphasize our experience here and how it's led us to work w/ Replica. The issue that the article (and Doug from WSJ) raises about whether the data that informs Replica has been obtained with consent is one I want to dig into further. I feel like that's a real concern, and that it's also out of the league of local governments to address. The way we discussed it with WSJ is that we hear concerns about privacy from our citizens, and at the same time we hear a very strong demand for us to use a data-driven approach to make the streets safer. This pilot with Replica is an opportunity to understand if those goals conflict and if so, how we can balnce those competing concerns. Eliot Rose Technology Strategist Metro -----Original Message----From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 8:48 AM To: Eliot Rose; Martin, Kevin Subject: FW: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones The story is evolving... https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replica-cellphone-data/ This one is more in line with what I'd imagine the WSJ is writing and, while the focus is primarily on SL, it's making cities look as if they're blindly entering into these agreements. To add to that, this article raises a new (but not unpredictable) concern that if SL can disaggregate and synthesize, why couldn't someone else un-synthesize and re-aggregate. Look forward to piecing together a plan for proactively working through these concerns. Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg -----Original Message----From: Kerr, Michael Sent: Monday, January 28, 2019 6:33 AM To: Kerr, Michael Subject: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Sent from my iPhone From: To: Cc: Subject: Date: Hesse, Eric Kerr, Michael; Pearce, Art; Siegel, Noah; Hurley, Peter; Eliot Rose; Martin, Kevin Patton, Jeramy; Warner, Chris RE: Sidewalk Labs - Council Wednesday, December 12, 2018 10:46:10 AM Excellent news!  Thanks for your expert shepherding skills, Mike!   ............... Eric Hesse Supervising Planner Policy Innovation + Regional Collaboration Pronouns: He/Him Portland Bureau of Transportation 1001 SW 5th Avenue, Suite 500 Portland, OR 97204 Phone: 503.823.4590 eric.hesse@portlandoregon.gov www.portlandoregon.gov/transportation twitter   facebook   instagram   publicalerts       The City of Portland ensures meaningful access to city programs, services, and activities to comply with Civil Rights Title VI and ADA Title II laws and reasonably provides: translation, interpretation, modifications, accommodations, alternative formats, auxiliary aids and services. To request these services, contact 503-823-5185, City TTY 503-823-6868, Relay Service: 711.   From: Kerr, Michael Sent: Wednesday, December 12, 2018 10:33 AM To: Pearce, Art ; Siegel, Noah ; Hesse, Eric ; Hurley, Peter ; Eliot Rose ; Martin, Kevin Cc: Patton, Jeramy ; Warner, Chris Subject: Sidewalk Labs - Council   All,   FYI – if you didn’t see it, Council (all members) just passed our Sidewalk Labs Ordinance.       Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: To: Subject: Date: Martin, Kevin Eliot Rose; Kerr, Michael Fw: Sidewalk Labs - Replica Wednesday, May 29, 2019 8:27:38 AM We're getting a few media inquiries related to the Geekwire article yesterday. Where do you two want these directed? Mike, assume John Brady is still the point of contact at PBOT? Thanks! Kevin Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. Begin forwarded message: From: Sarah Wray Date: May 29, 2019 at 02:23:31 PDT To: "christine.llobregat@portlandoregon.gov" Subject: Sidewalk Labs - Replica Hi Christine I am an editor at Smart Cities World – I saw this report https://www.geekwire.com/2019/portland-quietly-launches-mobile-location-dataproject-alphabets-controversial-sidewalk-labs/ and I wanted to check the facts directly. The article notes that Smart City PDX does not oversee the Replica project, but who does within Portland? Could you confirm the details and scope of this pilot or point/introduce me to someone who can? Thanks! Sarah From: To: Cc: Subject: Date: Attachments: Martin, Kevin Eliot Rose Kendrick, Christine Re: Replica validation: meeting 2 Thursday, April 25, 2019 2:23:57 PM DRAFT Replica Acceptance Criteria v2 notes.xlsx Hey Eliot, Apologies for the lack of response. It's been nutty. Copying Christine, she's going to be my lead on validation. You are in good hands. I'll continue to be involved, but want someone who can be more reliably involved, and brings the data expertise to the table. Thanks! Kevin From: Eliot Rose Sent: Wednesday, April 24, 2019 4:20 PM To: Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: RE: Replica validation: meeting 2   Hello all:   I haven’t received any edits on the Replica validation criteria. I’d like to share our draft with Sidewalk Labs in advance of next Thursday’s kickoff so that we can discuss them on the call. Please send me your edits or confirm that you don’t have any changes by the end of the week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, April 11, 2019 4:24 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Date: Attachments: Martin, Kevin Eliot Rose; Kerr, Michael Fw: ACLU briefing on Sidewalk Labs Replica pilot/Privacy Principles Tuesday, February 26, 2019 11:31:45 PM replica_summary_feb_2019.pdf pip_resolution_DRAFT_190226.pdf FYI. I'll keep you posted, loop you in once we get a response. K From: Martin, Kevin Sent: Tuesday, February 26, 2019 11:29 PM To: jcarson@aclu-or.org Subject: ACLU briefing on Sidewalk Labs Replica pilot/Privacy Principles Jann, It's be a while... we last spoke over a year ago as PBOT prepared to launch a pilot of streetlight-based sensors to count traffic/bikes/peds, your feedback was valuable to shaping that project. We are hoping to do a similar briefing regarding a new project that Metro, City of Portland and TriMet have partnered on to use Sidewalk Lab's Replica software to better understand travel patterns in the region (a significant data need). We'd also like to use the time to discuss a Privacy and Information Protection Resolution that we will be bringing to City Council in May. Is it possible that we could get a bit of time in the next few weeks to discuss? Happy to coordinate schedules on our side if you can float a few dates and times that work for you. A brief description of the Replica project is attached. OPB did a story back in December as well: https://www.opb.org/news/article/cellphone-location-data-portland-google-privacy/. And I've also attached a draft of the Privacy and Information Protection Resolution. Let me know. Hope all is well at ACLU! Thanks, Kevin Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: To: Subject: Date: Attachments: Chris Johnson Eliot Rose; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim RE: Replica pilot: validation Thursday, February 07, 2019 8:55:20 AM DRAFT Replica Acceptance Criteria.xlsx Kate_Validation_Report_August2017.pdf fhwa model validation handbook.pdf Hi allAttached are some draft validation criteria that I plan to walk through during today’s call. I’ve also attached some supporting documentation that will give you some context of how we approach validation in the travel modeling realm. I’m looking forward to this morning’s discussion. Thanks! -Chris     -----Original Appointment----From: Eliot Rose Sent: Monday, February 4, 2019 11:25 AM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim Subject: Replica pilot: validation When: Thursday, February 7, 2019 11:00 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro: Chris' office / (605) 472-5418, Access code: 616836 Agenda: · Introductions (5 min.) · Overview of acceptance process: goals, responsibilities, timeline (Eliot, 10 min.) · Review and discuss draft validation criteria (Chris, 40 min.) · Next steps (5 min.) From: To: Subject: Date: Kerr, Michael Martin, Kevin; Eliot Rose RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Monday, January 28, 2019 1:52:21 PM Let’s get ourselves together and aligned first.  I can make time for this next week, no problem.       Agreed that Eliot’s points are dead on.  Now it’s a matter of educating and sharing concrete examples of how what we do today isn’t all that different.  To the consent issue… it’s a point we’ve made in interviews (that Cities are being held to task for an issue that needs to get resolved Federally), but we definitely need to speak up about how we (Cities) are stepping into the void and doing everything we can to protect people’s privacy.   Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Martin, Kevin Sent: Monday, January 28, 2019 1:45 PM To: Eliot Rose ; Kerr, Michael Subject: Re: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Well said.   I think we need to get in a room sometime in the next few weeks and dig in. Repeating myself, but it’s going to be critical that we’re all on the same page with how w address questions and concerns around SL, and — on the City side — how it overlaps with our Privacy Resolution. Clearly all of this has struck a media nerve.   Think we can find an hour next week or the week after? Eliot, I can check Mike’s schedule and float a couple of times? Mike, do we need someone Ike John Brady in that room, or should we talk first?   Thanks, Kevin   kevin martin (he/him) smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com   From: Eliot Rose Sent: Monday, January 28, 2019 12:58 PM To: Kerr, Michael; Martin, Kevin Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Thanks - I'll pass this around to folks on our end. I think that the response to the add'l concern that you/the article raised about reverseengineering is that we get pitched on other products derived from cell data all the time and that the method that SWL is using - simulating travel based on the data - is a more compelling way to protect privacy. It's what we've used for years to protect the confidentiality of our travel survey respondents. This article makes it sound like Replica is the first to offer any of this data and the first time we've considered any of these issues. Seems like we need to emphasize our experience here and how it's led us to work w/ Replica. The issue that the article (and Doug from WSJ) raises about whether the data that informs Replica has been obtained with consent is one I want to dig into further. I feel like that's a real concern, and that it's also out of the league of local governments to address. The way we discussed it with WSJ is that we hear concerns about privacy from our citizens, and at the same time we hear a very strong demand for us to use a data-driven approach to make the streets safer. This pilot with Replica is an opportunity to understand if those goals conflict and if so, how we can balnce those competing concerns. Eliot Rose Technology Strategist Metro -----Original Message----From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 8:48 AM To: Eliot Rose; Martin, Kevin Subject: FW: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones The story is evolving... https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replicacellphone-data/ This one is more in line with what I'd imagine the WSJ is writing and, while the focus is primarily on SL, it's making cities look as if they're blindly entering into these agreements. To add to that, this article raises a new (but not unpredictable) concern that if SL can disaggregate and synthesize, why couldn't someone else un-synthesize and re-aggregate. Look forward to piecing together a plan for proactively working through these concerns. Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg -----Original Message----From: Kerr, Michael Sent: Monday, January 28, 2019 6:33 AM To: Kerr, Michael Subject: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Sent from my iPhone From: To: Subject: Date: Kerr, Michael Eliot Rose; Martin, Kevin FW: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Monday, January 28, 2019 8:47:42 AM The story is evolving... https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replica-cellphone-data/ This one is more in line with what I'd imagine the WSJ is writing and, while the focus is primarily on SL, it's making cities look as if they're blindly entering into these agreements.  To add to that, this article raises a new (but not unpredictable) concern that if SL can disaggregate and synthesize, why couldn't someone else un-synthesize and re-aggregate. Look forward to piecing together a plan for proactively working through these concerns.    Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-8235185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg -----Original Message----From: Kerr, Michael Sent: Monday, January 28, 2019 6:33 AM To: Kerr, Michael Subject: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Sent from my iPhone From: To: Cc: Subject: Date: Martin, Kevin Eliot Rose Kerr, Michael Re: Meet w/ privacy advocates re: Replica? Friday, January 25, 2019 5:12:37 PM Thanks Eliot. I think it would be good for us to meet and strategize. We're anticipating more questions related to the SL agreement as we start talking to community groups/media about the privacy principles resolution we are bring to City Council on 2/27 (or early March.... Mayor may take a vacation and force us to push the date a week or two). Imagine you two already have a good set of responses drafted because of the media frenzy a while back (maybe this memo is it)?  But I think it would be worth huddling to discuss, and then -- yes -- maybe some proactive outreach to community groups, other organizations. I want everyone involved in the privacy conversations on the City side to go into all rooms prepared to respond to questions about SL, and I want them to be saying the same thing you all are saying. Hector and Christine were caught a bit off guard by the number of questions when they were at PNCA last weekend. Absolutely do not want that happening in front of Council, especially with new Council members who were not there for the discussion back in December. How's that sound? Should we include comms folks? I will be at TAO, we can chat about this then also! Kevin kevin martin (he/him)  smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com From: Eliot Rose Sent: Friday, January 25, 2019 1:53:23 PM To: Martin, Kevin Cc: Kerr, Michael Subject: Meet w/ privacy advocates re: Replica?   Hello Kevin:   I heard from Mike that the Smart City PDX heard some concerns from advocates about the Replica project. Would it help at all for us to meet individually with some of those organizations to try and get out in front of their concerns? Our gov’t affairs team suggested that when I briefed our council on this, but I defer to you on what’s a good idea—I imagine that you have a pretty good idea from the sensor conversation about how that talk would go.   In case it helps, I’m attaching a memo we prepared for our Council to help prep them for any media inquiries or concerned citizens. Happy to talk more at the TAO meeting on Monday if you’re planning to be there.   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Kerr, Michael Eliot Rose; Martin, Kevin RE: Replica update Thursday, November 29, 2018 12:17:14 PM I have the budget (full $154K) NOW and can transfer funding once the IGA gets through Council.    As for Council, I can’t say when we’d actually get called given we’re on the Consent agenda.  May know more by the end of today as to whether or not there’s significant interest – which would indicate that there’s a good chance that I’ll get called.  So, for now, plan on coming for as long as you can (bring your laptop – do work while there).  I’ll circle back tomorrow.     Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Eliot Rose Sent: Tuesday, November 27, 2018 4:01 PM To: Kerr, Michael ; Martin, Kevin Subject: Replica update Hello Michael and Kevin:   Happy Thanksgiving! Hope y’all had a restful break.   I’ve been fielding a few questions from TriMet re: the Replica IGA, including one that relates to your payment schedule: when are you going to have funds available to put toward this project? TriMet has asked whether they can pick up the payments during the 2nd half of 2019 because that’s when they’ll have budget available. That means that we’d need PBOT to be able to make the whole April payment. See the proposed revisions to the payment schedule below. Does that work for you? If not, we can look into altering the schedule.   Payment date (approx.) Payment amount Responsibility (original) Responsibility (revised) January 2019 April 2019 $152,433.33 $101.622.22 July 2019 $101.622.22 October 2019 $101.622.22 Metro to pay in full Portland and TriMet each to reimburse Metro for $50,811.11 Portland and TriMet each to reimburse Metro for $50,811.11 Portland and TriMet each to reimburse Metro for $50,811.11 Metro to pay in full Portland to reimburse Metro in full Portland and TriMet each to reimburse Metro for $50,811.11 TriMet to reimburse Metro in full   TriMet also requested a few changes to clarify the validation and acceptance process. Shouldn’t be anything controversial; I think that it’ll be good to build some additional clarity into that. I’ll share with you once we make those revisions.   Finally, what time is the Replica item scheduled for the 12/5 Council meeting? I have the whole 9:3012 slot blocked off on my calendar, but I have a request to make a conflicting meeting, and I imagine you don’t need me there for the whole time.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, November 19, 2018 4:20 PM To: 'Kerr, Michael'; Martin, Kevin Subject: RE: Hold your calendars All I have drafted is the attached, which you’ve seen before. If there are specific points you’re trying to hit let me know and I’m happy to give you some thoughts about how I’d approach them.   Don’t forget to send me the time so I can be there!   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, November 19, 2018 9:55 AM To: Eliot Rose; Martin, Kevin Subject: RE: Hold your calendars Sounds good and thanks to both of you for your willingness to support (if/as needed).   Eliot, do you happen to have talking points that I build off of for my pre-submission to Council?    Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Eliot Rose Sent: Wednesday, November 14, 2018 9:06 AM To: Martin, Kevin ; Kerr, Michael Subject: RE: Hold your calendars I can probably be there, calendar is pretty open that day. What time, and do you need me to prep anything?   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Friday, November 09, 2018 7:16 PM To: Kerr, Michael; Eliot Rose Subject: Re: Hold your calendars I’ll be there. Love consent agenda! kevin martin smart city pdx/ tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com From: Kerr, Michael Sent: Friday, November 9, 2018 6:15 PM To: Martin, Kevin; eliot.rose@oregonmetro.gov Subject: Hold your calendars   We are officially on the 12/5 city council schedule for sidewalk labs (consent agenda). Flagging this for the two of you, in particular, due to the HIGH likelihood that I get called to speak. Will need all the technical support I can get and having the likes of Smart City PDX and Metro by my side would go a long way! Let me know if you happen to have time to sit around and support this smart cities push, Mike Sent from my iPhone From: To: Bcc: Subject: Date: Attachments: Eliot Rose Kerr, Michael; Gilligan, Mike; McHugh, Tim; Martin, Kevin 5days@followupthen.com Scheduling Replica visit, 12/16-12/18 Friday, November 08, 2019 10:29:00 AM External senderPortland Replica Use Cases.msg Hello all:   We nailed down some dates with Replica when they can visit Portland and offer training sessions. They’ll be in town 12/16 and 12/17. They would like to schedule a 3-4 hour training session with the likely Replica “power users” from each of our 3 agencies during that time. Can you please respond to items #1 and 2 below – ideally by next Tuesday so that we can nail down the schedule – and complete the other steps below to help me arrange these?   1. Let me know which time slots work best for folks on your end. Please pick a first and a second choice in case we have conflicts:   a. Monday 12/16 AM (8-12 or so) b. Monday 12/16 PM (1-5 or so) c. Monday 12/17 AM d. Monday 12/17 PM 2. Let me know if you’d like to host or prefer that Metro host. I’d encourage you to host at your own offices to make it easy for folks to attend, but I’m holding some meeting rooms here in case you need them. If you host, you’ll want to book a meeting room with a projector, good wifi, and outlets for folks to plug in on. These training sessions will be BYO laptop, so if people on your end don’t have laptops you’ll need to provide computers for them too. 3. Invite folks from your agency to participate. Replica is a little vague on what defines a “power user,” but in Metro’s case I’m interpreting it to mean people who provide data and analysis support to different departments and people who manage projects that are likely going to draw on Replica data. I’m trying to keep the group to no more than 30 people, and ideally closer to 10-20, so that there’s time for in-depth assistance. 4. Distribute Replica’s use case form (see attached email, which also contains some more info from Replica on how to fill it out) to participants to fill out – or if you prefer, set up a meeting w/ folks ahead of time to fill out a shared version, which is what I’m doing. Replica uses these to tailor training sessions to each agency. Due back to me by 12/6.   Thank you, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 Metro Making a great place From: To: Cc: Bcc: Subject: Date: Eliot Rose Kerr, Michael; Gilligan, Mike; Martin, Kevin; McHugh, Tim Joe Broach 3days@followupthen.com Re: Replica - custom geographies Wednesday, November 06, 2019 7:31:33 AM Hey all:  We’re uploading high-priority geographies and we plan to send the final list to Replica by Friday. We’re including all the requests we;’ve received in the list to see what they can do, but prioritizing stuff that’s not easily derivable from the other geographies and/or is widely-used.  Please hold 12/16-12/18 for a visit and trainings from Replica. This first round of trainings is going to be intended for the folks at your agency who are likely to be early / frequent users of the tool, who can then help onboard other users. I’m going to try and set up meeting rooms at Metro, especially on Wednesday, but I may reach out toward the end of this week about hosting at PBOT/TriMet.  Thanks, Eliot Eliot Rose Technology Strategist Metro eliot.rose@oregonmetro.gov On Oct 29, 2019, at 5:50 PM, Eliot Rose wrote: Hello all:    Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave.  Portland, OR 97232-2736 503-797-1825 Metro Making a great place From: To: Subject: Date: Eliot Rose Martin, Kevin Re: [External sender]Re: Replica - custom geographies Thursday, October 31, 2019 7:23:37 AM Yup! Just sent an invite for 5:30, but I can be flexible on the time if needed.  Eliot Rose Technology Strategist Metro eliot.rose@oregonmetro.gov On Oct 30, 2019, at 11:03 PM, Martin, Kevin wrote:   CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. ZGF thing was fine, but you did not miss much. My next couple of weeks are fun-filled. How about Wed 11/20? Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov  503.823.7710 https://www.smartcitypdx.com  @smartcitypdx The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Wednesday, October 30, 2019 3:35 PM To: Martin, Kevin Subject: RE: Replica - custom geographies   Oh, and thanks for the response on the ZGF event. I’m glad I was able to duck out… a couple fires to put out this afternoon. I hope it’s useful!   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov]  Sent: Wednesday, October 30, 2019 1:16 PM To: Eliot Rose Subject: [External sender]Re: Replica - custom geographies     CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Bet you're going to get a big list.   Hey, question for you. Can you get your hands on the Metro/PSU PORTAL IGA?   Another question. Ship Ahoy soon?   Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov  503.823.7710 https://www.smartcitypdx.com  @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Kerr, Michael Sent: Wednesday, October 30, 2019 1:14 PM To: Martin, Kevin Cc: Eliot Rose ; Gilligan, Mike ; McHugh, Tim ; Joe Broach Subject: Re: Replica - custom geographies   Have an inquiry out to a bunch of pbot planners (including Ning) with a request to get back to me ASAP.     Believe Ning derives city specific TAZs from Metro TAZs, but am pretty sure he was fine with the latter.   I’ll get back to everyone soon,   Mike   Sent from my iPhone On Oct 30, 2019, at 3:29 PM, Martin, Kevin wrote:   Mike, what geographies is Ning using for transportation modeling in PBOT? TAZs? Same TAZs as Metro (there are several versions, or used to be)?   That's my only thought, that we'd want alignment there. Seems like we could use the census geographies to aggregate to other areas of interest, deal with any slight differences.   Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov  503.823.7710 https://www.smartcitypdx.com  @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-8236868. From: Eliot Rose Sent: Tuesday, October 29, 2019 5:50 PM To: Kerr, Michael ; Gilligan, Mike ; Martin, Kevin ; McHugh, Tim Cc: Joe Broach Subject: Replica - custom geographies   Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place From: To: Cc: Subject: Date: Eliot Rose Kerr, Michael; Martin, Kevin; ningsheng.zhou@portlandoregon.gov Gilligan, Mike; McHugh, Tim; Joe Broach RE: Replica - custom geographies Wednesday, October 30, 2019 3:46:00 PM Thanks Michael! I should have mentioned that we also plan to ask for data at the new microTAZ level – though it’ll be a little experimental since we’re not sure about the sample size we’ll be drilling down to at that level. But if the City’s custom TAZs are built up from microTAZs then Ning’s needs might be covered. Adding him to the thread.   And while we’re on the topic… Replica is also looking to schedule a visit where they hold a series of trainings for Metro, Portland and TriMet staff. This will involve a couple 2-hour training sessions for each agency (hosted individually or jointly here at Metro) where people get logins and learn the basics of working with the program. These are designed to serve as the intro to Replica for the people that you initially intend to receive logins for Replica. They’re available the week of 12/2, the week of 12/9, and the first half of the week of 12/16 (12/16-12/18). Can you please let me know if there are dates that we should avoid (or that work particularly well) within that 12/2-12/16 window? With so many folks potentially attending I’m not planning to poll individual participants, but it would be good to know if there are dates that we should generally avoid due to other important meetings at PBOT/TriMet.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Wednesday, October 30, 2019 1:14 PM To: Martin, Kevin Cc: Eliot Rose; Gilligan, Mike; McHugh, Tim; Joe Broach Subject: [External sender]Re: Replica - custom geographies CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Have an inquiry out to a bunch of pbot planners (including Ning) with a request to get back to me ASAP. Believe Ning derives city specific TAZs from Metro TAZs, but am pretty sure he was fine with the latter. I’ll get back to everyone soon, Mike Sent from my iPhone On Oct 30, 2019, at 3:29 PM, Martin, Kevin wrote:  Mike, what geographies is Ning using for transportation modeling in PBOT? TAZs? Same TAZs as Metro (there are several versions, or used to be)?   That's my only thought, that we'd want alignment there. Seems like we could use the census geographies to aggregate to other areas of interest, deal with any slight differences.   Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Tuesday, October 29, 2019 5:50 PM To: Kerr, Michael ; Gilligan, Mike ; Martin, Kevin ; McHugh, Tim Cc: Joe Broach Subject: Replica - custom geographies Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Martin, Kevin RE: Replica - custom geographies Wednesday, October 30, 2019 3:36:00 PM Oh, and thanks for the response on the ZGF event. I’m glad I was able to duck out… a couple fires to put out this afternoon. I hope it’s useful!   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Wednesday, October 30, 2019 1:16 PM To: Eliot Rose Subject: [External sender]Re: Replica - custom geographies CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Bet you're going to get a big list.   Hey, question for you. Can you get your hands on the Metro/PSU PORTAL IGA?   Another question. Ship Ahoy soon?   Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Kerr, Michael Sent: Wednesday, October 30, 2019 1:14 PM To: Martin, Kevin Cc: Eliot Rose ; Gilligan, Mike ; McHugh, Tim ; Joe Broach Subject: Re: Replica - custom geographies Have an inquiry out to a bunch of pbot planners (including Ning) with a request to get back to me ASAP. Believe Ning derives city specific TAZs from Metro TAZs, but am pretty sure he was fine with the latter. I’ll get back to everyone soon, Mike Sent from my iPhone On Oct 30, 2019, at 3:29 PM, Martin, Kevin wrote:  Mike, what geographies is Ning using for transportation modeling in PBOT? TAZs? Same TAZs as Metro (there are several versions, or used to be)?   That's my only thought, that we'd want alignment there. Seems like we could use the census geographies to aggregate to other areas of interest, deal with any slight differences.   Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Tuesday, October 29, 2019 5:50 PM To: Kerr, Michael ; Gilligan, Mike ; Martin, Kevin ; McHugh, Tim Cc: Joe Broach Subject: Replica - custom geographies Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Martin, Kevin RE: Replica - custom geographies Wednesday, October 30, 2019 3:35:00 PM I welcome an exhaustive list of custom geographies. Let’s see what this Replica can do!   Asking around about the PORTAL IGA.   Yes to the Ship! Next Tuesday/Wednesday eves are looking good for me.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Wednesday, October 30, 2019 1:16 PM To: Eliot Rose Subject: [External sender]Re: Replica - custom geographies CAUTION: This email originated from an External source. Do not open links or attachments unless you know the content is safe. Bet you're going to get a big list.   Hey, question for you. Can you get your hands on the Metro/PSU PORTAL IGA?   Another question. Ship Ahoy soon?   Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Kerr, Michael Sent: Wednesday, October 30, 2019 1:14 PM To: Martin, Kevin Cc: Eliot Rose ; Gilligan, Mike ; McHugh, Tim ; Joe Broach Subject: Re: Replica - custom geographies Have an inquiry out to a bunch of pbot planners (including Ning) with a request to get back to me ASAP. Believe Ning derives city specific TAZs from Metro TAZs, but am pretty sure he was fine with the latter. I’ll get back to everyone soon, Mike Sent from my iPhone On Oct 30, 2019, at 3:29 PM, Martin, Kevin wrote:  Mike, what geographies is Ning using for transportation modeling in PBOT? TAZs? Same TAZs as Metro (there are several versions, or used to be)?   That's my only thought, that we'd want alignment there. Seems like we could use the census geographies to aggregate to other areas of interest, deal with any slight differences.   Kevin Martin (he/him/his) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Tuesday, October 29, 2019 5:50 PM To: Kerr, Michael ; Gilligan, Mike ; Martin, Kevin ; McHugh, Tim Cc: Joe Broach Subject: Replica - custom geographies Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Bcc: Subject: Date: Eliot Rose Kerr, Michael; Gilligan, Mike; Martin, Kevin; McHugh, Tim Joe Broach 3days@followupthen.com Replica - custom geographies Tuesday, October 29, 2019 5:51:00 PM Hello all:   Replica is preparing delivery of the first set of validation data, ETA mid/late November. They’ve asked what geographies we want data aggregated to. They do standard Census geos (block groups, tracts, cities, counties) and we plan to ask for some other custom Metro geographies (TAZs, school districts, neighborhoods). Are there any geographies that are important to Portland/TriMet that we should request?   Thanks, Eliot   Eliot Rose (he/him/his) Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Kerr, Michael; Martin, Kevin; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Joe Broach; Chris Johnson; Peter Bosa Replica data collection Friday, July 19, 2019 12:56:00 PM Hello all:   We kicked off data collection with Replica two weeks ago, and we’ve been working to get our internal process underway in the remaining weeks. Joe Broach (cc’d) will be leading data collection for Metro, and we’d love to get a good share of the data collected by the end of the month. I’m writing to ask you to keep moving on preparing and uploading your data, and to coordinate with Joe as you go. Here’s how we’d like that to work; please let me know if you have any questions or think a call is in order. We’ll be following up weekly from here on out to check in on progress.   ·         Portland, TriMet and Metro use the data collection template to update the status of the different datasets that each agency is responsible for providing (Portland’s already started, thanks Michael!) ·         Portland and TriMet collect and stage any data they are planning to share as part of the validation process on an internal site and provide Metro with access to that site (I believe that TriMet has already done this, but I don’t think that we have access). This is different from what Replica envisions, which is that we all upload data directly to their site. Staging data before sharing it with Replica should help streamline the process down the line, and ensure that we have a consistent process for withholding any data points we want to use for independent validation. ·         Metro documents all data points associated with each validation criterion, makes recommendations about which data points to withhold, and shares those with partners for feedback before uploading data to share with Replica.   ·         Once we agree on a plan for withholding data we will make a decision about how to standardize the data so that it’s in the format used by Replica.   Thanks, Eliot   Eliot Rose Technology Strategist Metro From: To: Cc: Subject: Date: Eliot Rose Jonathan Fink Kevin Martin RE: Conversation about Replica? Thursday, June 27, 2019 2:14:00 PM Thanks for the reminder, Jon. I’m happy to Monday afternoon after 2:30 or on the 24th/25th if you’d like to schedule something. We’ll be collecting ground-truth data and validating Replica over the next 3 months, and if it passes our test we’ll begin using it and allowing our partners access at the end of summer / early fall. I’m happy to talk with you now if you’re doing some long-term planning, but I’d also understand if you want to wait til later in the summer when we have more certainty about if/when we’ll have the data in hand.   Eliot Rose Technology Strategist Metro   From: Jonathan Fink [mailto:jonfink@pdx.edu] Sent: Thursday, June 27, 2019 1:53 PM To: Eliot Rose Cc: Kevin Martin Subject: Re: Conversation about Replica? Hi Eliot, Following up on whether we might have a conversation about the Replica project (I think Michael is out of town). Thanks. Jon On Tue, Jun 25, 2019 at 4:51 PM Jonathan Fink wrote: Hi Eliot and Michael, I'm wondering if I might be able to meet with one or both of you sometime in the next few weeks to learn more about the Replica project and whether there might be an opportunity for our Digital City Testbed Center to help you and your colleagues advance the transportation goals of the City and region. As I may have mentioned to you each before, our Center is in the process of setting up "digital corridors" on a few different campuses in our network, where we will be installing several different smart city applications. We'd be very interested in exploring whether it might make sense to include some of the Sidewalk Labs tools and data in our project. I'm around this week and next Monday, then will be back from July 18-29. Thanks. Jon -- Jonathan Fink Digital City Testbed Center Director Earth, Environment, and Society Doctoral Program Director Professor of Geology Portland State University Cascadia Urban Analytics Cooperative Co-Director Office of Research + Innovation Visiting Professor of Urban Analytics University of British Columbia (480) 239-4594 (c) jon.fink@pdx.edu jon.fink@ubc.ca @jonathanfink jonfink (Skype) https://www.linkedin.com/in/jonathanfink/ https://www.pdx.edu/digital-cities-testbed-center/ -Jonathan Fink Digital City Testbed Center Director Earth, Environment, and Society Doctoral Program Director Professor of Geology Portland State University Cascadia Urban Analytics Cooperative Co-Director Office of Research + Innovation Visiting Professor of Urban Analytics University of British Columbia (480) 239-4594 (c) jon.fink@pdx.edu jon.fink@ubc.ca @jonathanfink jonfink (Skype) https://www.linkedin.com/in/jonathanfink/ https://www.pdx.edu/digital-cities-testbed-center/ From: To: Cc: Bcc: Subject: Date: Attachments: Eliot Rose Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Lindstrom, Aubrey; McDonald, Mike; Gilligan, Mike; ningsheng.zhou@portlandoregon.gov; Owen, Jeffrey; Hesse, Eric; Sherman, Jacob; Hurley, Peter; Lonsdale, Stephanie Bryan Nguyen; Chris Johnson; Robert Kirkman; Joe Broach; Maribeth Todd; Jeff Frkonja 2days@followupthen.com Please review by 5/31: Replica Acceptance Criteria Thursday, May 30, 2019 10:10:00 AM Portland - Acceptance Criteria v5.docx Hello all:   I’m attaching the latest draft of the acceptance criteria. Portland and TriMet folks, can you please take a quick look and provide any feedback to me by the end of the week? Replica edited the previous version that we sent them following the kickoff meeting, and Metro reviewed and made a few tweaks to the criteria that are in our court to validate. We feel like this draft looks pretty good, and that the criteria and margins of error generally look reasonable given how we validate our model, but we have a few outstanding questions about the transit and active transportation criteria, which are flagged with comments in the attached version. We’ll relay any questions or feedback we receive from you to Replica on our check-in call next Monday, and aim to get these finalized by the end of next week. We’d like Tim and Mike to sign off via email that TriMet and Portland are OK with these acceptance criteria before we consider them final.   The acceptance criteria do not address some of the more detailed questions that we’ve heard from you – things like how many bike ped count locations are we validating, and are we planning to withhold some data so that we can test the Replica data independently? That’s by design. The acceptance criteria are going to go in the contract, and we want them to give us cover to include all the data that we want to include in the validation testing while still allowing for some flexibility as we all learn more about the Replica dataset and process. They should cover all the types of data that we want to include, and specify validation thresholds for each type of data, but not get into details on which data points should be included within each type of data. We’ll be getting into these details once we finalize the acceptance criteria and begin the data collection process. If you want to get a preview of what that will be like, check out the DATA subfolder on the Google Drive folder that Replica set up, especially the Ground Truth Data Formatting document. Let me know if you still need access to the Drive.   Lastly, Nathan from Replica is pushing back his visit to Portland a bit til we’re further along with the data collection. I’ll be in touch about rescheduling that.   Thanks for the quick turnaround on this. I’m available most of today and tomorrow before 1 if you’d like to give me a call with any questions.   Eliot Rose Technology Strategist Metro   From: To: Cc: Subject: Date: Eliot Rose Martin, Kevin; Kerr, Michael Nick Christensen RE: Sidewalk Labs - Replica Wednesday, May 29, 2019 9:48:00 AM Thanks for asking! Please refer any inquiries to Nick Christensen (cc’d) from our comms team.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Wednesday, May 29, 2019 9:34 AM To: Eliot Rose; Kerr, Michael Subject: Re: Sidewalk Labs - Replica Thanks Eliot. Is there a specific Metro Comms lead for Replica? We can point folks in that direction. Or do you want these inquiries to come to you?   Kevin From: Eliot Rose Sent: Wednesday, May 29, 2019 9:16 AM To: Kerr, Michael; Martin, Kevin Subject: RE: Sidewalk Labs - Replica Thanks for the heads-up. I just looped in our comms folks as well.   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Wednesday, May 29, 2019 9:02 AM To: Martin, Kevin Cc: Eliot Rose Subject: Re: Sidewalk Labs - Replica Yes all inquiries route through John first. Mike Sent from my iPhone On May 29, 2019, at 8:27 AM, Martin, Kevin wrote: We're getting a few media inquiries related to the Geekwire article yesterday. Where do you two want these directed? Mike, assume John Brady is still the point of contact at PBOT?   Thanks! Kevin   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-8237700 or use City TTY 503-823-6868.   Begin forwarded message: From: Sarah Wray Date: May 29, 2019 at 02:23:31 PDT To: "christine.llobregat@portlandoregon.gov" Subject: Sidewalk Labs - Replica Hi Christine I am an editor at Smart Cities World – I saw this report https://www.geekwire.com/2019/portland-quietly-launches-mobilelocation-data-project-alphabets-controversial-sidewalk-labs/ and I wanted to check the facts directly. The article notes that Smart City PDX does not oversee the Replica project, but who does within Portland? Could you confirm the details and scope of this pilot or point/introduce me to someone who can? Thanks! Sarah From: To: Subject: Date: Eliot Rose Kerr, Michael; Martin, Kevin RE: Sidewalk Labs - Replica Wednesday, May 29, 2019 9:15:00 AM Thanks for the heads-up. I just looped in our comms folks as well.   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Wednesday, May 29, 2019 9:02 AM To: Martin, Kevin Cc: Eliot Rose Subject: Re: Sidewalk Labs - Replica Yes all inquiries route through John first. Mike Sent from my iPhone On May 29, 2019, at 8:27 AM, Martin, Kevin wrote: We're getting a few media inquiries related to the Geekwire article yesterday. Where do you two want these directed? Mike, assume John Brady is still the point of contact at PBOT?   Thanks! Kevin   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-8237700 or use City TTY 503-823-6868.   Begin forwarded message: From: Sarah Wray Date: May 29, 2019 at 02:23:31 PDT To: "christine.llobregat@portlandoregon.gov" Subject: Sidewalk Labs - Replica Hi Christine I am an editor at Smart Cities World – I saw this report https://www.geekwire.com/2019/portland-quietly-launches-mobilelocation-data-project-alphabets-controversial-sidewalk-labs/ and I wanted to check the facts directly. The article notes that Smart City PDX does not oversee the Replica project, but who does within Portland? Could you confirm the details and scope of this pilot or point/introduce me to someone who can? Thanks! Sarah From: To: Subject: Date: Eliot Rose Martin, Kevin; Kerr, Michael RE: ~date replica data available? Friday, May 10, 2019 4:17:00 PM Timing is still uncertain. I’d guess we can expect to see uncalibrated data within a month, and we’re shooting to complete validation and take the first quarter of calibrated data by the end of June, but that’s ambitious. End of July seems more realistic. We did just get the contract signed, though.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Friday, May 10, 2019 2:54 PM To: Eliot Rose; Kerr, Michael Subject: ~date replica data available? I should probably know this. Apologies if I missed it in the kickoff.   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, April 11, 2019 4:24 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Date: Attachments: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike RE: Replica validation: meeting 2 Wednesday, April 24, 2019 4:20:00 PM DRAFT Replica Acceptance Criteria v2 notes.xlsx Hello all:   I haven’t received any edits on the Replica validation criteria. I’d like to share our draft with Sidewalk Labs in advance of next Thursday’s kickoff so that we can discuss them on the call. Please send me your edits or confirm that you don’t have any changes by the end of the week.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, April 11, 2019 4:24 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----- From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Importance: Eliot Rose Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse; ningsheng.zhou@portlandoregon.gov Canceled: Portland Metro Replica Kickoff High Hello all: Please hold this time for the Replica kickoff. I had to go with the 2nd best time because of a lack of available conference rooms, and I apologize for those that aren’t able to make it. I’m still scrounging for another room in our primary slot, and I’ll send a placeholder for that alternate time in case we find a venue. Hope to finalize this soon; thanks for your patience! -Eliot From: To: Subject: Eliot Rose Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse [ALTERNATE] Portland Metro Replica kickoff Hello all: This is an alternate date for the Replica kickoff; it was the time that worked best for all but we’re still seeking a meeting room. I’ll cancel this or the other invite and get a final time/location by the end of the week. -Eliot From: To: Subject: Eliot Rose Chris Johnson; Maribeth Todd; Kerr, Michael; joshua.lynch@portlandoregon.gov; Martin, Kevin; McHugh, Bibiana; steelem@trimet.org; Nathan Preheim; Lindstrom, Aubrey; Joe Broach; McDonald, Mike; Gilligan, Mike; Pamela Blackhorse Portland Metro Replica Kickoff Hello all: Please hold this time for the Replica kickoff. I had to go with the 2nd best time because of a lack of available conference rooms, and I apologize for those that aren’t able to make it. I’m still scrounging for another room in our primary slot, and I’ll send a placeholder for that alternate time in case we find a venue. Hope to finalize this soon; thanks for your patience! -Eliot From: To: Subject: Date: Eliot Rose Kerr, Michael; Martin, Kevin RE: Portland Metro Replica kickoff meeting Wednesday, April 17, 2019 4:47:00 PM All this is missing is you two and the PBOT planners… can you please get me responses? Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, April 16, 2019 9:30 AM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike'; 'Christine.Kendrick@portlandoregon.gov'; Joe Broach; 'Nathan Preheim'; Nick Bowden Subject: Portland Metro Replica kickoff meeting   Hello all:   Please fill out this poll to help us schedule the Replica kickoff meeting – preferably by tomorrow COB if possible. We’ll have people from Metro, City of Portland, TriMet and Sidewalk Labs on the call.   I know that some of you are still trying to loop in a couple of other folks from your organization; feel free to forward the scheduling link to them. Sidewalk Labs has some materials to share, and we’ll make sure that everyone on the meeting invite gets access to those.   Here’s the agenda, from Sidewalk Labs:     Phases of Deployment (5 minutes) Requirements Kickoff (45 minutes)      Data Accuracy + Acceptance Criteria      Data Trust      Product Utility Data Collection (15 minutes) Next Steps (5 minutes)   Most of the conversation centers around capturing as much requirements information about your community as possible. As you know we need a host of inputs so that we can build up a representative model. The acceptance criteria (which we call Data Accuracy) is a big component to that, but there are other dimensions that we also explore on the call.    The call is a kickoff, and it lays the foundation for all of the work and effort required to start the deployment process.   -Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 15, 2019 11:40 AM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello everyone:   Just a reminder to please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the Replica kickoff ASAP. I plan to send out a scheduling poll at the end of the day.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, April 11, 2019 4:24 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Date: Eliot Rose Martin, Kevin RE: Replica validation: meeting 2 Tuesday, April 16, 2019 3:19:00 PM Done! Glad y’all are joining us.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Monday, April 15, 2019 11:10 PM To: Eliot Rose Subject: Re: Replica validation: meeting 2   Hey Eliot. Apologies for the delay. Would you please include Christine Kendrick (christine.kendrick@portlandoregon.gov) and me in the kickoff Doodle?   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Monday, April 15, 2019 11:39 AM To: Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: RE: Replica validation: meeting 2   Hello everyone:   Just a reminder to please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the Replica kickoff ASAP. I plan to send out a scheduling poll at the end of the day.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, April 11, 2019 4:24 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Date: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike; Christine.Kendrick@portlandoregon.gov; Joe Broach; Nathan Preheim; Nick Bowden Portland Metro Replica kickoff meeting Tuesday, April 16, 2019 9:29:00 AM Hello all:   Please fill out this poll to help us schedule the Replica kickoff meeting – preferably by tomorrow COB if possible. We’ll have people from Metro, City of Portland, TriMet and Sidewalk Labs on the call.   I know that some of you are still trying to loop in a couple of other folks from your organization; feel free to forward the scheduling link to them. Sidewalk Labs has some materials to share, and we’ll make sure that everyone on the meeting invite gets access to those.   Here’s the agenda, from Sidewalk Labs:     Phases of Deployment (5 minutes) Requirements Kickoff (45 minutes)      Data Accuracy + Acceptance Criteria      Data Trust      Product Utility Data Collection (15 minutes) Next Steps (5 minutes)   Most of the conversation centers around capturing as much requirements information about your community as possible. As you know we need a host of inputs so that we can build up a representative model. The acceptance criteria (which we call Data Accuracy) is a big component to that, but there are other dimensions that we also explore on the call.    The call is a kickoff, and it lays the foundation for all of the work and effort required to start the deployment process.   -Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, April 15, 2019 11:40 AM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello everyone:   Just a reminder to please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the Replica kickoff ASAP. I plan to send out a scheduling poll at the end of the day.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, April 11, 2019 4:24 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Date: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike RE: Replica validation: meeting 2 Monday, April 15, 2019 11:39:00 AM Hello everyone:   Just a reminder to please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the Replica kickoff ASAP. I plan to send out a scheduling poll at the end of the day.   Thanks, Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Thursday, April 11, 2019 4:24 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: meeting 2   Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Date: Attachments: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike RE: Replica validation: meeting 2 Thursday, April 11, 2019 4:24:00 PM DRAFT Replica Acceptance Criteria v2 notes.xlsx Hello all:   Thanks for the conversation today. Attached is a version of the validation criteria with some add’l notes that I took during the call; I didn’t make any substantive changes. Data sources that are not publicly available are highlighted because we want to track how many of those there are any how they’re distributed across modes moving forward. Here are the next steps we discussed: ·         By the end of the week: please email me and let me know names and email addresses of anyone at your agency whom you would like to attend the kickoff and I’ll send out a scheduling poll. ·         By 4/19: Please send me your edits to the validation criteria.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Date: Attachments: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike RE: Replica validation: meeting 2 Wednesday, April 10, 2019 4:40:00 PM DRAFT Replica Acceptance Criteria v2.xlsx Hello all:   Looking forward to our conversation tomorrow. Attached is an updated draft of the acceptance criteria. Agenda: ·         Intros (5 min.) ·         Review draft acceptance criteria (Chris, 40 min.) o   Collect feedback on draft o   Discuss validation expectations for PBOT/TriMet (e.g., # of counts/cutlines) o   Discuss consistency in definitions (purpose, time of day) ·         Kickoff (Eliot, 15 min.) o   Scheduling o   Attendance o   Access to documents   Eliot Rose Technology Strategist Metro     -----Original Appointment----From: Eliot Rose Sent: Monday, April 08, 2019 12:13 PM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: Replica validation: meeting 2 When: Thursday, April 11, 2019 12:30 PM-1:30 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro (Chris' office) / call-in (605) 472-5418 / 616836#     Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks!   From: To: Subject: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Replica validation: meeting 2 Agenda to come. PBOT/TriMet folks, please let me know if you want to attend in person so I can grab a larger meeting room if necessary. Thanks! From: To: Subject: Date: Eliot Rose Martin, Kevin RE: Replica validation: scheduling next meeting Thursday, April 04, 2019 3:53:00 PM I tried, but I only got to 16 options.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Thursday, April 04, 2019 3:39 PM To: Eliot Rose Subject: Re: Replica validation: scheduling next meeting Who does not love a Doodle poll? My favorites are the ones with 30+ choices. From: Eliot Rose Sent: Thursday, April 4, 2019 3:14 PM To: Martin, Kevin Subject: RE: Replica validation: scheduling next meeting Seems like everyone else wants to move it. Or maybe they just secretly love completing Doodle polls. Doodle poll coming shortly.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Thursday, April 04, 2019 2:58 PM To: Eliot Rose Subject: Re: Replica validation: scheduling next meeting Thanks Eliot. I should still be able to call in. How did others feel?   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Thursday, April 4, 2019 9:32 AM To: Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: RE: Replica validation: scheduling next meeting Folks, my apologies – I had intended to schedule the next Replica validation meeting for tomorrow (Friday 4/5) at 11 and I just realized that the invitation has been languishing unsent in my outbox. Please let me know if you all happen to still be available at that time, otherwise I’ll send out a poll to find for next week. I’m sorry about the mishap.   Eliot Rose Technology Strategist Metro     _____________________________________________ From: Eliot Rose Sent: Monday, March 18, 2019 4:06 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: scheduling next meeting     Just a reminder to please fill out this poll if you haven’t already. I’d like to schedule something by tomorrow COB. Thanks!   Eliot Rose Technology Strategist Metro     _____________________________________________ From: Eliot Rose Sent: Thursday, March 14, 2019 12:52 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; McDonald, Mike; Lindstrom, Aubrey; 'Gilligan, Mike' Subject: Replica validation: scheduling next meeting     Hello all:   Please fill out this poll so we can schedule our next meeting. We’ll have a draft of the validation criteria to share beforehand.   Thanks Eliot Eliot Rose Technology Strategist k?etro From: To: Subject: Date: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike RE: Replica validation: scheduling next meeting Thursday, April 04, 2019 3:52:00 PM OK, seems like the preference is to reschedule for next week. Please fill out this poll by Friday COB – I promise to actually send out an invite for the preferred time slot this time.  Eliot Rose Technology Strategist Metro _____________________________________________ From: Eliot Rose Sent: Thursday, April 04, 2019 9:32 AM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: scheduling next meeting Importance: High Folks, my apologies – I had intended to schedule the next Replica validation meeting for tomorrow (Friday 4/5) at 11 and I just realized that the invitation has been languishing unsent in my outbox. Please let me know if you all happen to still be available at that time, otherwise I’ll send out a poll to find for next week. I’m sorry about the mishap. Eliot Rose Technology Strategist Metro _____________________________________________ From: Eliot Rose Sent: Monday, March 18, 2019 4:06 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: scheduling next meeting Just a reminder to please fill out this poll if you haven’t already. I’d like to schedule something by tomorrow COB. Thanks! Eliot Rose Technology Strategist Metro _____________________________________________ From: Eliot Rose Sent: Thursday, March 14, 2019 12:52 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; McDonald, Mike; Lindstrom, Aubrey; 'Gilligan, Mike' Subject: Replica validation: scheduling next meeting Hello all: Please fill out this poll so we can schedule our next meeting. We’ll have a draft of the validation criteria to share beforehand. Thanks, Eliot Eliot Rose Technology Strategist Metro From: To: Subject: Date: Eliot Rose Martin, Kevin RE: Replica validation: scheduling next meeting Thursday, April 04, 2019 3:14:00 PM Seems like everyone else wants to move it. Or maybe they just secretly love completing Doodle polls. Doodle poll coming shortly.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Thursday, April 04, 2019 2:58 PM To: Eliot Rose Subject: Re: Replica validation: scheduling next meeting Thanks Eliot. I should still be able to call in. How did others feel?   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com @smartcitypdx   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: Eliot Rose Sent: Thursday, April 4, 2019 9:32 AM To: Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike Subject: RE: Replica validation: scheduling next meeting Folks, my apologies – I had intended to schedule the next Replica validation meeting for tomorrow (Friday 4/5) at 11 and I just realized that the invitation has been languishing unsent in my outbox. Please let me know if you all happen to still be available at that time, otherwise I’ll send out a poll to find for next week. I’m sorry about the mishap.   Eliot Rose Technology Strategist Metro     _____________________________________________ From: Eliot Rose Sent: Monday, March 18, 2019 4:06 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: scheduling next meeting     Just a reminder to please fill out this poll if you haven’t already. I’d like to schedule something by tomorrow COB. Thanks!   Eliot Rose Technology Strategist Metro     _____________________________________________ From: Eliot Rose Sent: Thursday, March 14, 2019 12:52 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; McDonald, Mike; Lindstrom, Aubrey; 'Gilligan, Mike' Subject: Replica validation: scheduling next meeting     Hello all:   Please fill out this poll so we can schedule our next meeting. We’ll have a draft of the validation criteria to share beforehand.   Thanks, Eliot   Eliot Rose Technology Strategist Metro     From: To: Subject: Date: Importance: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike RE: Replica validation: scheduling next meeting Thursday, April 04, 2019 9:32:00 AM High Folks, my apologies – I had intended to schedule the next Replica validation meeting for tomorrow (Friday 4/5) at 11 and I just realized that the invitation has been languishing unsent in my outbox. Please let me know if you all happen to still be available at that time, otherwise I’ll send out a poll to find for next week. I’m sorry about the mishap. Eliot Rose Technology Strategist Metro _____________________________________________ From: Eliot Rose Sent: Monday, March 18, 2019 4:06 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; 'McDonald, Mike'; 'Lindstrom, Aubrey'; 'Gilligan, Mike' Subject: RE: Replica validation: scheduling next meeting Just a reminder to please fill out this poll if you haven’t already. I’d like to schedule something by tomorrow COB. Thanks! Eliot Rose Technology Strategist Metro _____________________________________________ From: Eliot Rose Sent: Thursday, March 14, 2019 12:52 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; McDonald, Mike; Lindstrom, Aubrey; 'Gilligan, Mike' Subject: Replica validation: scheduling next meeting Hello all: Please fill out this poll so we can schedule our next meeting. We’ll have a draft of the validation criteria to share beforehand. Thank; Eliot Eliot Rose Technology Strategist Metro From: To: Subject: Date: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike RE: Replica validation: scheduling next meeting Monday, March 18, 2019 4:06:00 PM Just a reminder to please fill out this poll if you haven’t already. I’d like to schedule something by tomorrow COB. Thanks! Eliot Rose Technology Strategist Metro _____________________________________________ From: Eliot Rose Sent: Thursday, March 14, 2019 12:52 PM To: Chris Johnson; 'Kerr, Michael'; 'Martin, Kevin'; Robert Kirkman; 'McHugh, Tim'; McDonald, Mike; Lindstrom, Aubrey; 'Gilligan, Mike' Subject: Replica validation: scheduling next meeting Hello all: Please fill out this poll so we can schedule our next meeting. We’ll have a draft of the validation criteria to share beforehand. Thanks, Eliot Eliot Rose Technology Strategist Metro From: To: Bcc: Subject: Date: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; McDonald, Mike; Lindstrom, Aubrey; Gilligan, Mike 5days@followupthen.com Replica validation: scheduling next meeting Thursday, March 14, 2019 12:51:00 PM Hello all: Please fill out this poll so we can schedule our next meeting. We’ll have a draft of the validation criteria to share beforehand. Thanks, Eliot Eliot Rose Technology Strategist Metro From: To: Subject: Date: Attachments: Eliot Rose Martin, Kevin FW: Replica documentation Tuesday, March 12, 2019 8:06:00 PM IEEE_IOHMM.PDF Replica_DataDisclouse_Summary_011119.pdf Great seeing you today!   Replica methodology article is attached, as is the data disclosure doc they passed along, which has a brief synopsis of the privacy audit at the end.   Eliot Rose Technology Strategist Metro From: To: Subject: Date: Eliot Rose Martin, Kevin RE: ACLU briefing on Sidewalk Labs Replica pilot/Privacy Principles Wednesday, February 27, 2019 7:08:00 PM Thanks Kevin!   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Tuesday, February 26, 2019 11:32 PM To: Eliot Rose; Kerr, Michael Subject: Fw: ACLU briefing on Sidewalk Labs Replica pilot/Privacy Principles FYI. I'll keep you posted, loop you in once we get a response.   K   From: Martin, Kevin Sent: Tuesday, February 26, 2019 11:29 PM To: jcarson@aclu-or.org Subject: ACLU briefing on Sidewalk Labs Replica pilot/Privacy Principles Jann,   It's be a while... we last spoke over a year ago as PBOT prepared to launch a pilot of streetlight-based sensors to count traffic/bikes/peds, your feedback was valuable to shaping that project. We are hoping to do a similar briefing regarding a new project that Metro, City of Portland and TriMet have partnered on to use Sidewalk Lab's Replica software to better understand travel patterns in the region (a significant data need). We'd also like to use the time to discuss a Privacy and Information Protection Resolution that we will be bringing to City Council in May.   Is it possible that we could get a bit of time in the next few weeks to discuss? Happy to coordinate schedules on our side if you can float a few dates and times that work for you.   A brief description of the Replica project is attached. OPB did a story back in December as well: https://www.opb.org/news/article/cellphone-location-data-portland-google-privacy/. And I've also attached a draft of the Privacy and Information Protection Resolution.   Let me know. Hope all is well at ACLU!   Thanks, Kevin   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com   The City of Portland is committed to providing meaningful access. For accommodations, modifications, translation, interpretation or other services, please contact at 503-823-7700 or use City TTY 503-823-6868. From: To: Subject: Date: Attachments: Eliot Rose Martin, Kevin; Kerr, Michael RE: Replica: add"l talking points and guidance Tuesday, February 26, 2019 9:05:00 AM Replica summary Feb 2019.docx Sounds good. Here’s an updated one-pager; I think that it would be good to share the OPB article as well.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Monday, February 25, 2019 7:45 PM To: Kerr, Michael Cc: Eliot Rose Subject: Re: Replica: add'l talking points and guidance Cool. Think it changes the way we approach. We are being proactive rather than reactive. They appreciated this approach with the sensors, makes sense to get out in front.   I'll email Jann, copy you both, we'll go from there. Do we have a 1-pager on SL I can share? If not, I may just point her to the OPB article? Sound ok? I'll also be sharing the privacy resolution for discussion as well.   Thanks, K From: Kerr, Michael Sent: Monday, February 25, 2019 7:35 PM To: Martin, Kevin Cc: Eliot Rose Subject: Re: Replica: add'l talking points and guidance No, but I’d be surprised if they weren’t somewhat concerned given their involvement in the OR House Bill restricting the acquisition of geospatial data (can’t remember the number off the top of my head but they’re a lead sponsor). The bill is an overreach in my opinion, but makes a strong point that the public sector has a lot more to do when it comes to protecting individual privacy. Sent from my iPhone On Feb 25, 2019, at 7:09 PM, Martin, Kevin wrote: Do we know if SL is currently on ACLU's radar?   Kevin From: Eliot Rose Sent: Monday, February 25, 2019 5:37 PM To: Martin, Kevin; Kerr, Michael Subject: RE: Replica: add'l talking points and guidance Sorry it got bumped! That just prolongs the nervousness.   I’m going to defer to you on how to reach out to ACLU and who needs to be included. Y’all have the existing relationship and are also the most likely to face public scrutiny. I’m happy to reach out directly, include/consult with Robert, etc. My only requests are that someone from PBOT join in the meeting and that we get to talk with whomever at the City is most familiar with ACLU’s concerns in advance of talking with Jann to make sure that we’re prepared.   Thanks!   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Monday, February 25, 2019 5:27 PM To: Eliot Rose; Kerr, Michael Subject: Re: Replica: add'l talking points and guidance Yes, it's been on my list to reach out on the Privacy Resolution. Unfortunately, today our resolution got bumped to 5/22. Too much nervousness in City Hall.   Jann Carlson, Deputy Director of the Portland office, is my contact (jcarson@acluor.org). Want me to intro and help coordinate? I may loop in Robert Taylor here from our City Attorney's office, he's lead on matters related to ACLU.   Let me know.   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com   From: Eliot Rose Sent: Monday, February 25, 2019 5:12 PM To: Martin, Kevin; Kerr, Michael Subject: RE: Replica: add'l talking points and guidance Kevin, can you please connect us with the ACLU contact you mentioned—assuming you still think that it’s a good idea to speak with her? We’re closing in on a final contract w/ SWL, and I’d ideally like to reach out to her this week to stay on track.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, February 12, 2019 10:34 AM To: Martin, Kevin; 'Kerr, Michael' Subject: RE: Replica: add'l talking points and guidance   Hey team:   I followed up with Nick, who’s been busy, yesterday and he confirmed that this add’l comms guidance is good to go. I’m not sure that we need to loop in add’l Metro folks since all questions about this are going to flow to me or Nick, but below is an updated list of talking points that you can distribute to your teams. I’m planning to share these with TriMet’s PIO and PM too and give them an update via email, but let me know if you’d prefer that I set up a meeting instead, or if there’s anything I else I can do to help support. It seems like PBOT is going to be fielding the lion’s share of questions on these.   Kevin, welcome back! Hope you shredded hard. Can you please connect us with the ACLU contact that you mentioned?   I also followed up on a couple of other items that Mike and I had discussed with Nick from SWL: · The current version of the online Replica tool does allow users to query and download disaggregate data, and Mike was going to check in on whether PBOT staff want access to that data or only want aggregate data. Please let me know how you want to proceed with that. We can ask about turning that feature off, but we’d want to raise that conversation sooner rather than later, and make sure that TriMet is OK with that too. · SWL isn’t willing to share the full privacy audit, but he did let us know who the author is (Jean-Louis Tambay, a Canadian privacy expert; googled hi mand found some publications but didn’t read too deeply yet), and I asked about the possibility of us having a conversation with him. Let me know if you think that would help to follow up on that.   Replica talking points This project is about understanding travel patterns, not individuals. We are exploring new data on travel patterns so that we can better tackle issues like congestion, safety, and transportation equity. The data we are getting through this project are totally private. We are not getting access to any information on real individuals’ travel behavior. We will get a detailed simulation of neighborhood-level travel patterns. This is a super-rich tool. We are going to be able to do things with this information that we would never be able to do with the data that we normally have access to: If we make more space for bikes and pedestrians at an intersection, we can use Replica to examine whether people who wouldn’t otherwise walk or bike are using those facilities. If we increase bus service to East Portland, we can use Replica to assess whether people of color and low-income communities are taking advantage of that service. We can look at where and how people are using ride-hailing to understand where it supports transit and reduces congestion – and where ride-hailing creates more traffic. This project is about saving public resources. We spend a lot of money to get information on travel patterns, often by sending people out to count cars or bicyclists by hand. If this tool works, we can spend fewer public dollars on understanding the transportation system and more on improving it. We get approached by a lot of data vendors, and we’re well aware of the products that are out there. We are working with Replica because it is a unique product that best meets our and our residents’ needs: It covers all modes, including bicycle and pedestrian trips and new modes like ride-hailing. It covers all types of people – including people of color and people living on low incomes, who are typically underrepresented in transportation data – while protecting privacy. It covers why people are traveling, so that we can use the data to meet their needs whether they’re going to work, shopping, school or another destination. It’s available to all transportation agencies in the Portland region, so that we can coordinate to best serve our residents. We get to test Replica before committing to use it. We won’t use or pay for the data if it doesn’t meet our criteria for accuracy and privacy protection. Even if it does pass our test, we have only signed up to use it for a year, and then we will evaluate whether it meets our goals. We will only have access to data on how a simulated version of our region’s population travels during an average week. We will not have access to any information on real people’s travel behavior. We take privacy seriously. We appreciate that the media is drawing attention to the privacy concerns raised by data companies’ use of personal information, and people should opt out of sharing their personal and location data if they are concerned. We are doing this as a pilot so that we can understand any implications for citizens’ privacy of working with information that’s based on location data. Our residents are concerned about serious transportation safety and equity issues, and have asked us to get better data so that we can figure out how to best tackle these issues. At the same time, our residents are also increasingly concerned about privacy. This pilot is an opportunity to understand whether there are conflicts between the need for better data and the right to privacy, and if so, how we can address those conflicts. Sidewalk Labs has committed to protecting privacy, and our draft contract ensures that Replica does not include personal information or make it possible to identify any individual. We have not yet signed a contract with Sidewalk Labs, so we have the opportunity to address any additional concerns that are raised.   General guidance for staff who receive questions from the public about Replica: Don’t attempt to speak about the Replica pilot if you don’t feel informed about it. Instead, refer questions to your PIO, or contact the staff managing this project for technical questions about the pilot (Michael Kerr, PBOT; Tim McHugh, TriMet; Eliot Rose, Metro). · Don’t attempt to defend Replica or the companies involved in its creation. This project is an opportunity for us to hear from people about privacy concerns, which we take seriously, and we have the opportunity to address these concerns as we draft an agreement with Sidewalk Labs.     · Emphasize the value of better transportation data in improving the safety, equity, and efficiency of the transportation system. · Emphasize that Replica is a unique product that provides detailed and comprehensive data on travel patterns while protecting privacy. · Acknowledge privacy concerns, commit to relaying them to the Replica project team, and encourage concerned people to take action to protect their privacy. · Emphasize that this is a pilot project. We are testing Replica rigorously before we accept the data, and we are only planning to use the data for a year and then decide whether to continue. · Coverage so far: The Intercept: https://theintercept.com/2019/01/28/google-alphabet-sidewalklabs-replica-cellphone-data/ · OPB: https://www.opb.org/news/article/cellphone-location-data-portland-googleprivacy/ · BikePortland: https://bikeportland.org/2018/12/05/portland-ponders-pilot-ofpowerful-transportation-data-tool-292817 · The Wall Street Journal is doing a deep investigation of Replica/Sidewalk Labs and has filed public records requests with Portland and Metro for our correspondence related to this project, as well as interviewing staff. They’re also looking into the ~5 other regions that are exploring this project. Their story should be out in a week or so, and I suspect that it will raise similar concerns to those in the piece by the Intercept. · In addition, a waterfront development by Sidewalk Labs in Toronto has met with controversy. That’s not directly related to this project, but it colors people’s perception of it. Eliot Rose Technology Strategist Metro     From: To: Subject: Date: Eliot Rose Martin, Kevin; Kerr, Michael RE: Replica: add"l talking points and guidance Monday, February 25, 2019 5:37:00 PM Sorry it got bumped! That just prolongs the nervousness.   I’m going to defer to you on how to reach out to ACLU and who needs to be included. Y’all have the existing relationship and are also the most likely to face public scrutiny. I’m happy to reach out directly, include/consult with Robert, etc. My only requests are that someone from PBOT join in the meeting and that we get to talk with whomever at the City is most familiar with ACLU’s concerns in advance of talking with Jann to make sure that we’re prepared.   Thanks!   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Monday, February 25, 2019 5:27 PM To: Eliot Rose; Kerr, Michael Subject: Re: Replica: add'l talking points and guidance Yes, it's been on my list to reach out on the Privacy Resolution. Unfortunately, today our resolution got bumped to 5/22. Too much nervousness in City Hall.   Jann Carlson, Deputy Director of the Portland office, is my contact (jcarson@aclu-or.org). Want me to intro and help coordinate? I may loop in Robert Taylor here from our City Attorney's office, he's lead on matters related to ACLU.   Let me know.   Kevin Martin (he/him) Smart City PDX/Tech Services Manager City of Portland Bureau of Planning & Sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com   From: Eliot Rose Sent: Monday, February 25, 2019 5:12 PM To: Martin, Kevin; Kerr, Michael Subject: RE: Replica: add'l talking points and guidance Kevin, can you please connect us with the ACLU contact you mentioned—assuming you still think that it’s a good idea to speak with her? We’re closing in on a final contract w/ SWL, and I’d ideally like to reach out to her this week to stay on track.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, February 12, 2019 10:34 AM To: Martin, Kevin; 'Kerr, Michael' Subject: RE: Replica: add'l talking points and guidance   Hey team:   I followed up with Nick, who’s been busy, yesterday and he confirmed that this add’l comms guidance is good to go. I’m not sure that we need to loop in add’l Metro folks since all questions about this are going to flow to me or Nick, but below is an updated list of talking points that you can distribute to your teams. I’m planning to share these with TriMet’s PIO and PM too and give them an update via email, but let me know if you’d prefer that I set up a meeting instead, or if there’s anything I else I can do to help support. It seems like PBOT is going to be fielding the lion’s share of questions on these.   Kevin, welcome back! Hope you shredded hard. Can you please connect us with the ACLU contact that you mentioned?   I also followed up on a couple of other items that Mike and I had discussed with Nick from SWL: · The current version of the online Replica tool does allow users to query and download disaggregate data, and Mike was going to check in on whether PBOT staff want access to that data or only want aggregate data. Please let me know how you want to proceed with that. We can ask about turning that feature off, but we’d want to raise that conversation sooner rather than later, and make sure that TriMet is OK with that too. · SWL isn’t willing to share the full privacy audit, but he did let us know who the author is (Jean-Louis Tambay, a Canadian privacy expert; googled hi mand found some publications but didn’t read too deeply yet), and I asked about the possibility of us having a conversation with him. Let me know if you think that would help to follow up on that.   Replica talking points This project is about understanding travel patterns, not individuals. We are exploring new data on travel patterns so that we can better tackle issues like congestion, safety, and transportation equity. The data we are getting through this project are totally private. We are not getting access to any information on real individuals’ travel behavior. We will get a detailed simulation of neighborhood-level travel patterns. This is a super-rich tool. We are going to be able to do things with this information that we would never be able to do with the data that we normally have access to: If we make more space for bikes and pedestrians at an intersection, we can use Replica to examine whether people who wouldn’t otherwise walk or bike are using those facilities. If we increase bus service to East Portland, we can use Replica to assess whether people of color and low-income communities are taking advantage of that service. We can look at where and how people are using ride-hailing to understand where it supports transit and reduces congestion – and where ride-hailing creates more traffic. This project is about saving public resources. We spend a lot of money to get information on travel patterns, often by sending people out to count cars or bicyclists by hand. If this tool works, we can spend fewer public dollars on understanding the transportation system and more on improving it. We get approached by a lot of data vendors, and we’re well aware of the products that are out there. We are working with Replica because it is a unique product that best meets our and our residents’ needs: It covers all modes, including bicycle and pedestrian trips and new modes like ride-hailing. It covers all types of people – including people of color and people living on low incomes, who are typically underrepresented in transportation data – while protecting privacy. It covers why people are traveling, so that we can use the data to meet their needs whether they’re going to work, shopping, school or another destination. It’s available to all transportation agencies in the Portland region, so that we can coordinate to best serve our residents. We get to test Replica before committing to use it. We won’t use or pay for the data if it doesn’t meet our criteria for accuracy and privacy protection. Even if it does pass our test, we have only signed up to use it for a year, and then we will evaluate whether it meets our goals. We will only have access to data on how a simulated version of our region’s population travels during an average week. We will not have access to any information on real people’s travel behavior. We take privacy seriously. We appreciate that the media is drawing attention to the privacy concerns raised by data companies’ use of personal information, and people should opt out of sharing their personal and location data if they are concerned. We are doing this as a pilot so that we can understand any implications for citizens’ privacy of working with information that’s based on location data. Our residents are concerned about serious transportation safety and equity issues, and have asked us to get better data so that we can figure out how to best tackle these issues. At the same time, our residents are also increasingly concerned about privacy. This pilot is an opportunity to understand whether there are conflicts between the need for better data and the right to privacy, and if so, how we can address those conflicts. Sidewalk Labs has committed to protecting privacy, and our draft contract ensures that Replica does not include personal information or make it possible to identify any individual. We have not yet signed a contract with Sidewalk Labs, so we have the opportunity to address any additional concerns that are raised.   General guidance for staff who receive questions from the public about Replica: Don’t attempt to speak about the Replica pilot if you don’t feel informed about it. Instead, refer questions to your PIO, or contact the staff managing this project for technical questions about the pilot (Michael Kerr, PBOT; Tim McHugh, TriMet; Eliot Rose, Metro). · Don’t attempt to defend Replica or the companies involved in its creation. This project is an opportunity for us to hear from people about privacy concerns, which we take seriously, and we have the opportunity to address these concerns as we draft an agreement with Sidewalk Labs.     Emphasize the value of better transportation data in improving the safety, equity, and efficiency of the transportation system. Emphasize that Replica is a unique product that provides detailed and comprehensive data on travel patterns while protecting privacy. Acknowledge privacy concerns, commit to relaying them to the Replica project team, and encourage concerned people to take action to protect their privacy. Emphasize that this is a pilot project. We are testing Replica rigorously before we accept the data, and we are only planning to use the data for a year and then decide whether to continue. Coverage so far: The Intercept: https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replicacellphone-data/ OPB: https://www.opb.org/news/article/cellphone-location-data-portland-google-privacy/ BikePortland: https://bikeportland.org/2018/12/05/portland-ponders-pilot-of-powerfultransportation-data-tool-292817 The Wall Street Journal is doing a deep investigation of Replica/Sidewalk Labs and has filed public records requests with Portland and Metro for our correspondence related to this project, as well as interviewing staff. They’re also looking into the ~5 other regions that are exploring this project. Their story should be out in a week or so, and I suspect that it will raise similar concerns to those in the piece by the Intercept. In addition, a waterfront development by Sidewalk Labs in Toronto has met with controversy. That’s not directly related to this project, but it colors people’s perception of it. Eliot Rose Technology Strategist Metro     From: To: Subject: Date: Eliot Rose Martin, Kevin; Kerr, Michael RE: Replica: add"l talking points and guidance Monday, February 25, 2019 5:12:00 PM Kevin, can you please connect us with the ACLU contact you mentioned—assuming you still think that it’s a good idea to speak with her? We’re closing in on a final contract w/ SWL, and I’d ideally like to reach out to her this week to stay on track.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, February 12, 2019 10:34 AM To: Martin, Kevin; 'Kerr, Michael' Subject: RE: Replica: add'l talking points and guidance   Hey team:   I followed up with Nick, who’s been busy, yesterday and he confirmed that this add’l comms guidance is good to go. I’m not sure that we need to loop in add’l Metro folks since all questions about this are going to flow to me or Nick, but below is an updated list of talking points that you can distribute to your teams. I’m planning to share these with TriMet’s PIO and PM too and give them an update via email, but let me know if you’d prefer that I set up a meeting instead, or if there’s anything I else I can do to help support. It seems like PBOT is going to be fielding the lion’s share of questions on these.   Kevin, welcome back! Hope you shredded hard. Can you please connect us with the ACLU contact that you mentioned?   I also followed up on a couple of other items that Mike and I had discussed with Nick from SWL: · The current version of the online Replica tool does allow users to query and download disaggregate data, and Mike was going to check in on whether PBOT staff want access to that data or only want aggregate data. Please let me know how you want to proceed with that. We can ask about turning that feature off, but we’d want to raise that conversation sooner rather than later, and make sure that TriMet is OK with that too. · SWL isn’t willing to share the full privacy audit, but he did let us know who the author is (Jean-Louis Tambay, a Canadian privacy expert; googled hi mand found some publications but didn’t read too deeply yet), and I asked about the possibility of us having a conversation with him. Let me know if you think that would help to follow up on that.   Replica talking points This project is about understanding travel patterns, not individuals. We are exploring new data on travel patterns so that we can better tackle issues like congestion, safety, and transportation equity. The data we are getting through this project are totally private. We are not getting access to any information on real individuals’ travel behavior. We will get a detailed simulation of neighborhood-level travel patterns. This is a super-rich tool. We are going to be able to do things with this information that we would never be able to do with the data that we normally have access to: If we make more space for bikes and pedestrians at an intersection, we can use Replica to examine whether people who wouldn’t otherwise walk or bike are using those facilities. If we increase bus service to East Portland, we can use Replica to assess whether people of color and low-income communities are taking advantage of that service. We can look at where and how people are using ride-hailing to understand where it supports transit and reduces congestion – and where ride-hailing creates more traffic. This project is about saving public resources. We spend a lot of money to get information on travel patterns, often by sending people out to count cars or bicyclists by hand. If this tool works, we can spend fewer public dollars on understanding the transportation system and more on improving it. We get approached by a lot of data vendors, and we’re well aware of the products that are out there. We are working with Replica because it is a unique product that best meets our and our residents’ needs: It covers all modes, including bicycle and pedestrian trips and new modes like ride-hailing. It covers all types of people – including people of color and people living on low incomes, who are typically underrepresented in transportation data – while protecting privacy. It covers why people are traveling, so that we can use the data to meet their needs whether they’re going to work, shopping, school or another destination. It’s available to all transportation agencies in the Portland region, so that we can coordinate to best serve our residents. We get to test Replica before committing to use it. We won’t use or pay for the data if it doesn’t meet our criteria for accuracy and privacy protection. Even if it does pass our test, we have only signed up to use it for a year, and then we will evaluate whether it meets our goals. We will only have access to data on how a simulated version of our region’s population travels during an average week. We will not have access to any information on real people’s travel behavior. We take privacy seriously. We appreciate that the media is drawing attention to the privacy concerns raised by data companies’ use of personal information, and people should opt out of sharing their personal and location data if they are concerned. We are doing this as a pilot so that we can understand any implications for citizens’ privacy of working with information that’s based on location data. Our residents are concerned about serious transportation safety and equity issues, and have asked us to get better data so that we can figure out how to best tackle these issues. At the same time, our residents are also increasingly concerned about privacy. This pilot is an opportunity to understand whether there are conflicts between the need for better data and the right to privacy, and if so, how we can address those conflicts. Sidewalk Labs has committed to protecting privacy, and our draft contract ensures that Replica does not include personal information or make it possible to identify any individual. We have not yet signed a contract with Sidewalk Labs, so we have the opportunity to address any additional concerns that are raised.   General guidance for staff who receive questions from the public about Replica: Don’t attempt to speak about the Replica pilot if you don’t feel informed about it. Instead, refer questions to your PIO, or contact the staff managing this project for technical questions about the pilot (Michael Kerr, PBOT; Tim McHugh, TriMet; Eliot Rose, Metro). · Don’t attempt to defend Replica or the companies involved in its creation. This project is an opportunity for us to hear from people about privacy concerns, which we take seriously, and we have the opportunity to address these concerns as we draft an agreement with Sidewalk Labs.     Emphasize the value of better transportation data in improving the safety, equity, and efficiency of the transportation system. Emphasize that Replica is a unique product that provides detailed and comprehensive data on travel patterns while protecting privacy. Acknowledge privacy concerns, commit to relaying them to the Replica project team, and encourage concerned people to take action to protect their privacy. Emphasize that this is a pilot project. We are testing Replica rigorously before we accept the data, and we are only planning to use the data for a year and then decide whether to continue. Coverage so far: The Intercept: https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replicacellphone-data/ OPB: https://www.opb.org/news/article/cellphone-location-data-portland-google-privacy/ BikePortland: https://bikeportland.org/2018/12/05/portland-ponders-pilot-of-powerfultransportation-data-tool-292817 The Wall Street Journal is doing a deep investigation of Replica/Sidewalk Labs and has filed public records requests with Portland and Metro for our correspondence related to this project, as well as interviewing staff. They’re also looking into the ~5 other regions that are exploring this project. Their story should be out in a week or so, and I suspect that it will raise similar concerns to those in the piece by the Intercept. In addition, a waterfront development by Sidewalk Labs in Toronto has met with controversy. That’s not directly related to this project, but it colors people’s perception of it. Eliot Rose Technology Strategist Metro     From: To: Cc: Subject: Eliot Rose Kerr, Michael; Martin, Kevin Chris Johnson; Robert Kirkman Portland Replica data This call is to clarify the level of detail that will be available through Replica data, both in the online tool and via download/query. Mike and I will be on for sure; Robb, Chris and Kevin are welcome to join if available. From: To: Bcc: Subject: Date: Eliot Rose Martin, Kevin; Kerr, Michael 2days@followupthen.com RE: Replica: add"l talking points and guidance Tuesday, February 12, 2019 10:33:00 AM Hey team:   I followed up with Nick, who’s been busy, yesterday and he confirmed that this add’l comms guidance is good to go. I’m not sure that we need to loop in add’l Metro folks since all questions about this are going to flow to me or Nick, but below is an updated list of talking points that you can distribute to your teams. I’m planning to share these with TriMet’s PIO and PM too and give them an update via email, but let me know if you’d prefer that I set up a meeting instead, or if there’s anything I else I can do to help support. It seems like PBOT is going to be fielding the lion’s share of questions on these.   Kevin, welcome back! Hope you shredded hard. Can you please connect us with the ACLU contact that you mentioned?   I also followed up on a couple of other items that Mike and I had discussed with Nick from SWL: · The current version of the online Replica tool does allow users to query and download disaggregate data, and Mike was going to check in on whether PBOT staff want access to that data or only want aggregate data. Please let me know how you want to proceed with that. We can ask about turning that feature off, but we’d want to raise that conversation sooner rather than later, and make sure that TriMet is OK with that too. · SWL isn’t willing to share the full privacy audit, but he did let us know who the author is (Jean-Louis Tambay, a Canadian privacy expert; googled hi mand found some publications but didn’t read too deeply yet), and I asked about the possibility of us having a conversation with him. Let me know if you think that would help to follow up on that.   Replica talking points This project is about understanding travel patterns, not individuals. We are exploring new data on travel patterns so that we can better tackle issues like congestion, safety, and transportation equity. The data we are getting through this project are totally private. We are not getting access to any information on real individuals’ travel behavior. We will get a detailed simulation of neighborhood-level travel patterns. This is a super-rich tool. We are going to be able to do things with this information that we would never be able to do with the data that we normally have access to: If we make more space for bikes and pedestrians at an intersection, we can use Replica to examine whether people who wouldn’t otherwise walk or bike are using those facilities. If we increase bus service to East Portland, we can use Replica to assess whether people of color and low-income communities are taking advantage of that service. We can look at where and how people are using ride-hailing to understand where it supports transit and reduces congestion – and where ride-hailing creates more traffic. This project is about saving public resources. We spend a lot of money to get information on travel patterns, often by sending people out to count cars or bicyclists by hand. If this tool works, we can spend fewer public dollars on understanding the transportation system and more on improving it. We get approached by a lot of data vendors, and we’re well aware of the products that are out there. We are working with Replica because it is a unique product that best meets our and our residents’ needs: It covers all modes, including bicycle and pedestrian trips and new modes like ride-hailing. It covers all types of people – including people of color and people living on low incomes, who are typically underrepresented in transportation data – while protecting privacy. It covers why people are traveling, so that we can use the data to meet their needs whether they’re going to work, shopping, school or another destination. It’s available to all transportation agencies in the Portland region, so that we can coordinate to best serve our residents. We get to test Replica before committing to use it. We won’t use or pay for the data if it doesn’t meet our criteria for accuracy and privacy protection. Even if it does pass our test, we have only signed up to use it for a year, and then we will evaluate whether it meets our goals. We will only have access to data on how a simulated version of our region’s population travels during an average week. We will not have access to any information on real people’s travel behavior. We take privacy seriously. We appreciate that the media is drawing attention to the privacy concerns raised by data companies’ use of personal information, and people should opt out of sharing their personal and location data if they are concerned. We are doing this as a pilot so that we can understand any implications for citizens’ privacy of working with information that’s based on location data. Our residents are concerned about serious transportation safety and equity issues, and have asked us to get better data so that we can figure out how to best tackle these issues. At the same time, our residents are also increasingly concerned about privacy. This pilot is an opportunity to understand whether there are conflicts between the need for better data and the right to privacy, and if so, how we can address those conflicts. Sidewalk Labs has committed to protecting privacy, and our draft contract ensures that Replica does not include personal information or make it possible to identify any individual. We have not yet signed a contract with Sidewalk Labs, so we have the opportunity to address any additional concerns that are raised.   General guidance for staff who receive questions from the public about Replica: Don’t attempt to speak about the Replica pilot if you don’t feel informed about it. Instead, refer questions to your PIO, or contact the staff managing this project for technical questions about the pilot (Michael Kerr, PBOT; Tim McHugh, TriMet; Eliot Rose, Metro). · Don’t attempt to defend Replica or the companies involved in its creation. This project is an opportunity for us to hear from people about privacy concerns, which we take seriously, and we have the opportunity to address these concerns as we draft an agreement with Sidewalk Labs.     Emphasize the value of better transportation data in improving the safety, equity, and efficiency of the transportation system. Emphasize that Replica is a unique product that provides detailed and comprehensive data on travel patterns while protecting privacy. Acknowledge privacy concerns, commit to relaying them to the Replica project team, and encourage concerned people to take action to protect their privacy. Emphasize that this is a pilot project. We are testing Replica rigorously before we accept the data, and we are only planning to use the data for a year and then decide whether to continue. Coverage so far: The Intercept: https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replicacellphone-data/ OPB: https://www.opb.org/news/article/cellphone-location-data-portland-google-privacy/ BikePortland: https://bikeportland.org/2018/12/05/portland-ponders-pilot-of-powerfultransportation-data-tool-292817 The Wall Street Journal is doing a deep investigation of Replica/Sidewalk Labs and has filed public records requests with Portland and Metro for our correspondence related to this project, as well as interviewing staff. They’re also looking into the ~5 other regions that are exploring this project. Their story should be out in a week or so, and I suspect that it will raise similar concerns to those in the piece by the Intercept. In addition, a waterfront development by Sidewalk Labs in Toronto has met with controversy. That’s not directly related to this project, but it colors people’s perception of it. Eliot Rose Technology Strategist Metro     From: To: Subject: Date: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; Gilligan, Mike; aubrey.lindstrom@portlandoregon.gov; McDonald, Mike RE: Replica pilot: validation Thursday, February 07, 2019 12:31:00 PM Hello all: Thanks for participating in the validation call today. We’re going to reconvene the week of 2/18 to review a more detailed version of the Replica criteria along the lines of what’s in the template that Sidewalk Labs provided that Chris will draft with input from PBOT and TriMet. Please let us know who from your agencies needs to participate in that call. Here are some of the other next steps that we discussed on the call today: · TriMet/Portland: email Chris (chris.johnson@oregonmetro.gov) with suggestions/concerns/questions about the validation criteria that are in your court.  · Portland/Metro: think about criteria w/ respect to privacy. · Eliot: email Nick to ask about withholding data, ped data detail, etc. Eliot Rose Technology Strategist Metro _____________________________________________ From: Chris Johnson Sent: Thursday, February 07, 2019 8:55 AM To: Eliot Rose; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim Subject: RE: Replica pilot: validation Hi allAttached are some draft validation criteria that I plan to walk through during today’s call. I’ve also attached some supporting documentation that will give you some context of how we approach validation in the travel modeling realm. I’m looking forward to this morning’s discussion. Thanks! -Chris  << File: DRAFT Replica Acceptance Criteria.xlsx >>  << File: Kate_Validation_Report_August2017.pdf >>  << File: fhwa model validation handbook.pdf >> -----Original Appointment----From: Eliot Rose Sent: Monday, February 4, 2019 11:25 AM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim Subject: Replica pilot: validation When: Thursday, February 7, 2019 11:00 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro: Chris' office / (605) 472-5418, Access code: 616836 Agenda: · Introductions (5 min.) · Overview of acceptance process: goals, responsibilities, timeline (Eliot, 10 min.) · Review and discuss draft validation criteria (Chris, 40 min.) · Next steps (5 min.) From: To: Cc: Subject: Date: Attachments: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim; aubrey.lindstrom@portlandoregon.gov McDonald, Mike RE: Replica pilot: validation Thursday, February 07, 2019 11:17:00 AM ExB Replica Example Acceptance Criteria.pdf Calibration Report – Replica KC Jan-Mar 2018 v1.0 (2).pdf Attached is the template for the Replica acceptance criteria that Sidewalk labs originally provided us. Eliot Rose Technology Strategist Metro -----Original Appointment----From: Eliot Rose Sent: Monday, February 04, 2019 11:25 AM To: Eliot Rose; Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim Cc: McDonald, Mike Subject: Replica pilot: validation When: Thursday, February 07, 2019 11:00 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro: Chris' office / (605) 472-5418, Access code: 616836 Agenda: · Introductions (5 min.) · Overview of acceptance process: goals, responsibilities, timeline (Eliot, 10 min.) · Review and discuss draft validation criteria (Chris, 40 min.) · Next steps (5 min.) From: To: Subject: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman; McHugh, Tim Replica pilot: validation Agenda: * Introductions (5 min.) * Overview of acceptance process: goals, responsibilities, timeline (Eliot, 10 min.) * Review and discuss draft validation criteria (Chris, 40 min.) * Next steps (5 min.) From: To: Cc: Subject: Date: Attachments: Eliot Rose Nick Christensen Martin, Kevin; Kerr, Michael Replica: add"l talking points and guidance Tuesday, February 05, 2019 5:34:00 PM FW Loop in comms people re Replica.msg Hello Nick:   I just talked with Portland (Kevin Martin and Mike Kerr, cc’d) about some of the questions that their staff fielded from community groups about Sidewalk when they were discussing the new privacy policy that they’re developing with the public. People continued to raise the question that the WSJ raised about whether we are endorsing the use of cell phone data that people feel has been collected without adequate consent by working with Sidewalk Labs, and also about the fact that we are sole-sourcing this. We feel like we want to add a bit to the talking points that we’ve drafted so far (see attached) to cover that concern, and also include some general guidance for technical staff that are likely to field questions on Replica as Portland’s privacy policy comes up. Can you please take a look at the items below and let me know if you have any suggested edits or additions?   Also, we’re getting closer to a signed contract, and it would be nice to time that after the WSJ story drops. Would it be appropriate to reach out to Douglas and ask him for an update on the timing? (Assuming we can reach him; he’s changed jobs).   Thanks, Eliot   Add’l talking points: · We get approached by a lot of data vendors, and we’re well aware of the products that are out there. We are working with Replica because it is a unique product that best meets our needs: o It covers all modes, including bicycle and pedestrian trips and new modes like ridehailing. o It covers all types of people – including people of color and people living on low incomes, who are typically underrepresented in transportation data – while protecting privacy. o It covers why people are traveling, so that we can use the data to meet their needs whether they’re going to work, shopping, school or another destination. o It’s available to all transportation agencies in the Portland region, so that we can coordinate to best serve our residents. o We get to test Replica before committing to use it. We won’t use or pay for the data if it doesn’t meet our criteria for accuracy and privacy protection. Even if it does pass our test, we have only signed up to use it for a year, and then we will evaluate whether it meets our goals. · Our residents are concerned about serious transportation safety and equity issues, and have asked us to get better data so that we can figure out how to best tackle these issues. At the same time, our residents are also increasingly concerned about privacy. This pilot is an opportunity to understand whether there are conflicts between the need for better data and the right to privacy, and if so, how we can address those conflicts.   General guidance for staff who receive questions from the public about Replica: Don’t attempt to speak about the Replica pilot if you don’t feel informed about it. Instead, refer questions to your PIO, or contact the staff managing this project for technical questions about the pilot (Michael Kerr, PBOT; Tim McHugh, TriMet; Eliot Rose, Metro). Emphasize the value of better transportation data in improving the safety, equity, and efficiency of the transportation system. Emphasize that Replica is a unique product that provides detailed and comprehensive data on travel patterns while protecting privacy. Emphasize that this is a pilot project. We are testing Replica rigorously before we accept the data, and we are only planning to use the data for a year and then decide whether to continue. Acknowledge that privacy concerns are valid, and encourage people to take action to protect their privacy. Don’t defend Google or Sidewalk Labs. Note that we take privacy concerns seriously, and have the opportunity to address these concerns as we draft an agreement with Sidewalk Labs. Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Attachments: Eliot Rose Kerr, Michael; Martin, Kevin FW: Loop in comms people re: Replica Tuesday, February 05, 2019 3:40:00 PM Council memo - media coverage of Replica pilot.docx     Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Tuesday, January 29, 2019 4:54 PM To: 'Altstadt, Roberta' Subject: RE: Loop in comms people re: Replica   Hello Roberta! It’s great to hear from you, and I apologize for not looping you in sooner.   We’ve already been coordinating with Portland on messaging because they’ve been the first responder to most of the public/media inquiries, and they continue to get comments from concerned members of the public. I’m meeting with their staff next week to discuss whether there’s a need to take a more proactive approach to addressing these concerns, and if so we’ll loop in PIOs. In the meantime, I’ll give you an email rundown of what we’ve heard and done so far, and I’m happy to talk this week if you’d like to get up to speed—this is heady stuff.   I’m attaching a background memo that we prepared for Metro’s Council, and listing our talking points and coverage to date below. Can you please make sure that TriMet staff who might get questions on Replica are up to speed on talking points and that your team knows to coordinate with PIOs at Portland (John.Brady@portlandoregon.gov) and Metro (Nick.Christensen@oregonmetro.gov) if you get any media inquiries?   Thanks, Eliot   Talking points (Our PIOs drafted these with Portland’s back when their Council first heard the IGA to fund Replica in December): This project is about understanding travel patterns, not individuals. We are exploring new data on travel patterns so that we can better tackle issues like congestion, safety, and transportation equity. The data we are getting through this project are totally private. We are not getting access to any information on real individuals’ travel behavior. We will get a detailed simulation of neighborhood-level travel patterns.   This is a super-rich tool. We are going to be able to do things with this information that we would never be able to do with the data that we normally have access to: If we make more space for bikes and pedestrians at an intersection, we can use Replica to examine whether people who wouldn’t otherwise walk or bike are using those facilities. If we increase bus service to East Portland, we can use Replica to assess whether people of color and low-income communities are taking advantage of that service. We can look at where and how people are using ride-hailing to understand where it supports transit and reduces congestion – and where ride-hailing creates more traffic. This project is about saving public resources. We spend a lot of money to get information on travel patterns, often by sending people out to count cars or bicyclists by hand. If this tool works, we can spend fewer public dollars on understanding the transportation system and more on improving it. We will only have access to data on how a simulated version of our region’s population travels during an average week. We will not have access to any information on real people’s travel behavior. We take privacy seriously. We appreciate that the media is drawing attention to the privacy concerns raised by data companies’ use of personal information, and people should opt out of sharing their personal and location data if they are concerned. We are doing this as a pilot so that we can understand any implications for citizens’ privacy of working with information that’s based on location data. Sidewalk Labs has committed to protecting privacy, and our draft contract ensures that Replica does not include personal information or make it possible to identify any individual. We have not yet signed a contract with Sidewalk Labs, so we have the opportunity to address any additional concerns that are raised.   Coverage so far: ·         The Intercept: https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replicacellphone-data/ ·         OPB: https://www.opb.org/news/article/cellphone-location-data-portland-google-privacy/ ·         BikePortland: https://bikeportland.org/2018/12/05/portland-ponders-pilot-of-powerfultransportation-data-tool-292817 ·         The Wall Street Journal is doing a deep investigation of Replica/Sidewalk Labs and has filed public records requests with Portland and Metro for our correspondence related to this project, as well as interviewing staff. They’re also looking into the ~5 other regions that are exploring this project. Their story should be out in a week or so, and I suspect that it will raise similar concerns to those in the piece by the Intercept. ·         In addition, a waterfront development by Sidewalk Labs in Toronto has met with controversy. That’s not directly related to this project, but it colors people’s perception of it.     Eliot Rose Technology Strategist Metro   From: Altstadt, Roberta [mailto:altstadr@TriMet.org] Sent: Tuesday, January 29, 2019 3:47 PM To: Eliot Rose Subject: FW: Loop in comms people re: Replica   Good afternoon Eliot, I would agree that we need to get together and align our messaging. When are you thinking?   Roberta   Roberta Altstadt Media Relations & Communications Manager Office: 503-962-5669  Email: altstadr@trimet.org PIO team: pio@trimet.org or 503-273-7402   Begin forwarded message: From: Eliot Rose Date: January 27, 2019 at 4:58:56 PM PST To: "McHugh, Tim" Subject: Loop in comms people re: Replica Hello Tim:   Portland continues to get tough questions from community about Replica and privacy. I was talking with Kevin Martin and we feel like it’s time to get the communications folks together to refine our talking points and talk about some more proactive approaches to communicating why we’re pursuing this tool. Who’s a good comms person to loop in to help make sure that TriMet staff are prepared to answer any questions from community or the media about Replica? I chatted with Jeff Owen about this and he suggested reaching out to Roberta Altstadt.   Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Subject: Date: Eliot Rose Martin, Kevin RE: Replica pilot: validation Monday, February 04, 2019 3:06:00 PM Catch some air out there! Eliot Rose Technology Strategist Metro -----Original Appointment----From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Monday, February 04, 2019 2:58 PM To: Eliot Rose Subject: Tentative: Replica pilot: validation When: Thursday, February 07, 2019 11:00 AM-12:00 PM (UTC-08:00) Pacific Time (US & Canada). Where: Metro: Chris' office / (605) 472-5418, Access code: 616836 I'll be out (Mt. Bachelor), but holding on my calendar to remind me to follow up with you all after the meeting. Thanks Eliot! Kevin From: To: Subject: Eliot Rose Chris Johnson; Kerr, Michael; Martin, Kevin; Robert Kirkman Replica pilot: validation From: To: Cc: Bcc: Subject: Date: Attachments: Eliot Rose Kerr, Michael; McHugh, Tim; Ken.McGair@portlandoregon.gov; Martin, Kevin Michelle Bellia; Chris Johnson; Robert Kirkman 2days@followupthen.com RE: Replica IGA for signature and next steps Friday, February 01, 2019 5:33:00 PM SaaSPortland Comparison Metro 20190201.docx Hello all:   Thanks for submitting your signed Replica IGAs. In the meantime, here is the draft contract for your review. Per the IGA, PBOT and TriMet have two weeks to review the draft contract; we’ll be expecting feedback by COB 2/15. PBOT folks, I haven’t received any responses from you on the scheduling polls linked below—and Tim, I haven’t seen any response from Erik or other TriMet attorneys. Please fill the polls out and pass the links along to anyone else who needs to attend on your end ASAP so that we can find time to talk next week; I’m planning to finalize a time based on whoever has responded by Monday at noon.   This contract has been through extensive edits—too many to track without making a mess of things. The most significant changes, which we’ll summarize on the contracting call, primarily focused on protecting our intended use cases, ensuring protection of PII by both SWL and Metro, and giving us access to data via export and download in addition to via the online tool. All changes agreed to by SWL have been accepted; only those that remain outstanding are tracked in the current version. The contract does not include a draft of the acceptance criteria; we will be sharing some ideas when we hold the call to discuss those.   Have a great weekend, and I look forward to chatting soon. -Eliot   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, January 25, 2019 1:37 PM To: 'Kerr, Michael'; 'McHugh, Tim' Cc: Michelle Bellia; Chris Johnson; Robert Kirkman Subject: RE: Replica IGA for signature and next steps   Hello all:   Just a reminder to please fill out the polls below so that we can get moving on next steps once we get the IGA signed. I adjusted the time slots to account for the delays in signing the IGA.   Contracting call (https://doodle.com/poll/47accz5gghhx6vvf): TriMet and Portland have two weeks from signing the IGA to review the draft contract with SWL and provide feedback to Metro. We’ve been going back and forth with SWL to make sure that we have a contract that protects both our use cases and our residents’ privacy, and we feel good about where the draft is at. We’ll give a summary of the conversation and the changes so far to facilitate your review. Please include project managers and attorneys.   Validation process (https://doodle.com/poll/52356das6txt95rb): SWL has already been developing a draft Replica for our region. Once we sign the contract we’ll receive the draft data and our first step will be to validate it against acceptance criteria. Portland is responsible for validating the TNC and bike/ped data and TriMet is responsible for validating transit data. We’ll share draft acceptance criteria and discuss the plan for the validation process. Please include the people who oversee the relevant data and those who will be managing it as part of the validation process, plus project managers if those folks need additional background on the project.       Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Wednesday, January 16, 2019 2:44 PM To: 'Kerr, Michael'; 'McHugh, Tim' Cc: Michelle Bellia; Chris Johnson; Robert Kirkman Subject: Replica IGA for signature and next steps   Hello Mike and Tim:   Here’s the Replica IGA for signature. You can each sign and return individual copies and then we’ll countersign. We’d like to get a signed IGA in place by the end of next week at the latest; please let me know if we’ll need more time. Please make sure that you also pass on the info below about follow-up calls so that we can keep this project moving.   There have been a couple of changes made since Portland’s City Council approved the IGA, and I’ve attached a version with those tracked to make it easy for you to review them as well as a clean version for signature. I’ve discussed the more significant changes with both of you – clarifying the acceptance process and smoothing out the payment schedule – so there shouldn’t be any surprises, but we also updated the timeline and term of the agreement and corrected a few minor errors. Mike, note that those involve two minor changes to the front matter of the IGA; please let me know if you have any concerns about that.   Once we get this signed, we’d like to hold a call to discuss next steps. Or rather, two separate calls with the relevant staff in your organizations, one to discuss the contract and one on the validation process, so that we make the best use of people’s times. Please forward this to the relevant folks in your organizations and have them fill out the associated polls:   Contracting call (https://doodle.com/poll/47accz5gghhx6vvf): TriMet and Portland have two weeks from signing the IGA to review the draft contract with SWL and provide feedback to Metro. We’ve been going back and forth with SWL to make sure that we have a contract that protects both our use cases and our residents’ privacy, and we feel good about where the draft is at. We’ll give a summary of the conversation and the changes so far to facilitate your review. Please include project managers and attorneys.   Validation process (https://doodle.com/poll/52356das6txt95rb): SWL has already been developing a draft Replica for our region. Once we sign the contract we’ll receive the draft data and our first step will be to validate it against acceptance criteria. Portland is responsible for validating the TNC and bike/ped data and TriMet is responsible for validating transit data. We’ll share draft acceptance criteria and discuss the plan for the validation process. Please include the people who oversee the relevant data and those who will be managing it as part of the validation process, plus project managers if those folks need additional background on the project.     Thanks, Eliot   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: Eliot Rose To: MarIin, Kevin Subject: Accepted: Sidewalk Labs discussion From: To: Subject: Date: Eliot Rose Martin, Kevin; Kerr, Michael RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Tuesday, January 29, 2019 9:05:00 AM 3 at Congress Center works. I’ve got a series of meetings in SW that afternoon, so it’s convenient for me to stick around over there.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Tuesday, January 29, 2019 12:54 AM To: Eliot Rose; Kerr, Michael Subject: Re: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Great, all good points below.   How about later next Tuesday 2/5, 3 or 3:30pm? If that works, do you two have a preference of venue? Congress Center and Metro are both close to my ride home, so happy to visit either!   kevin martin (he/him)  smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com From: Eliot Rose Sent: Monday, January 28, 2019 3:14:33 PM To: Kerr, Michael; Martin, Kevin Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Great—I love talking with you guys about this stuff. Next week is pretty open for me.   Agree that it’s not necessary to invite John, but it does seem like we need to make sure we have a plan to get a consistent message out to other PIOs and staff who might field questions before we get any other tough ones. I’m getting TriMet’s PIO up to speed this week, and y’all should feel free to forward the points/memo I’ve sent to any Smart Cities Team members or other CoP staff who might get questions. Please let me know if I can help there.   Can you also please ask CoP comms staff give Metro (myself or Nick Christensen, NickC@oregonmetro.gov) the heads-up if they get further inquiries? Thanks for being on the front lines!   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 1:52 PM To: Martin, Kevin; Eliot Rose Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Let’s get ourselves together and aligned first.  I can make time for this next week, no problem.       Agreed that Eliot’s points are dead on.  Now it’s a matter of educating and sharing concrete examples of how what we do today isn’t all that different.  To the consent issue… it’s a point we’ve made in interviews (that Cities are being held to task for an issue that needs to get resolved Federally), but we definitely need to speak up about how we (Cities) are stepping into the void and doing everything we can to protect people’s privacy.   Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Martin, Kevin Sent: Monday, January 28, 2019 1:45 PM To: Eliot Rose ; Kerr, Michael Subject: Re: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Well said.   I think we need to get in a room sometime in the next few weeks and Repeating myself, but it’s going to be critical that we’re all on page with how w address questions and concerns around SL, and — on side — how it overlaps with our Privacy Resolution. Clearly all of struck a media nerve.   dig in. the same the City this has Think we can find an hour next week or the week after? Eliot, I can check Mike’s schedule and float a couple of times? Mike, do we need someone Ike John Brady in that room, or should we talk first?   Thanks, Kevin   kevin martin (he/him) smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com   From: Eliot Rose Sent: Monday, January 28, 2019 12:58 PM To: Kerr, Michael; Martin, Kevin Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Thanks - I'll pass this around to folks on our end. I think that the response to the add'l concern that you/the article raised about reverseengineering is that we get pitched on other products derived from cell data all the time and that the method that SWL is using - simulating travel based on the data - is a more compelling way to protect privacy. It's what we've used for years to protect the confidentiality of our travel survey respondents. This article makes it sound like Replica is the first to offer any of this data and the first time we've considered any of these issues. Seems like we need to emphasize our experience here and how it's led us to work w/ Replica. The issue that the article (and Doug from WSJ) raises about whether the data that informs Replica has been obtained with consent is one I want to dig into further. I feel like that's a real concern, and that it's also out of the league of local governments to address. The way we discussed it with WSJ is that we hear concerns about privacy from our citizens, and at the same time we hear a very strong demand for us to use a data-driven approach to make the streets safer. This pilot with Replica is an opportunity to understand if those goals conflict and if so, how we can balnce those competing concerns. Eliot Rose Technology Strategist Metro -----Original Message----From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 8:48 AM To: Eliot Rose; Martin, Kevin Subject: FW: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones The story is evolving... https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replicacellphone-data/ This one is more in line with what I'd imagine the WSJ is writing and, while the focus is primarily on SL, it's making cities look as if they're blindly entering into these agreements. To add to that, this article raises a new (but not unpredictable) concern that if SL can disaggregate and synthesize, why couldn't someone else un-synthesize and re-aggregate. Look forward to piecing together a plan for proactively working through these concerns. Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg -----Original Message----From: Kerr, Michael Sent: Monday, January 28, 2019 6:33 AM To: Kerr, Michael Subject: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Sent from my iPhone From: To: Subject: Date: Eliot Rose Kerr, Michael; Martin, Kevin RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Monday, January 28, 2019 3:14:00 PM Great—I love talking with you guys about this stuff. Next week is pretty open for me.   Agree that it’s not necessary to invite John, but it does seem like we need to make sure we have a plan to get a consistent message out to other PIOs and staff who might field questions before we get any other tough ones. I’m getting TriMet’s PIO up to speed this week, and y’all should feel free to forward the points/memo I’ve sent to any Smart Cities Team members or other CoP staff who might get questions. Please let me know if I can help there.   Can you also please ask CoP comms staff give Metro (myself or Nick Christensen, NickC@oregonmetro.gov) the heads-up if they get further inquiries? Thanks for being on the front lines!   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 1:52 PM To: Martin, Kevin; Eliot Rose Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Let’s get ourselves together and aligned first.  I can make time for this next week, no problem.       Agreed that Eliot’s points are dead on.  Now it’s a matter of educating and sharing concrete examples of how what we do today isn’t all that different.  To the consent issue… it’s a point we’ve made in interviews (that Cities are being held to task for an issue that needs to get resolved Federally), but we definitely need to speak up about how we (Cities) are stepping into the void and doing everything we can to protect people’s privacy.   Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Martin, Kevin Sent: Monday, January 28, 2019 1:45 PM To: Eliot Rose ; Kerr, Michael Subject: Re: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Well said.   I think we need to get in a room sometime in the next few weeks and dig in. Repeating myself, but it’s going to be critical that we’re all on the same page with how w address questions and concerns around SL, and — on the City side — how it overlaps with our Privacy Resolution. Clearly all of this has struck a media nerve.   Think we can find an hour next week or the week after? Eliot, I can check Mike’s schedule and float a couple of times? Mike, do we need someone Ike John Brady in that room, or should we talk first?   Thanks, Kevin   kevin martin (he/him) smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com   From: Eliot Rose Sent: Monday, January 28, 2019 12:58 PM To: Kerr, Michael; Martin, Kevin Subject: RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones   Thanks - I'll pass this around to folks on our end. I think that the response to the add'l concern that you/the article raised about reverseengineering is that we get pitched on other products derived from cell data all the time and that the method that SWL is using - simulating travel based on the data - is a more compelling way to protect privacy. It's what we've used for years to protect the confidentiality of our travel survey respondents. This article makes it sound like Replica is the first to offer any of this data and the first time we've considered any of these issues. Seems like we need to emphasize our experience here and how it's led us to work w/ Replica. The issue that the article (and Doug from WSJ) raises about whether the data that informs Replica has been obtained with consent is one I want to dig into further. I feel like that's a real concern, and that it's also out of the league of local governments to address. The way we discussed it with WSJ is that we hear concerns about privacy from our citizens, and at the same time we hear a very strong demand for us to use a data-driven approach to make the streets safer. This pilot with Replica is an opportunity to understand if those goals conflict and if so, how we can balnce those competing concerns. Eliot Rose Technology Strategist Metro -----Original Message----From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 8:48 AM To: Eliot Rose; Martin, Kevin Subject: FW: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones The story is evolving... https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replicacellphone-data/ This one is more in line with what I'd imagine the WSJ is writing and, while the focus is primarily on SL, it's making cities look as if they're blindly entering into these agreements. To add to that, this article raises a new (but not unpredictable) concern that if SL can disaggregate and synthesize, why couldn't someone else un-synthesize and re-aggregate. Look forward to piecing together a plan for proactively working through these concerns. Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg -----Original Message----From: Kerr, Michael Sent: Monday, January 28, 2019 6:33 AM To: Kerr, Michael Subject: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Sent from my iPhone From: To: Subject: Date: Eliot Rose Kerr, Michael; Martin, Kevin RE: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Monday, January 28, 2019 12:53:00 PM Thanks - I'll pass this around to folks on our end. I think that the response to the add'l concern that you/the article raised about reverse-engineering is that we get pitched on other products derived from cell data all the time and that the method that SWL is using - simulating travel based on the data - is a more compelling way to protect privacy. It's what we've used for years to protect the confidentiality of our travel survey respondents. This article makes it sound like Replica is the first to offer any of this data and the first time we've considered any of these issues. Seems like we need to emphasize our experience here and how it's led us to work w/ Replica. The issue that the article (and Doug from WSJ) raises about whether the data that informs Replica has been obtained with consent is one I want to dig into further. I feel like that's a real concern, and that it's also out of the league of local governments to address. The way we discussed it with WSJ is that we hear concerns about privacy from our citizens, and at the same time we hear a very strong demand for us to use a data-driven approach to make the streets safer. This pilot with Replica is an opportunity to understand if those goals conflict and if so, how we can balnce those competing concerns.  Eliot Rose Technology Strategist Metro -----Original Message----From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, January 28, 2019 8:48 AM To: Eliot Rose; Martin, Kevin Subject: FW: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones The story is evolving... https://theintercept.com/2019/01/28/google-alphabet-sidewalk-labs-replica-cellphone-data/ This one is more in line with what I'd imagine the WSJ is writing and, while the focus is primarily on SL, it's making cities look as if they're blindly entering into these agreements.  To add to that, this article raises a new (but not unpredictable) concern that if SL can disaggregate and synthesize, why couldn't someone else un-synthesize and re-aggregate. Look forward to piecing together a plan for proactively working through these concerns.    Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-8235185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg -----Original Message----- From: Kerr, Michael Sent: Monday, January 28, 2019 6:33 AM To: Kerr, Michael Subject: Google’s Sidewalk Labs Plans to Package and Sell Location Data on Millions of Cellphones Sent from my iPhone From: To: Cc: Subject: Date: Attachments: Eliot Rose Martin, Kevin Kerr, Michael RE: Meet w/ privacy advocates re: Replica? Sunday, January 27, 2019 4:51:00 PM FW Replica communications planning.msg Thanks Kevin. Happy to strategize further—let’s make sure that we chat at TAO tomorrow; I’m happy to meet beforehand if you’d like. I think it’s a good idea to get the comms folks together, but I’d like to make sure that we have all the right comms people at the table and we have a clear ask to make of them.   We did work with both Metro and PBOT comms folks to draft some talking points that were the basis for the memo I sent (initial draft of the talking points is attached). Sounds like we need to loop in BPS folks as well as technical staff who might face questions about Replica, and we should also engage TriMet comms staff—I’ve been keeping them in the loop but haven’t talked with them directly yet. Anyone else?   I know that our comms folks were going to reach out to PBOT’s about coordinating before any further media inquiries, and they’re ready to respond, but we haven’t discussed anything more proactive. I think a more proactive approach is a great idea but don’t know who the stakeholders are, so I’m glad to hear that it sounds like you’ve got some thoughts on whom to reach out to.   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Friday, January 25, 2019 5:12 PM To: Eliot Rose Cc: Kerr, Michael Subject: Re: Meet w/ privacy advocates re: Replica? Thanks Eliot.   I think it would be good for us to meet and strategize. We're anticipating more questions related to the SL agreement as we start talking to community groups/media about the privacy principles resolution we are bring to City Council on 2/27 (or early March.... Mayor may take a vacation and force us to push the date a week or two). Imagine you two already have a good set of responses drafted because of the media frenzy a while back (maybe this memo is it)?  But I think it would be worth huddling to discuss, and then -- yes -- maybe some proactive outreach to community groups, other organizations. I want everyone involved in the privacy conversations on the City side to go into all rooms prepared to respond to questions about SL, and I want them to be saying the same thing you all are saying. Hector and Christine were caught a bit off guard by the number of questions when they were at PNCA last weekend. Absolutely do not want that happening in front of Council, especially with new Council members who were not there for the discussion back in December.   How's that sound? Should we include comms folks?   I will be at TAO, we can chat about this then also!   Kevin   kevin martin (he/him)  smart city pdx/tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 https://www.smartcitypdx.com From: Eliot Rose Sent: Friday, January 25, 2019 1:53:23 PM To: Martin, Kevin Cc: Kerr, Michael Subject: Meet w/ privacy advocates re: Replica?   Hello Kevin:   I heard from Mike that the Smart City PDX heard some concerns from advocates about the Replica project. Would it help at all for us to meet individually with some of those organizations to try and get out in front of their concerns? Our gov’t affairs team suggested that when I briefed our council on this, but I defer to you on what’s a good idea—I imagine that you have a pretty good idea from the sensor conversation about how that talk would go.   In case it helps, I’m attaching a memo we prepared for our Council to help prep them for any media inquiries or concerned citizens. Happy to talk more at the TAO meeting on Monday if you’re planning to be there.   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Attachments: Eliot Rose Martin, Kevin Kerr, Michael Meet w/ privacy advocates re: Replica? Friday, January 25, 2019 1:53:00 PM Council memo - media coverage of Replica pilot.docx Hello Kevin:   I heard from Mike that the Smart City PDX heard some concerns from advocates about the Replica project. Would it help at all for us to meet individually with some of those organizations to try and get out in front of their concerns? Our gov’t affairs team suggested that when I briefed our council on this, but I defer to you on what’s a good idea—I imagine that you have a pretty good idea from the sensor conversation about how that talk would go.   In case it helps, I’m attaching a memo we prepared for our Council to help prep them for any media inquiries or concerned citizens. Happy to talk more at the TAO meeting on Monday if you’re planning to be there.   Eliot Rose Technology Strategist   Metro 600 NE Grand Ave. Portland, OR 97232-2736 503-797-1825 www.oregonmetro.gov   Metro Making a great place   From: To: Cc: Subject: Date: Eliot Rose Kerr, Michael Martin, Kevin RE: Update - SL Monday, January 14, 2019 10:29:00 AM PS – I didn’t respond to your item about bike/ped counts. A couple of thoughts there. First, it’s ideal to validate on locations where you have continuous counts set up, like the bridges, but you can also use locations where you conduct periodic manual/sensor counts – you’ll just have to allow for a larger margin of error in the validation. Second, Metro also conducts annual bike/ped counts on our trails that we might be able to use in the validation process if you’d like to expand the number of locations at play.   Also, I talked to the WSJ last week and briefed our leadership on the potential issues that their article and other media coverage might raise. They requested that you please let us know ASAP if you receive any other media inquiries. I’ll be prepping a briefing sheet for our councilors that incorporates the talking points that we drafted together that I’ll share as well in case it’s useful.   One other question that they raised (which may be more of a question for Kevin): I recall that PBOT reached out to the ACLU and potentially other local privacy advocates over the sensor deployments on 122nd / Division / Hawthorne. Would it be worth reaching out to those folks about this project as well to see if they have any privacy concerns that we may want to address in the contract?   As always, happy to chat if that’s easier.   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Friday, January 11, 2019 4:10 PM To: 'Kerr, Michael' Subject: RE: Update - SL   Thanks! We should have the IGA ready for you to sign early next week. We’ll be looking to you to sign first. Once you sign we’ll turn over the contract, which is in near-final state as far as we’re concerned, for you to review. I you’d like we can have a call to get you up to speed on the issues that we’ve worked through so far; it might help you save time on your review.   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Friday, January 11, 2019 11:33 AM To: Eliot Rose Subject: Update - SL   Eliot,   Quick update – meeting with those “in the know” about each of our areas of validation responsibility between M-Wed of next week.  The general question is, how to equip ourselves with data we can validate Replica against for both Peds and Bikes (if you were unaware, we’re data poor for those two modes… which is why SL was so appealing to begin with).  For TNCs it’s different story – we should be able to spin up a validation approach using the data we receive today – I’ll know for sure next week.    As for the Trimet edit to the IGA, I’m fine with it.    So, expect to hear from me late next week about final steps involved in getting the IGA signed.     Questions, let me know.   Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: To: Cc: Subject: Date: Eliot Rose Hesse, Eric; Kerr, Michael; Pearce, Art; Siegel, Noah; Hurley, Peter; Martin, Kevin Patton, Jeramy; Warner, Chris RE: Sidewalk Labs - Council Wednesday, December 12, 2018 11:10:00 AM Yes, way to go, Mike!   And on top of that, OPB ran its story on the project this morning and the Metro folks who heard it all felt that it was a balanced story that highlighted the benefits of this project while presenting the privacy concerns appropriately in context.  So our work yesterday to coordinate talking points paid off; both Portland and Metro staff got quoted. The story is not yet posted to OPB’s website.   Eliot Rose Technology Strategist Metro   From: Hesse, Eric [mailto:Eric.Hesse@portlandoregon.gov] Sent: Wednesday, December 12, 2018 10:46 AM To: Kerr, Michael; Pearce, Art; Siegel, Noah; Hurley, Peter; Eliot Rose; Martin, Kevin Cc: Patton, Jeramy; Warner, Chris Subject: RE: Sidewalk Labs - Council   Excellent news!  Thanks for your expert shepherding skills, Mike!   ............... Eric Hesse Supervising Planner Policy Innovation + Regional Collaboration Pronouns: He/Him Portland Bureau of Transportation 1001 SW 5th Avenue, Suite 500 Portland, OR 97204 Phone: 503.823.4590 eric.hesse@portlandoregon.gov www.portlandoregon.gov/transportation twitter   facebook   instagram   publicalerts       The City of Portland ensures meaningful access to city programs, services, and activities to comply with Civil Rights Title VI and ADA Title II laws and reasonably provides: translation, interpretation, modifications, accommodations, alternative formats, auxiliary aids and services. To request these services, contact 503-823-5185, City TTY 503-823-6868, Relay Service: 711.   From: Kerr, Michael Sent: Wednesday, December 12, 2018 10:33 AM To: Pearce, Art ; Siegel, Noah ; Hesse, Eric ; Hurley, Peter ; Eliot Rose ; Martin, Kevin Cc: Patton, Jeramy ; Warner, Chris Subject: Sidewalk Labs - Council   All,   FYI – if you didn’t see it, Council (all members) just passed our Sidewalk Labs Ordinance.       Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: To: Subject: Date: Eliot Rose Kerr, Michael; Martin, Kevin RE: Replica update Friday, November 30, 2018 10:34:00 AM Sounds good.   We finally connected with TriMet over their proposed changes to the IGA, and we’ve received more details from Sidewalk about the process. There are a couple edits that I’d like to make to the IGA, but I don’t want to throw off your approvals process. I can make the changes below today and send a draft. Feel free to give me a call (503.797.1825) if you want to discuss. · TriMet requested clarification to the terms under which we can all agree to back out of the agreement if Replica doesn’t meet our acceptance criteria. I think that this is a good idea, and it seems to be in the spirit of what your lawyers requested review of the contract, and the acceptance criteria are a critical part of that. · SWL’s updated acceptance criteria include criteria for their estimates of TNC movement. I know that’s of interest to everyone involved, but PBOT would need to be responsible for validating those movements. I’d like to update the IGA to clarify that Portland will be responsible for validating bicycle, pedestrian, and TNC volumes and movements. But do you feel comfortable committing to that? The TNC data is under NDA, Parking / Regulatory keeps a tight lid on it (or so I’ve heard), and I presume they’d need to share it internally with someone who has data crunching expertise to conduct the validation. I’m hoping you’ve made some strides sharing that data internally given that y’all have a congestion study in the works that draws on it. You would only need aggregate data to conduct the analysis.      Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Thursday, November 29, 2018 12:17 PM To: Eliot Rose; Martin, Kevin Subject: RE: Replica update I have the budget (full $154K) NOW and can transfer funding once the IGA gets through Council.    As for Council, I can’t say when we’d actually get called given we’re on the Consent agenda.  May know more by the end of today as to whether or not there’s significant interest – which would indicate that there’s a good chance that I’ll get called.  So, for now, plan on coming for as long as you can (bring your laptop – do work while there).  I’ll circle back tomorrow.     Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Eliot Rose Sent: Tuesday, November 27, 2018 4:01 PM To: Kerr, Michael ; Martin, Kevin Subject: Replica update Hello Michael and Kevin:   Happy Thanksgiving! Hope y’all had a restful break.   I’ve been fielding a few questions from TriMet re: the Replica IGA, including one that relates to your payment schedule: when are you going to have funds available to put toward this project? TriMet has asked whether they can pick up the payments during the 2nd half of 2019 because that’s when they’ll have budget available. That means that we’d need PBOT to be able to make the whole April payment. See the proposed revisions to the payment schedule below. Does that work for you? If not, we can look into altering the schedule.   Payment date (approx.) January 2019 April 2019 Payment amount July 2019 $101.622.22 October 2019 $101.622.22 $152,433.33 $101.622.22 Responsibility (original) Metro to pay in full Portland and TriMet each to reimburse Metro for $50,811.11 Portland and TriMet each to reimburse Metro for $50,811.11 Portland and TriMet each to reimburse Metro for $50,811.11 Responsibility (revised) Metro to pay in full Portland to reimburse Metro in full Portland and TriMet each to reimburse Metro for $50,811.11 TriMet to reimburse Metro in full   TriMet also requested a few changes to clarify the validation and acceptance process. Shouldn’t be anything controversial; I think that it’ll be good to build some additional clarity into that. I’ll share with you once we make those revisions.   Finally, what time is the Replica item scheduled for the 12/5 Council meeting? I have the whole 9:3012 slot blocked off on my calendar, but I have a request to make a conflicting meeting, and I imagine you don’t need me there for the whole time.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, November 19, 2018 4:20 PM To: 'Kerr, Michael'; Martin, Kevin Subject: RE: Hold your calendars All I have drafted is the attached, which you’ve seen before. If there are specific points you’re trying to hit let me know and I’m happy to give you some thoughts about how I’d approach them.   Don’t forget to send me the time so I can be there!   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, November 19, 2018 9:55 AM To: Eliot Rose; Martin, Kevin Subject: RE: Hold your calendars Sounds good and thanks to both of you for your willingness to support (if/as needed).   Eliot, do you happen to have talking points that I build off of for my pre-submission to Council?    Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Eliot Rose Sent: Wednesday, November 14, 2018 9:06 AM To: Martin, Kevin ; Kerr, Michael Subject: RE: Hold your calendars I can probably be there, calendar is pretty open that day. What time, and do you need me to prep anything?   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Friday, November 09, 2018 7:16 PM To: Kerr, Michael; Eliot Rose Subject: Re: Hold your calendars I’ll be there. Love consent agenda! kevin martin smart city pdx/ tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com From: Kerr, Michael Sent: Friday, November 9, 2018 6:15 PM To: Martin, Kevin; eliot.rose@oregonmetro.gov Subject: Hold your calendars   We are officially on the 12/5 city council schedule for sidewalk labs (consent agenda). Flagging this for the two of you, in particular, due to the HIGH likelihood that I get called to speak. Will need all the technical support I can get and having the likes of Smart City PDX and Metro by my side would go a long way! Let me know if you happen to have time to sit around and support this smart cities push, Mike Sent from my iPhone From: To: Subject: Date: Eliot Rose Kerr, Michael; Martin, Kevin Replica update Tuesday, November 27, 2018 4:01:00 PM Hello Michael and Kevin:   Happy Thanksgiving! Hope y’all had a restful break.   I’ve been fielding a few questions from TriMet re: the Replica IGA, including one that relates to your payment schedule: when are you going to have funds available to put toward this project? TriMet has asked whether they can pick up the payments during the 2nd half of 2019 because that’s when they’ll have budget available. That means that we’d need PBOT to be able to make the whole April payment. See the proposed revisions to the payment schedule below. Does that work for you? If not, we can look into altering the schedule.   Payment date (approx.) January 2019 April 2019 Payment amount July 2019 $101.622.22 October 2019 $101.622.22 $152,433.33 $101.622.22 Responsibility (original) Metro to pay in full Portland and TriMet each to reimburse Metro for $50,811.11 Portland and TriMet each to reimburse Metro for $50,811.11 Portland and TriMet each to reimburse Metro for $50,811.11 Responsibility (revised) Metro to pay in full Portland to reimburse Metro in full Portland and TriMet each to reimburse Metro for $50,811.11 TriMet to reimburse Metro in full   TriMet also requested a few changes to clarify the validation and acceptance process. Shouldn’t be anything controversial; I think that it’ll be good to build some additional clarity into that. I’ll share with you once we make those revisions.   Finally, what time is the Replica item scheduled for the 12/5 Council meeting? I have the whole 9:3012 slot blocked off on my calendar, but I have a request to make a conflicting meeting, and I imagine you don’t need me there for the whole time.   Thanks!   Eliot Rose Technology Strategist Metro   From: Eliot Rose Sent: Monday, November 19, 2018 4:20 PM To: 'Kerr, Michael'; Martin, Kevin Subject: RE: Hold your calendars All I have drafted is the attached, which you’ve seen before. If there are specific points you’re trying to hit let me know and I’m happy to give you some thoughts about how I’d approach them.   Don’t forget to send me the time so I can be there!   Eliot Rose Technology Strategist Metro   From: Kerr, Michael [mailto:Michael.Kerr@portlandoregon.gov] Sent: Monday, November 19, 2018 9:55 AM To: Eliot Rose; Martin, Kevin Subject: RE: Hold your calendars Sounds good and thanks to both of you for your willingness to support (if/as needed).   Eliot, do you happen to have talking points that I build off of for my pre-submission to Council?    Michael Kerr Division Manager, Strategy, Innovation, and Performance PBOT Business Services 1120 SW 5th Avenue,  Suite 800 Portland, OR  97204 503-823-5808 Michael.Kerr@portlandoregon.gov website twitter facebook publicalerts     The City of Portland complies with all non‐discrimination, Civil Rights laws including Civil Rights Title VI and ADA Title II. To help ensure equal access to City programs, services and activities, the City of Portland will reasonably modify policies/procedures and provide auxiliary aids/services to persons with disabilities. Call 503-823-5185, TTY 503-823-6868 or Oregon Relay Service: 711 with such requests, or visit http://bit.ly/13EWaCg   From: Eliot Rose Sent: Wednesday, November 14, 2018 9:06 AM To: Martin, Kevin ; Kerr, Michael Subject: RE: Hold your calendars I can probably be there, calendar is pretty open that day. What time, and do you need me to prep anything?   Eliot Rose Technology Strategist Metro   From: Martin, Kevin [mailto:Kevin.Martin@portlandoregon.gov] Sent: Friday, November 09, 2018 7:16 PM To: Kerr, Michael; Eliot Rose Subject: Re: Hold your calendars I’ll be there. Love consent agenda! kevin martin smart city pdx/ tech services manager city of portland bureau of planning & sustainability kevin.martin@portlandoregon.gov 503.823.7710 www.smartcitypdx.com From: Kerr, Michael Sent: Friday, November 9, 2018 6:15 PM To: Martin, Kevin; eliot.rose@oregonmetro.gov Subject: Hold your calendars   We are officially on the 12/5 city council schedule for sidewalk labs (consent agenda). Flagging this for the two of you, in particular, due to the HIGH likelihood that I get called to speak. Will need all the technical support I can get and having the likes of Smart City PDX and Metro by my side would go a long way! Let me know if you happen to have time to sit around and support this smart cities push, Mike Sent from my iPhone