oteps

oteps copied to clipboard

Ongoing experiments, research, and draft specification for a multi-variate columnar encoding of OTLP.

At the request of @jmacd

The committers listed above are authorized under a signed CLA.

• :white_check_mark: login: lquerel / name: Laurent Quérel (95da4aea60cf6644c5f4827df7f7fead0cace449, aec7656ca4c0f21f03d905d5a1183e68d3a76c7d, a3fbdb730fac42e834ac8c6e1303c8efb76f8a17, 6b550f4e09f211463d2dd162159d7db50dca1f01, 3496ca11caa5b86a15d7baf76dc77661ac7a5ba8, 56b3357c61e60997d2bf5c193889fd75eadd3667, d760154fe981d566299fff838c1f6e70a3a2a356, 589dc8b3b012bfaf9a04032bc60809fc256955bb, a51696224f325ab63cafe2bb6b7ac03dad8072d5, ecf45d2062ab73bd0dd0fde4ab2368b4af8d97b4, 58e9a5363e14c884a05454596c43411e47a65e00, c809b1be52a9bd72408970d2f0d34b1019f8d81d, 150d987df98acc54feb6318428b17addb4380553, 43340eaedcee3ae0ba0e495a94f8b6dd24019908, 1448c911301088bbd10e5a4dadf7a5e029c41aac, 0db08ec3c1bd945e0e219b34967ede302108e2bc, f3739cf5ab591ef7bc9e6ff21694428e4d438461, 2218179b994985abc3e9032de38051502a4dc364, ec813fec7739d163a31a28de9b146ba08f79aa05, 8c34a9153b520dabdf93904e2a05bb22efc80412, e3c38db3d607cac160c05833261b0a51383bb492, 68b4ed6635a68c69140a17e9de1f98bdf0e4fdc6, e25b3c60586f3e6b3f635ebc60d11ea2daea9098, 07714b11d5d0fd735831634a0db999c5a9128e4b, e82db1a8afe79482d8a9c0a1e3c17e86fe09ab96, d9a621a8baafe6fa7f4c9a587017dbfcc9701810, 8c7333e31a6fcfdc396049b80107077cebf20b63, 91f9561a54db95e1831eaadacebdc489b6f6dca3, c58a4ade6fb66ab6208b1dfcfbc3686afe0ded42, 52a85f77762ef61a0bcb747b5a6c2f572990b63c, 0b3f02d6514a6c0a0ccf5e6ef6159e93e37e32df, e8c6ef11862738cdc2c12f0af81210965d4f0487, dc74dd39a07bf905dbd26e99820206b689730cf1, fffce3353f238666772d2c74521f94c919c1fc06, 72768c4f01ad5db9f365d0ef1f6fc37d218af42c, 869559197d5569d633a5f54ea5bcbf7098f0c7f5, 9139969285083d70b8424d98a13ace3551b3164e, 6499c63b36646eb2f093294c34060c7bccb3578a, aa9c513ba708544a3652348a562c8131d79aa843, f0c6b90daf64b4fd323b7de77ab3140e63c1ccd7, fce05afe6a9df4351fa1f15ea8499b91d86010f7, 33ea57d6b441b719dac2d85d3bccfffb04553cc1, 3d636d57065a7ed1f4a3d48d80089b6f989d77cb, 19653e1ce6b4f30c609a3e01de3cb46ced66016a, 0590cb852727c3a2a9f8e76aba57dd29d061b2da, 8449b67828571dc0525a4fab9a2f1f66c277a6df, 0be8ae33d164df7ca9207c612bf9d636490dc9cd, ab93d23e6ce3b93a1ce68fe0d725cf9c6e05f6a9, f48c9522acf743f0e580446f625478600a460fae, 0e6b9ff6b3f727cfba4da9fd77b2bb1df44a4710, 577b4bd4575eef5a32d63ebcaf2bd9cd416c3104, 5b62001c42c926850f735f355f4ca4b747538afb, ec0a62391cd962cf8248224e31e242e791fb99ff, c0d134d25746baca34ad06af1bddb2a2ad02cc2c, 4f311f87a6e73afc8b331f08504558720059539d, 4baab68856107f6bb3431305a5a24924f5c8e815, 6b3d916a9dd171c462b3bb24147b8382bab55f6e, 3cc34584b57ff602df4b5b3c7942aaa461e515cd, f24cb9cabd58806f6f06bcb60fc284c60ee536cb, efa0aad8b7bcc803a78d69e0d887c2374c068bcb, b8cbf5564e90dac31df9f6e05bfec7affcda83a7, b3b00c0a832ec63ee5ba9a65e3fecb8193ffe5aa, b4a95df94a496053c007ab33fcbfb8172f2c3de2, 8271a6567428157e2369ab564ed66721f95099a8, 50557bc75df8f7ea830f813327e076267e43f302, e0dae9943648070c4ea45dd9aa5457e67c97f3b3, 321bbc582ab0b1968e5b117f4cf2afacc5fd0084, 50d4cd392c49c238c7995819785a224816fb2f5d, 5f1e2d8871b1993a5f8a4e2ef075ce8fbfd7f10e, 9e38c4736bee7afe144d2d973f6dad9582c20753, 44b2743a54dde56059f42817fbbe0b45bc68523d, 198f114f2121d14ccd7e0e2196ed69306343a673, 778cf0aa05e63bf7b857baee54ffb0d42034a1c2, e520b036d004a2c0c59e2011425424601d1672ea, 3952c527205a8182a9fb60fd4e3a8293554acdad, 21cd6651b6ea4b191e94f626f9e50001ade71068, ef2c09f0d02f724795515bbce1cd8ed980cebbaf, ed78a511cd09c32f0c9b76fdf8f349ba87e32bde, 1524d84a072f16bc3d342ee60f1d6f926368d655, f0dc75325684ed5b85e211bddee0228b604483b4, 8e65c10c78dc44fea72b47b272282979943f9d3d, 3f73d39af66b63d44084e437b713873983afb1fa, c78c78efd9aab18c83244f99e5815d506e9f932c, 20b4e308fbc8aca62b09bcf85849f44d117e5943, c5c747caf136c2295bab35d6435a39a1be3ca393, 87b535264d83f5d25edd5087faffcf4e050e4433, d648aba156e2521961866b14f55c1b6e2b39b9ea, 942c98e2aa06cfba2ed27a532604787c77168333, d5e6a89d37a3dd2fc992fd904c7b35df89d30099, 4a10df3227402c9c27aa0c61a091ca86ca0c0085, f0dd6389897145418d6425989f001a9d5767d043, 1f0e9fc49081d101927e80d22c4eca9af56f7e54, 4454d02138ee6e7218cb2c49a40a3a2cbfca4fb9, efeb9e943a62fe708507005a19f7bf7c7f0726c8, eade6b5234f3d09208dcdccabad3f0bff40a2a6d, e6f7dc6e9e1586db6bed03a9db1b9ae84e5adcd7, 1fde1082a588ca9641fc2681179bd3697edd5d68, 8a2b7135b9289051908055cc1e9fbfbbf154785a, 686467c8e1df99366cf2293cc580b77bce7d9563, e08d43d6eff87e735adaa07c9d2fab82cb9b9a00, 110990ba8b65746fd61eb387262828aa562732b0, b9e84522974adc628bc71d0b9df55f7a7f0a1c53, 6786b8c6e3df8d2b3999903c466b49fc9292e806, df286554fc53f04f7c851a097388e343d9ceb06a, e73155df136f2584882f3b8e0af398d608c2b22e, 84dac539756b2f96810681eb4bbc198af8760e26, 1b78ba689e66b3a924b2f400dd9d74bc2bbbc4bf, 8f82586061504365568729104630c397c098be46, 72fbebacc266935cf268ba63ec61fb7c54ec606b, fb17f4bd74457ae674ec415ed448fe78fcd491a2, 2aa20479090575ebbbb6024a27f0d802d49d1b21, 21bd7cb8e6c78afccb2c4def41a3c9b15e2184f2, 621475f52acae81112b890659a241d075ab4a362, 4bed578d662b1e5827c011e88618de7653cb080b, 09749cc07d04c9a92cc2f52f6882d31fbb4159b6, 549061c48352ab463ba0325d20d7f05837db87ac, 64f4a1d908b97808a00adeeb71f52a796e908866, ea6fd96b7d2e8c8f0e62d44a46dbbe1e8c703426, ec7736858725e2dc8108d5fb89f2aa2c9b00d1dd, 6c2e9b5d6fd54e376327337215f075a27fe5d2c3, ce274de035b6d4f6d7af2947f7352ac141c9729f, df06cea421b00b2f5baad67131660f0ba4eb8082, f83a3788f69ccd53222f0f7a69055acbef3d120b, 9bdeab28efe2fa3cf347470651139ebad858352a, d5e8966542989d72795f7dad74cd029d5466755f, fae225a87a0f532ba18b6d546fbd8133e160b881, 1892d310148e986374a3a125bfa369b015ad846b, 55122642991e190629568dccb89ba85d1b8122f2, 560ca64666dd94092949ca362f9b8870fdb687bb, 9f68a84e0f99ad680b657658c9e0600eb61d6f84, 34fd4df6c50a04c5278bea4adc099dadb3d5b89f, 25bf657737d6dc070725687ad7175c84f58fd083, 7fbd2693524f86edc59946141d52ef77028fc717, 0c3742d8796ec94170d2834e6084a305dfdbd0c4, 0ffddbf601dc12aa0fc2c9d25aae2052b17d5014, e5c64e968245af32fd60b97953c5374c32be915d, 0f1faf1016d04df57b0f438997354a9b474e9ce9, f26c725236e9e1896f397100035f446093d31f8a, ad0669385b7eb3831e6a713b07fdc3c0eaf71a06, a924a99177bcd549180bb3c00ce7b74db6e736e1)
• :white_check_mark: login: carlosalberto / name: Carlos Alberto Cortez (81eb951aed85751b9e04773b812cb68277aa5162)
• :white_check_mark: login: atoulme / name: Antoine Toulme (55122642991e190629568dccb89ba85d1b8122f2, 560ca64666dd94092949ca362f9b8870fdb687bb, 34fd4df6c50a04c5278bea4adc099dadb3d5b89f, 25bf657737d6dc070725687ad7175c84f58fd083)
• :white_check_mark: login: reyang / name: Reiley Yang (0c3742d8796ec94170d2834e6084a305dfdbd0c4, 0ffddbf601dc12aa0fc2c9d25aae2052b17d5014, e5c64e968245af32fd60b97953c5374c32be915d, 0f1faf1016d04df57b0f438997354a9b474e9ce9, f26c725236e9e1896f397100035f446093d31f8a, ad0669385b7eb3831e6a713b07fdc3c0eaf71a06, 5bd1d003e287b16a5e3e9dcad740ea81a46ee92d)

Closing/reopening to see if EasyCLA passes.

Closing/reopening to see if EasyCLA passes.

@tigrannajaryan I have been able to fix the CLA issue by adding my professional email into my Github account.

@tigrannajaryan and @yurishkuro is there anything I can do to complete/clarify this OTEP?

@tigrannajaryan and @yurishkuro is there anything I can do to complete/clarify this OTEP?

@lquerel I may have missed it but I don't see answers to these questions:

Apache Arrow data is represented as an opaque byte array in the BatchEvent protobuf message. Is the intent here that it is essentially unprocessable in intermediaries, such as OpenTelemetry Collector and will be simply forwarded as is? If that is not the intent then how can this data be processed? Given that OpenTelemetry Collector is fundamentally built on the current row-oriented data model and all processing capabilities in the Collector work in terms of the row-oriented data model how can the telemetry received in BatchEvent be processed? Do we expect that columnar data will be converted to row-oriented data when it hits the Collector, is processed in row format then converted back to columnar data to exit the Collector? If this is the intent what’s the performance impact of this?

@tigrannajaryan and @yurishkuro is there anything I can do to complete/clarify this OTEP?

@lquerel I may have missed it but I don't see answers to these questions:

Apache Arrow data is represented as an opaque byte array in the BatchEvent protobuf message. Is the intent here that it is essentially unprocessable in intermediaries, such as OpenTelemetry Collector and will be simply forwarded as is? If that is not the intent then how can this data be processed? Given that OpenTelemetry Collector is fundamentally built on the current row-oriented data model and all processing capabilities in the Collector work in terms of the row-oriented data model how can the telemetry received in BatchEvent be processed? Do we expect that columnar data will be converted to row-oriented data when it hits the Collector, is processed in row format then converted back to columnar data to exit the Collector? If this is the intent what’s the performance impact of this?

@tigrannajaryan I had added yesterday in the section "OpenTelemetry entities to Arrow mapping" some elements to answer your questions and I submitted today some additional clarifications. Please let me know if this is what you expected or if I missed something.

I spent a bit more thinking about this proposal.

I think it can be very efficient representation for specialized cases. It is not clear if it can serve as a general-purpose telemetry protocol (the role that OTLP serves today), primarily because it is not clear how intermediary notes like Collector can process data in this format efficiently, while also allowing to define such transformations in a user-friendly way, like current Collector processors do.

I think the best way forward for this proposal would be to:

• Come up with a new protocol name, to make it is clear this is not OTLP, to avoid confusion. Don’t try to encapsulate this in existing OTLP messages, it won’t help existing receivers, they won't be able to interpret it anyway. This is fundamentally a new format. I also would advise against calling this simply OTLP v2, it will likely result in confusion and will prevent more evolutionary changes to current OTLP.
• Get some vendor support who are interested in seeing Otel SDKs emitting this data and contribute exporters for the new protocol to some Otel SDKs.
• Come up with a plan on how the protocol will traverse intermediary nodes (Otel Collector). I still don’t see this in the proposal. The classic way to support a new format is to implement a new receiver and a new exporter for Otel Collector. This will have performance implications that likely are important to know before one can use the format. If one uses the new format instead of OTLP how much you gain in SDK and in the backend and how much you loose in Collector may be an important factor.
• Define when exactly it is beneficial to use this new format/protocol vs OTLP (we can hope that the answer is "always" but likely it will not, so guidelines will be needed).

I second defining this as a new OTLP format (e.g. otlp-arrow), and have the usual prototypes in a few languages.

Is my understanding correct that this format's intent isn't RPC but to save for analytics processing? So we'd expect collector to still accept OTLP but save this format to S3?

@tigrannajaryan, @jmacd thanks for your feedback. I will work on them as soon as possible. I have some other urgent tasks to complete first at F5.

Is my understanding correct that this format's intent isn't RPC but to save for analytics processing? So we'd expect collector to still accept OTLP but save this format to S3?

@anuraaga, no the intent of this format is to optimize data transfer and in-memory data processing based on a columnar representation. Apache Arrow buffers can be easily converted into Parquet format for storage.

Is my understanding correct that this format's intent isn't RPC but to save for analytics processing? So we'd expect collector to still accept OTLP but save this format to S3?

@anuraaga - While this format can help for analytics processing, it serves a much greater purpose:

1. Enables multi-variate time-series data. This is critical for systems where metric values are related and wouldn't make sense as individual univariate values (e.g., firewall metrics, multi-axis devices, browser events, etc.). The protocol today only supports univariate (w/ multi-dimensions: attributes, resources) metrics.

2. OTLP as it stands is not performant/cost efficient enough for capturing all data from high-throughput systems (e.g., edge proxies, load-balancers, etc.). This proposal will not only enable capturing data from high-throughput systems, but will significantly reduce the operating cost for processing and receiving telemetry (egress, ingress, compute, etc.).

OTLP as it stands is not performant/cost efficient enough for capturing all data

Lightstep will corroborate this statement. Customers are looking to Sampling today as an approach to lowering data collection costs for OpenTelemetry trace data. The proposal in this OTEP suggests that users could pay more than an order of magnitude less to collect the same amount of Span data. Where we see this being used to the customer's advantage: OTel collectors would receive ordinary OTLP and then--in an exporter--perform Columnar compression for sending outside their network, i.e., to the vendor across an expensive network link. OTel collector would not need a new pipeline, only a new exporter.

It would be unusual for an exporter to exist and no corresponding receiver or pipeline. Receiving column-compressed metric event data is not very different than receiving Statsd events, they're just more compressed. The receiver will have to perform multivariate-to-univariate metric aggregation in order to inject ordinary OTLP into an OTel collector metrics pipeline. As a user this is not what I'm looking for, but it's something we could build if there is interest.

It would be unusual for an exporter to exist and no corresponding receiver or pipeline. Receiving column-compressed metric event data is not very different than receiving Statsd events, they're just more compressed. The receiver will have to perform multivariate-to-univariate metric aggregation in order to inject ordinary OTLP into an OTel collector metrics pipeline. As a user this is not what I'm looking for, but it's something we could build if there is interest.

+1 - I believe both a new receiver (multivariate to univariate) and a new straight event pipeline will be needed. There are users ( I am not at liberty to mention their names ) that are looking forward to this enhancement so they can collect and process multivariate time-series. These users currently have multiple environments setup where they use collectors in varying configurations to buffer, filter, and export telemetry.

Where we see this being used to the customer's advantage: OTel collectors would receive ordinary OTLP and then--in an exporter--perform Columnar compression for sending outside their network, i.e., to the vendor across an expensive network link. OTel collector would not need a new pipeline, only a new exporter.

Trying to re-take the conversation - I agree 100% with the previous paragraph. Once we could verify and tune that it works as expected, we could extend this to the rest fo the SDKs later on.

Where we see this being used to the customer's advantage: OTel collectors would receive ordinary OTLP and then--in an exporter--perform Columnar compression for sending outside their network, i.e., to the vendor across an expensive network link. OTel collector would not need a new pipeline, only a new exporter.

Trying to re-take the conversation - I agree 100% with the previous paragraph. Once we could verify and tune that it works as expected, we could extend this to the rest fo the SDKs later on.

While an exporter is a good start, a pipeline that supports end-to-end multivariate support will also be needed for the Collector. There are fortune 50 companies that use the Collector today and they want to be able to process multivariate time-series using their existing tooling.

One thing I haven't seen in this proposal (and I'm curious what the plan for this is going forward) is when you aggregate data in columnar format given the timestamps are shared.

1. Do you ever envision a columnar representations joining Traces Metrics + Logs in one "event"/"batch"?
2. Metrics in OTLP today are arbitrarily aggregated and exported at an interval (vs. actually being multi-variate or sampled at an important event). If we support columnar metrics, should these be synchronous co-exported in place or is the current practice of aggregating (e.g. latency) make sense? Would it make more sense to, e.g. export a Span + related metric points in one columnar export?
3. Logs vs. Events are already called out. Didn't catch the answer to that one.

@tigrannajaryan @jmacd @jsuereth @carlosalberto @gramidt @yurishkuro Sorry guys for the delay. Between my PTOs and some urgent tasks on my day to day job I have not been able to make any progress in the last few weeks. Fortunately, this will change next week.

@tigrannajaryan @jmacd @carlosalberto @gramidt @jsuereth @anuraaga @yurishkuro I've tried to summarize in a one diagram the direction I'm taking based on the various feedback. Let me know what you think.

Unless there is a strong disagreement, I'd like to continue to integrate logs and more importantly traces into the benchmark to clarify whether this approach could be global or more focused on the multivariate time-series scenario.

I also think we could phase this project by implementing the multivariate time-series first followed by traces and logs (if the benchmarks confirm my assumption).

Thank you for your patience.

@jsuereth

One thing I haven't seen in this proposal (and I'm curious what the plan for this is going forward) is when you aggregate data in columnar format given the timestamps are shared.

1. Do you ever envision a columnar representations joining Traces Metrics + Logs in one "event"/"batch"?

No I don't think so. In order to be efficient, batches must be homogeneous. So by design, a batch will be a collection of events sharing the same schema. However, when that makes sense we could represent complex events such as a span event containing related metrics in a single schema (as you suggested in the next section) and consequently build homogeneous batches of such complex events.

2. Metrics in OTLP today are arbitrarily aggregated and exported at an interval (vs. actually being multi-variate or sampled at an important event). If we support columnar metrics, should these be synchronous co-exported in place or is the current practice of aggregating (e.g. latency) make sense? Would it make more sense to, e.g. export a Span + related metric points in one columnar export?

I'm not sure if I understood the questions correctly, so please rephrase if the following answers are not what you expected. Regarding multivariate time-series aggregation, I don't think there is a fundamental difference with the univariate scenario. We should still be able to aggregate multivariate events sharing the same attributes/labels while respecting the rules of aggregation of metrics according to their nature (gauge, counter, ...). Regarding span + related metric, this seems to me a valid scenario of multivariate time-series. We have a set of metrics related to a set of complex attributes representing a span. This representation should avoid data duplication and should also improve processing speed/complexity by eliminating the need to recombine the spans with their corresponding univariate metrics. We might need to separate these complex events from the standard metrics/logs/spans for compatibility reasons. In oltp-arrow metrics, logs and spans can be converted into their counterpart in oltp with sometimes 1:N conversions (e.g. multivariate to n univariate). Things will probably be more challenging for complex events like the one you described. Separating them in a "not oltp compatible" category will make that clear.

@lquerel Sorry for delayed response. Thanks for posting the diagram. It makes the intent clear.

I believe what we see here is a need to introduce a new signal type for metrics (metrics-arrow or metrics-columnar, whatever we call it) in the Collector and possibly also for logs and traces similarly. This then necessitates new processors that works with the new signal types. This is potentially a very large amount of work, especially if we are want to be close to feature parity with what processors exist today.

We likely also need to support converting regular data to columnar and back so that existing receivers and exporters for non-OTLP are not left out of the ecosystem. Likely also support for signal type converting in pipelines to make it easier to use together with non-columnar types.

Then we are also looking at implementing columnar exporters in Otel SDKs, likely also other changes necessitated to support columnar data type.

I have a feeling that while technically it is feasible it is unlikely that we as a community are currently able to lift such a major undertaking in the near future.

I do see the benefits and understand how it can lead to significant operational cost savings but at the moment I am very doubtful that this can happen soon simply because I don't see the engineering resources available that can turn this idea into reality.

It is certainly possible to start small, it is not an all-or-nothing endeavour, but again some critical amount of support is needed to get value out of it, which is still non-trivial amount of work and even I don't know if we will be able to lift off.

Sorry if this sounds negative. I like the proposal on the technical merit, but given the limited resources we have at Otel I personally don't see a good way forward right now. I hope I am wrong and there is a way to find engineers to work on this.

@tigrannajaryan Thank you for your detailed comment. I am aware of the amount of work involved in implementing this OTEP. I am in the process of checking with my company to see if I can invest some of my time in it. I should have an answer by the end of next week. I will most likely start with the multivariate time series implementation as this is the most important part for F5.

I am also creating a trace-based benchmarking tool to allow other companies to test this approach on their own data (OLTP --> OLTP-ARROW, and Columnar SDK --> OLTP-ARROW [this one is not done yet]). @jmacd's team will probably test it soon. See this git repo --> https://github.com/lquerel/oltp-arrow

It is certainly possible to start small, it is not an all-or-nothing endeavour, but again some critical amount of support is needed to get value out of it, which is still non-trivial amount of work and even I don't know if we will be able to lift off.

I agree with the feeling, and hence I would like to state again that working on this feature for Collector export would be a good, small, yet foundational step (the Collector pipeline would stay the same, but additional OTLP-arrow exporters would be created in parallel).

In Lightstep we are interested in this and we may be able to provide some cycles as well.

@carlosalberto IMO, implementing an OLTP-ARROW exporter will be relatively easy to implement for traces and logs. But I'm still not sure that we can generalize the transformation of a set of related univariate time-series to a multivariate time-series without the collaboration of the time-series producer.

Just a few words to confirm that I will be able to spend several weeks (this quarter) implementing an OLTP-to-OLTP-Arrow receiver, a basic columnar oriented processor, and an OLTP-Arrow exporter.

@bogdandrutu @tigrannajaryan @jmacd @jsuereth - We're working on getting further support from a small team within F5 to work on this; however, support from other member organizations would help expedite the progress. Based on the data, the ROI for anyone who receives OTel telemetry would be significant.

I would like to help establish a working group to get this completed. Are any of you able to help recruit resources given the ROI for your organizations?

Are any of you able to help recruit resources given the ROI for your organizations?

Unfortunately I don't think I can at the moment.

@tigrannajaryan @jmacd @carlosalberto @gramidt @jsuereth @anuraaga @yurishkuro Hi all, I'm organizing a meeting next Thursday for a demo of an end-to-end implementation of the OTLP Arrow protocol. Please select your best slots in the following doodle poll and feel free to send this link to anyone interested. Thanks

https://doodle.com/meeting/participate/id/PdRPn3Ea

@tigrannajaryan @jmacd @carlosalberto @gramidt @jsuereth @anuraaga @yurishkuro Hi all, I'm organizing a meeting next Thursday for a demo of an end-to-end implementation of the OTLP Arrow protocol. Please select your best slots in the following doodle poll and feel free to send this link to anyone interested. Thanks

https://doodle.com/meeting/participate/id/PdRPn3Ea

Done. It would be useful to add some more slots another day to make it easier to find time that works for all.

@tigrannajaryan added Friday morning