AWS Support of CloudEvents

[timbray]

Hi. We're working on a bunch of event-related stuff at AWS with a view to releases in 2019. It has been internally proposed that we "Support CloudEvents" and this seems to me like a good idea. No promises, but my opinion is that if we could find a plausible way to do it, there is a high likelihood that we would.

Important This is not an official communication, this is Tim looking for options I can use internally.

Background

We already have an AWS Event envelope that is widely used across AWS chiefly through the "CloudWatch Events" service. There's no formal schema or anything but it's documented here.

The required top-level fields are:

• version: (Corresponds to CloudEvents specversion)
• detail-type: (type)
• source: (source)
• id: (id)
• time: (time)
• detail: (data)

Fields in AWS Events not in CloudEvents: region, account, resources.

Fields in CloudEvents not in AWS Events: datacontenttype (So far we're all-JSON all the time.)

Problems

1. There are already huge numbers of AWS Events per second flowing through our system and there's no API, the only documentation is of the bits on the wire. And, there are a huge number of AWS customers who are making use of this service. Therefore there is no reasonable prospect of us changing field names.
2. CloudEvents isn't finished, but we want to ship production software in 2019. In theory, the working group could do a wholesale redesign of the attributes at any time. Can any organization plausibly claim to "Support CloudEvents" at this stage in history? It's not obvious to me how.

Appeal for input

Before I dive any deeper, I'd welcome general comments along the lines of "No, just wait till CloudEvents is finished", or "That is a really bad idea because X" for some value of X, or "You can effectively support CloudEvents by doing A, B, and C." Thanks in advance.

Some notes about possibilities

1. CloudEvents could consider freezing the field names. I.e. adopt some sort of policy that whereas new attributes may be added, there's a commitment not to change the names of the ones that exist. 1.1. And if you wanted to try making AWS an offer it couldn't refuse, consider changing the field names to match ours. We could add support for the extra CloudEvents fields and our extras could be extensions. There are some semantic incompatibilities (e.g. our "source" field looks like a relative URI reference) but probably there are workarounds. (OK, long shot).
2. Make sure all the CloudEvents SDKs include getters (I don't think setters?) for all the specified attributes. That way, if someone like us has CloudEvents-like constructs but with incompatible names, we could implement the SDK and then claim we're compatible because you can always use the CloudEvents SDK to access the attributes. Is this sane?
3. At the very least we could ship libraries that transcode between AWS Events and CloudEvents. Would that be enough to make users happy?

[cneijenhuis]

Sorry, I already wanted to draft a reply last week. I hope it helps.

CloudEvents isn't finished, but we want to ship production software in 2019. In theory, the working group could do a wholesale redesign of the attributes at any time. Can any organization plausibly claim to "Support CloudEvents" at this stage in history? It's not obvious to me how.

Well, an organization can claim to support CloudEvents at a specific version. E.g. Azure supports 0.1 currently.

CloudEvents could consider freezing the field names. I.e. adopt some sort of policy that whereas new attributes may be added, there's a commitment not to change the names of the ones that exist.

The policy (semver) is hidden in here: https://github.com/cloudevents/spec/blob/master/GOVERNANCE.md#release-process-and-versioning (Took me a while to find the place as well :wink:) 1.0 will do what you ask for.

Fields in AWS Events not in CloudEvents: region, account, resources.

I'd argue that in CloudEvents lingo, they should be part of the source. E.g. the (ce-)source is /region/account/resource/(aws-)source

At the very least we could ship libraries that transcode between AWS Events and CloudEvents. Would that be enough to make users happy?

I'd think that would be a good first step. Even better if the transcoding can be enabled at service level (i.e. I can ask S3 to release CloudEvents instead of AWS Events. BTW: This would also be a good place to do the versioning. I could ask S3 to do CE 1.0, 0.3 or (at some point...) 2.0.

---

If I'm getting you correctly, when you talk about AWS support for CloudEvents, you talk about the events that AWS produces. This is certainly important. However, at least as important (to me) is that the eventing/messaging products that AWS has natively support CloudEvents. E.g. for SNS it'd be great if I could filter on CloudEvents attributes (at least the standard ones such as source and types) It would also be great if SNS supports the HTTP transport binding for publishing and receiving events.

Anyway, AWS producing CloudEvents would already be a great first step!

[jordo1138]

Eventbridge yeah? Looks like a platform that ought to support the spec, at least, users would be looking for it at some point

[cneijenhuis]

@jordo1138 Came here to say the same thing 😉

@timbray Do you have plans to support CloudEvents at EventBridge soon-ish? If not/if later, is it ok if I go ahead and create a proprietary transport?

[timbray]

As for a proprietary transport, the AWS CloudWatch Events format is already documented on that page, and EventBridge is continuing to use it, so I think we're covered. As I said at the top of this thread, I'd like to figure out a good way for AWS to be in the community, and I'm happy the group is converging on a stable 1.0, which will make it easier.

[timbray]

Hey @cneijenhuis , thanks for the note. I believe pretty strongly that once 1.0 gets announced, if it gets any adoption, it'll become difficult to extend and change and further version. I speak from experience as one who is running $BIG_NUMBER events/second through a bus and has $BIG_NUMBER of customers processing them. Nobody wants to do any versioning on this stuff once it's running. Note to self: Ping @clemensv to figure out if Azure's going to migrate from 0.1 to 1.0. If they're not, it might make more sense for us to stick with 0.1.

Interesting points about the value of "source"; at first glance I think I might agree.

[cneijenhuis]

@timbray I tend to agree on the versioning.

With Clemens on vacation - I think he mentioned on a call that Azure is planning to support 1.0. I don't know if they'll migrate, or if they'll support both side-by-side for some time.

As for a proprietary transport, the AWS CloudWatch Events format is already documented on that page

Are you referring to https://github.com/cloudevents/spec/blob/master/adapters/aws-s3.md ?

I'll try to find some time to get some hands-on experience with EventBridge at the end of next week (independently of CloudEvents). My gut-feeling is that what we've defined as an adapter in https://github.com/cloudevents/spec/blob/master/adapters/aws-s3.md works for AWS-produced CloudWatch Event -> CloudEvent, but doesn't scale to Any CloudEvent -> CloudWatch Event -> CloudEvent (with the original and the "transported" CloudEvent being unchanged)... but maybe I'm simply looking at the wrong file?

[timbray]

I'm referring to https://github.com/cloudevents/spec/blob/master/primer.md#aws---cloudwatch-events

On Thu., Jul. 18, 2019, 10:02 a.m. Christoph Neijenhuis, < [email protected]> wrote:

@timbray https://github.com/timbray I tend to agree on the versioning.

With Clemens on vacation - I think he mentioned on a call that Azure is planning to support 1.0. I don't know if they'll migrate, or if they'll support both side-by-side for some time.

As for a proprietary transport, the AWS CloudWatch Events format is already documented on that page

Are you referring to https://github.com/cloudevents/spec/blob/master/adapters/aws-s3.md ?

I'll try to find some time to get some hands-on experience with EventBridge at the end of next week (independently of CloudEvents). My gut-feeling is that what we've defined as an adapter in https://github.com/cloudevents/spec/blob/master/adapters/aws-s3.md works for AWS-produced CloudWatch Event -> CloudEvent, but doesn't scale to Any CloudEvent -> CloudWatch Event -> CloudEvent (with the original and the "transported" CloudEvent being unchanged)... but maybe I'm simply looking at the wrong file?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cloudevents/spec/issues/435?email_source=notifications&email_token=AAAEJE4UEMJCUNHTX6ZLDLLQACO3TA5CNFSM4HVFEG42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2JEMMY#issuecomment-512902707, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAEJEYMOJC6WOT22L3O2MLQACO3TANCNFSM4HVFEG4Q .

[cneijenhuis]

Thanks! That doesn't define a mapping from/to CloudEvents, though - which is what I'd be interested in, but I may just be too impatient 😉

[to-masz]

@timbray I would like to highlight what @cneijenhuis said about AWS support - it would be good to define how to use CloudEvents with AWS services like SNS/SQS.

1. whether you prefer to add support for HTTP transport binding (https://github.com/cloudevents/spec/blob/master/http-transport-binding.md#314-examples), e.g. as dedicated SNS endpoint to publish messages in that form and then to support it for HTTP endpoint delivery
2. or just write down SNS transport binding, e.g. similar to https://github.com/cloudevents/spec/blob/master/kafka-transport-binding.md#325-example

[duglin]

@timbray is there anything we need to do on this one? Should we close it?

[timbray]

Thanks for the input, everyone

[moltar]

@timbray are there any updates on this? Is there an approach that works with AWS?

[cneijenhuis]

@moltar Tim has left AWS: https://www.tbray.org/ongoing/When/202x/2020/04/29/Leaving-Amazon

[trahloff]

Hi, is there any plan to support AWS as an event source long-term?

[duglin]

@timbray I know you're no longer at AWS - but any comment? Or know of anyone at AWS we could reach out to?

[Vlaaaaaaad]

CC @nikp, @nickste, @embano1, and maybe @boyney123?

[timbray]

I'm no longer an AWS insider but it is clear to me that if AWS were going to do anything ambitious around CloudEvents, it would have to be led by the EventBridge product team. The very best way to get AWS's attention would be to get some AWS customers to say "We want CloudEvents support." In fact, my opinion is that that's probably the only way to get any traction.

[embano1]

Hi, is there any plan to support AWS as an event source long-term?

I'm a PM in the EventBridge team and also trying to remain in involved in CloudEvents (mostly on the SDK side). Perhaps we should open a separate issue since this is closed. Can you please elaborate (perhaps on the new issue) what you mean by "AWS as an event source long-term"?

[trahloff]

Got it, thanks for the insight. Will flag this with our AWS account team and report back. Thanks everybody for the quick response!

@embano1

Can you please elaborate (perhaps on the new issue) what you mean by "AWS as an event source long-term"?

Sure! We found a workaround by having a Lambda stream EventBridge events to a CloudEvents target (basically this setup). My phrasing of "long-term" was a bit misleading. I was talking about a native integration like, for example, Azure Event Grid has with CloudEvents. This would be a more sustainable solution long term.

What makes the most sense from your perspective, reopening this ticket to have the whole discussion in one thread? If not, I can happily create a new ticket and port the main talking points from this one.

[embano1]

Got it, thanks for the insight. Will flag this with our AWS account team and report back.

Feel free to loop me into this email thread so I can work with my peers. Looks like we're in the same time zone :) (You can also find me on Twitter if you want to DM)

Sure! We found a workaround by having a Lambda stream EventBridge events to a CloudEvents target (basically this setup). My phrasing of "long-term" was a bit misleading. I was talking about a native integration like, for example, Azure Event Grid has with CloudEvents. This would be a more sustainable solution long term.

Thx for sharing! Wish you wouldn't have to do all those workarounds.

What makes the most sense from your perspective, reopening this ticket to have the whole discussion in one thread? If not, I can happily create a new ticket and port the main talking points from this one.

@duglin shall we keep going here (needs to be reopened) or create a new issue?

[moltar]

Maybe there's a way to use newly launched Amazon EventBridge Pipes to create a transformer into CE?

[embano1]

Maybe there's a way to use newly launched Amazon EventBridge Pipes to create a transformer into CE?

Yes, I know customers using Step Functions as an enrichment step in a Pipe to convert to CE. However, it remains a wrapped CE within the EB envelope when delivered to the target.

[duglin]

I've reopened the issue, but I'm wondering what the action would be for the CE group. This feels more like an AWS issue than a CE issue at this point. Unless there's some collaboration we need to help kick-off??

[embano1]

I've reopened the issue, but I'm wondering what the action would be for the CE group. This feels more like an AWS issue than a CE issue at this point. Unless there's some collaboration we need to help kick-off??

Thx. ACK, I might get back to you at some point with some advice, but currently there's no AI on the CE side. Let's keep this open for some time and close if there's no further interactions from user(s).

[j-fulbright]

Now that CE is officially "graduated", maybe this can be looked at again

[embano1]

Now that CE is officially "graduated", maybe this can be looked at again

@j-fulbright a small step, but we're moving into the right direction: https://aws.amazon.com/about-aws/whats-new/2024/02/amazon-eventbridge-api-destinations-content-type-header-customization/

Happy to discuss the topic in the greater AWS (EventBridge) scheme. Feel free to connect.

[duglin]

AWS EventBridge now supports CE: https://aws.amazon.com/about-aws/whats-new/2024/02/amazon-eventbridge-api-destinations-content-type-header-customization/

I'm going to propose that we close this issue on tomorrow's call... if anyone objects please speak up here or on the call

[embano1]

FWIW, we also wrote a blog post to explain the current integration possibilities: https://aws.amazon.com/blogs/compute/sending-and-receiving-cloudevents-with-amazon-eventbridge/

[timbray]

Hey Michael,

Under "Publishing CloudEvents using Amazon EventBridge", is the value of the data.dataschema field really right? Why go through the console?

[embano1]

Hey Michael,

Hi Tim, great to see you around!

Under "Publishing CloudEvents using Amazon EventBridge", is the value of the data.dataschema field really right? Why go through the console?

Good catch! Let me file with the author and get this fixed!