CloudEvents Connectors Proposal

[JieDing]

Objectives

Not all event producers will produce CloudEvents natively. As a result, some "adapter" might be needed to convert these events into CloudEvents.

Currently, CE spec has 5 adapters to describe how to convert non-CEs from external services into CloudEvents. However, even with these adapter specs, CE adopters still need to implement their programs to finish the conversion process. In order to better boost the adoption of CE and avoid redundant conversion implementations, I propose creating a new repo to provide implementations of CE adapters specs. Each implementation will be published as an image, enabling CE adopters to skip the complex integration with external services.

Proposal

New repository

Create a new repo called connectors (or other names) to collect source codes of connectors.

Differences between an adapter and a connector

An adapter is used to describe how a non-CloudEvent can be converted into a CloudEvent. It can be regarded as an important part of a connector. A valid connector should at least consist of three parts:

1. obtains data from underlying data producers(this part may include building a connection with the data producer)
2. converts data into CloudEvents(this part is the implementation of the adapter spec)
3. delivers converted CEs to targets

What is a valid connector

A valid connector obtains data from underlying data producers, converts it into CloudEvents, and delivers CEs to targets. There is no limitation to the way a connector gains data. It can pull data from a message queue (pull-model) or wait for data to be sent to it like an HTTP server (push-model). Eventually, CloudEvents will be sent as POST requests in HTTP format (might support other protocols in the future).

How to use a connector

Each connector will be published as an image, which means a container runtime will be the only pre-requisite to use our connectors.

How to maintain the repo

There should be a GitHub team, who is responsible for maintaining the repo by doing the following stuff:

• Specify the process to contribute a new connector
• Review codes of new connectors
• Make plans to increase connectors
• Publish connector images
• ......

How to add a new connector

The maintainers team will regularly open issues to list planned connectors. Contributors can comment to ask maintainers to assign that connector to them. Or they can open an issue to suggest a new connector which is not on the planned list.

The process of building a new connector generally consists of:

1. Writing a connector proposal in the connector repo to illustrate how you would like to do this.
2. Submitting an adapter spec in the spec repo to describe how external events will be converted to CEs.
3. Implementing your connector based on connector templates for different programming languages.
4. Passing unit test checker.
5. Passing integration test checker.
6. Publishing the connector image.

[JieDing]

@duglin

[duglin]

Overall I like the idea of a project to provide canned adapters for folks to use. The proposal talks about how there might be different ways of getting the original events - I'm curious to see how that'll be designed. Do you see one image per type of adapter, and each image supports all of the various CE protocols? And then based on the configuration of the image it'll use the appropriate protocol to get the events? Could be a good way to force at least one of our SDKs to support all of the protocols and use that as the front-end for the adapters. Then we only need to support each protocol once and just plug-in each adapter behind it.

Would love to hear what other folks think.

[JieDing]

Overall I like the idea of a project to provide canned adapters for folks to use. The proposal talks about how there might be different ways of getting the original events - I'm curious to see how that'll be designed.

I edited the proposal to clarify the differences between an adapter and a connector. Basically an adapter is used to describe how a non-CloudEvent can be converted into a CloudEvent. It doesn't care about how to get the original events. But a valid connector does care. It obtains data from underlying data producers, converts it into CloudEvents(this part is described in the adapter spec), and delivers converted CEs to targets. So an adapter is partial logic of a connector.

How a connector gets the original events is totally up to the type of data sources it deals with. For example, a GitHub Webhook connector will run an HTTP server and wait for events to be sent to it, then converts the original events into CloudEvents and deliver them to the downstream target. But an AWS SQS connector might use aws-sdk to pull messages from SQS cluster, then converts the original events into CloudEvents and deliver them to the downstream target. So we don't limit how a connector will obtain original events and leave it to developers to implement based on events producers they are dealing with.

[JieDing]

Do you see one image per type of adapter, and each image supports all of the various CE protocols? And then based on the configuration of the image it'll use the appropriate protocol to get the events? Could be a good way to force at least one of our SDKs to support all of the protocols and use that as the front-end for the adapters. Then we only need to support each protocol once and just plug-in each adapter behind it.

Yeah each connector will be published as an image. Each image supports all of the various CE protocols and use a config to choose protocols will be a good idea for users to use connectors with flexibility. But that will also increase the complexity and difficulty for developers to implement a connector since they may not be familiar with all protocols in CE. We can require at least one of our sdks to fully implement all protocols in CE. But this may cause developers who is not familiar with that language-specific-sdk can never contribute a connector. So I suggest we can choose HTTP protocol as our default protocol to use in connectors since almost all sdks support CE HTTP protocol well. And we can gradually add other protocols to our connectors to provide protocol flexibility.

[duglin]

Just brainstorming... we could create an image that supports all protocols (doesn't matter which SDK we use), and then people can use that image as their BASE image to build a new connector image. They would just need to place an HTTP server (any language) that accepts a raw event for the http request and the http response would be a CE. They would just need to include our connector exe in their resulting image. Either via "FROM OUR_BASE" or "COPY --from OUR_BASE/connector /connector /" to copy the exe into their image. Just need to make sure that both their exe and our exe are running.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Mark as fresh by updating e.g., adding the comment /remove-lifecycle stale.

[duglin]

@JieDing is this still being looked at or should we close the issue?

[g41797]

Hi i implemented (in your terms) syslog connector framework With plugins it has possibility to publish syslog messages to any broker. now i have two working examples:

• syslog connector for Memphis is part of memphis-protocol-adapter project
• syslog connector for NATS - syslog2nats

what do you think regarding providing similar Cloud Events connector framework ? at least for go

"Just brainstorming..."

[duglin]

I think it would be interesting to see how @g41797's idea and @JieDing's idea might play together.

@JieDing any thoughts?

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Mark as fresh by updating e.g., adding the comment /remove-lifecycle stale.

[g41797]

/remove-lifecycle stale

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Mark as fresh by updating e.g., adding the comment /remove-lifecycle stale.

[duglin]

@JieDing any update on this one? Any plans or should we close it?

[JieDing]

@JieDing any update on this one? Any plans or should we close it?

No updates for now Doug, I'm afraid maybe we should close it.

[duglin]

Ok - we can always reopen it. thanks!