Pulsar + Delta Integration

[sijie]

trafficstars

Motivation

Pulsar is a cloud-native messaging and streaming platform. It has been well adopted by many companies in different regions. There is a need requested from the Pulsar community to have a native integration between Pulsar and Delta lake.

There can be many different integrations between Pulsar and Delta Lake. Here are some ideas:

1. A Pulsar source connector to capture the data changes from a Delta Lake table.
2. A Pulsar sink connector to write the Pulsar streams to Delta Lake.

This is a master issue for tracking the progress on integrating Pulsar and Delta Lake.

[frankxieke]

I will work on this issue this week. Please assign this issue to me and hang chen.

[hangc0276]

I will work on this issue this week. Please assign this issue to me and hang chen.

Yes, we are working on this issue. Please assign this issue to me and @frankxieke, thanks.

[frankxieke]

https://github.com/streamnative/pulsar-io-delta-lake/pull/3

[vkorukanti]

This repo has been deprecated and the code is moved under connectors module in https://github.com/delta-io/delta repository. Please create the issue in repository https://github.com/delta-io/delta. See delta-io/connectors#556 for details.