Create Google pub/sub topic and Google pub/sub subscription entities within OM

[jetpacula]

Is your feature request related to a problem? Please describe. In case of ETL streaming Kafka topic -> Google pub/sub topic -> Google pub/sub subscription -> BigQuery table Lineage connectivity is only available with omitting Google pub/sub topic and Google pub/sub subscription in which leads to loss of valuable information on data streams. This is caused due to absence of Google pub/sub topic and Google pub/sub subscription entities within OMD.

Describe the solution you'd like Existence of Google pub/sub topic and Google pub/sub subscription entities within OM for ability to create lineage edges manually.

Describe alternatives you've considered Linking Kafka topic directly to BQ table entity, omitting Google pub/sub topic and Google pub/sub subscription.

[TeddyCr]

@jetpacula would you like to contribute the connectors?

[adisin17]

Hi @TeddyCr , I am new here and would like to take this up. How can I contribute?

[TeddyCr]

You can check out our contribution doc, make sure to check the new connector section of the contribution page.

Finally you can join our slack in the #contribution channel for any questions you may have.

[malimsZen]

Hi @TeddyCr I'd like to contribute to this issue. Can I help?

[TeddyCr]

@adisin17 checking in to see how it is going on your end. Let's know if you still want to contribute this otherwise we'll assign it @malimsZen

@malimsZen feel free to start working on this if we don't hear back from @adisin17 in the next couple of days. 😊

[malimsZen]

@TeddyCr I'm assuming this issue!

[TeddyCr]

@malimsZen assigned it to you 😊You can check the resources mentioned here for any help