Serialize across multiple (master-detail) tables

[robinroos]

Any chance this Connector could serialize a message containing [ ... ] across a hierarchy of tables?

Clearly UPSERT semantics would have to be replaced with DELETE (and Delete Cascade on the Foreign Keys).

I'm hoping to use such a capability to write to a data warehouse.

If that is not on the roadmap for kafka-connect-jdbc, is anyone aware of similar connector implementations which might have this capability?

[robinroos]

In the absence of a suitable Connector, my alternative is to implement a Sink processor which deserializes to POJO graphs, and writes those through JPA with EntityManager.get() and EntityManager.merge().

[gharris1727]

@robinroos I don't believe we have any plans to support processing of complex data types (struct, map, list) or inserting into a hierarchy of tables. If you need to do this from a single input topic, I'm not aware of a connector that has that functionality, and you'd have to implement one yourself.

You might be able to accomplish this by having a stream processing job take the one input topic and produce a new topic for each table you want to appear in the final database. Each of those topics could then use the stock JDBC Connector to write each topic to the corresponding table.

If you think that you have a good idea for how to implement this in the JDBC connector such that it would generalize to many different databases, then I would encourage you to implement the feature and open a PR. It certainly sounds like a useful feature, and an interesting challenge.

[ifr1m]

@robinroos I encountered the same problem. Have you found a solutions ?

[tomyChr]

@robinroos same probleme here. Is there a better solution in the meantime? Otherwise looks like that I have to clone the connector and add my logic on top.