atlas icon indicating copy to clipboard operation
atlas copied to clipboard

ATLAS-4323: Add keys to Kafka notifications

Open bellemare opened this issue 3 years ago • 2 comments

In draft to seek feedback:

Important points:

  1. Key is fixed to String, which is the format of GUID.
  2. Keys will be GUID based, as the GUID field does not change. This will ensure data locality within the Kafka topic's partitions.
  3. Currently, no support for arbitrary keys. It is not clear to me that there are any use-cases that would require us to provide the user with the ability to key against arbitrary fields / data structures.

bellemare avatar Jun 08 '21 15:06 bellemare

  • adding a field for 'key' in HookNotification and EntityNotification might be a simpler/smaller change. I suggest to consider this approach.
  • how are the keys generated, and where are they assigned to each notification being sent? I could see this in unit tests. Can you please point to appropriate code lines?

mneethiraj avatar Jun 12 '21 18:06 mneethiraj

* adding a field for 'key' in HookNotification and EntityNotification might be a simpler/smaller change. I suggest to consider this  approach.

I'm trying this now, but it doesn't seem like it works conceptually.

For instance, it seems that HookNotification is extended by EntityCreateRequestV2. This extension, however, seems to pack in multiple entities via AtlasEntitiesWithExtInfo, meaning that there is no single key that HookNotification can use.

I think I need some help conceptually here: Is HookNotification and EntityNotification supposed to be a bundle of an arbitrary number of entities? I was originally under the impression that there were meant to be 1:1 (eg: one EntityNotification per entity).

bellemare avatar Jun 14 '21 14:06 bellemare