atlas
atlas copied to clipboard
ATLAS-4323: Add keys to Kafka notifications
In draft to seek feedback:
Important points:
- Key is fixed to String, which is the format of GUID.
- Keys will be GUID based, as the GUID field does not change. This will ensure data locality within the Kafka topic's partitions.
- Currently, no support for arbitrary keys. It is not clear to me that there are any use-cases that would require us to provide the user with the ability to key against arbitrary fields / data structures.
- adding a field for 'key' in HookNotification and EntityNotification might be a simpler/smaller change. I suggest to consider this approach.
- how are the keys generated, and where are they assigned to each notification being sent? I could see this in unit tests. Can you please point to appropriate code lines?
* adding a field for 'key' in HookNotification and EntityNotification might be a simpler/smaller change. I suggest to consider this approach.
I'm trying this now, but it doesn't seem like it works conceptually.
For instance, it seems that HookNotification
is extended by EntityCreateRequestV2
. This extension, however, seems to pack in multiple entities via AtlasEntitiesWithExtInfo
, meaning that there is no single key that HookNotification
can use.
I think I need some help conceptually here: Is HookNotification
and EntityNotification
supposed to be a bundle of an arbitrary number of entities? I was originally under the impression that there were meant to be 1:1 (eg: one EntityNotification per entity).