kanadi
kanadi copied to clipboard
zalando-nakadi
Nakadi clients resilience to partial outage and partial success
Nakadi publishing API accepts events in batches. It can fail to publish some events from the batch to underlying storage (Apache Kafka). In that case Nakadi publishing API will return error that batch was partially successful. It can create problems the following problems, depending on how the Nakadi client and the publishing application deals with this partial success response:
- increase in traffic on Nakadi publishing API due to Nakadi clients retrying the whole batch over and over
- the application retries identical batches which prevents application from progressing
The following should be done to decrease the possibility of mentioned problems:
-
Nakadi client should contain a note to developers that publishing can experience partial success. This should be in the client documentation and ideally also within the self contained code documentation, raising awareness for the users, e.g. via docstrings.
-
An optional retry method on batch level can be provided for the whole batch, but the default strategy must contain a backoff - solution in case of continued errors to publish to Nakadi.
-
An optional retry method can be provided that only re-publishes unsuccessful events to Nakadi. This retry must also support a backoff strategy by default.
-
Clients must expose the result of a publishing request in a way that developers can understand that there is the possibility of a partial success for batch publishing.
Hi @adyach Thank you for reaching out,
At the moment there should be a backoff logic that retries the publishing. And the result of partial publishing is already exposed to the clients.
I'll update the readme to clarify the behavior.
@adyach sorry, being quite busy :( will finish the doc by the end of tomorrow (Oct 10, 2023)
@adyach just made a PR, please take a look: https://github.com/zalando-nakadi/kanadi/pull/219