Test Kafka ingestion under load

[tillrohrmann]

We should know and understand the limits of the current Kafka ingestion. Especially handling bursty ingestion rates in combination with the invoker's concurrency limit should be tested.

[slinkydeveloper]

I have some tools I used last time for testing Kafka + Restate:

• xk6 for sending load to kafka: https://github.com/mostafa/xk6-kafka
• Grafana + Prometheus to observe
• Rust SDK as receiver, doing just hello world

I added all the notes + config files here: https://drive.google.com/file/d/1j72zZPG_RJuwmQ02GI5lhQB_UBrhJ5gA/view?usp=sharing

[joebowbeer]

@tillrohrmann what did you mean by "invoker's concurrency limit"?

[tillrohrmann]

The invoker has a concurrency limit configurable via https://docs.restate.dev/references/server-config#param-concurrent-invocations-limit which controls how many concurrent invocations can be executed by the invoker. There is one invoker per partition. This is primarily to protect the system from overloading the service endpoint. This is not a perfect solution and will hopefully be replaced soon: https://github.com/restatedev/restate/issues/3602