Deepthi Devaki Akkoorath

retries=3 ``` > zdb state list -cf=JOBS -p snapshots/481713-2-2347316-2347287 {"data":[ {"cf":"JOBS","key":"00 00 00 00 00 00 00 10 00 10 00 00 00 0b 2f f6","value":{"jobRecord":{"deadline":1710409644293,"timeout":-1,"worker":"worker#sendMessagePing","retries":3,"retryBackoff":30000,"recurringTime":1710409615694,"type":"send-message-ping","customHeaders":{},"variables":"gA==","errorMessage":"","errorCode":"","bpmnProcessId":"ping-pong-message","processDefinitionVersion":1,"processDefinitionKey":2251799813685606,"processInstanceKey":4503599628092493,"jobKind":"BPMN_ELEMENT","elementId":"Activity_0k20esx","elementInstanceKey":4503599628103669,"tenantId":""}}}, {"cf":"JOBS","key":"00 00 00 00...

No. back-off time is [set to 30s](https://github.com/camunda/zeebe-e2e-test/blob/9c0673375fdd1385cc59296ec419ce76ce866b5d/loadgen/src/main/java/io/camunda/zeebe/e2e/loadgen/worker/Worker.java#L31). So failed jobs should be activatable again after 30s.

@nicpuppa The logs are still available in the link provided. Could you check it to verify @oleschoenburg 's hypothesis? > Also this log > Expected to find a subscription with...

I don't think warnings related to subscriptions are relevant for this issue. Unfortunately, the cluster is deleted. So we cannot get the data of other partitions.

Me and @nicpuppa looked into it. We have a hypothesis, but could not really reproduce the failure to verify it yet. We think it is a race-condition when back-off checker...

It seems @korthout and @oleschoenburg is looking [into a similar issue with timers](https://camunda.slack.com/archives/C06QDDLEYQZ) and came to the same conclusion that DueDateChecker is not thread-safe. @nicpuppa Please sync with @korthout so...

There are several e2e tests failed this week and the symptoms looks similar to this bug. FYI @nicpuppa

Sequencer queue size is 128, which might be too small for a high load test. https://github.com/camunda/zeebe/blob/29204ad74980ecae6f88729bde83e7d6ac2e8c9a/logstreams/src/main/java/io/camunda/zeebe/logstreams/impl/log/Sequencer.java#L44

:bell: Revert https://github.com/camunda/zeebe/pull/18512/commits/43187d32655de695b5235fe585ee232ed1c62ea2 when this issue is implemented.

I implemented the solution that we discussed [here](https://github.com/camunda/camunda/issues/18030#issuecomment-2133473576) to support default "enabled" for exporters added in static configuration. It seems feasible. But there is one potential problem during rolling update....