advance ack-level to avoid querying the same (empty) tasks next time

[dkrotx]

Every time pollers are reaching out matching we acuire a new lease in the DB extending maxReadLevel. Unfortunately, if there is no writes happening to the task-list, we are pushing maxReadLevel further and further away from the previous ackLevel (there is nothing to ack!). At some moment after 1000 restarts of matching service we could have 1000 (empty) gettask requests and drain the whole matching.PersistenceMaxQPS which will reject writes to other [active] task-lists with "Max QPS reached" error.

We are advancing ack-level for task-lists even when read zero tasks.

This is required to prevent spiky load to DB / hitting rate-limit after cadence-matching restart.

Will do on staging environment, it is Draft by now.

We could skip some tasks making workflows to stuck

Release notes

Documentation Changes