Dynamically adjusting MaxConcurrentActivityExecutionSize without restarting a worker

[pattersp]

Is your feature request related to a problem? Please describe. Not related to problem, just a feature that would be nice to have. Currently, it is possible to set MaxConcurrentActivityExecutionSize when starting a worker. It would nice to be able to dynamically adjust this value without restarting the worker.

Describe the solution you'd like Without restarting a worker, I would like to be able to adjust the value of MaxConcurrentActivityExecutionSize.

If the value is decreased and the worker is subsequently executing a number of activities that exceeds the threshold, that is fine -- it can be more of an "eventual" threshold.

Describe alternatives you've considered As a workaround solution, we could restart the worker to reset MaxConcurrentActivityExecutionSize, but that's a lot of overhead.

Additional context See https://community.temporal.io/t/dynamically-adjusting-the-number-of-concurrent-activities-on-a-given-worker/3052

[cretz]

At this time, dynamically adjusting such a value is incredibly difficult on the internals due to how pool-transfer would have to work. I'm afraid the current best way to increase the size is to start a new worker.

I am leaving this issue opened to potentially solve in our next iteration of the SDK where the internals may be replaced by a different core and make such a task easier.

[cretz]

Rethinking this a bit, we could basically expose pollerRequestCh, but we also have to allow control over task completion and get a slot count for metrics. I think we could have this interface:

type SlotLimiter interface {
    AcquireSlot()
    ReleaseSlot()
    AvailableSlots() int
}

Then have a default slot limiter that uses today's approach (fixed max concurrent), and have advanced worker options for ActivityTaskSlotLimiter SlotLimiter and WorkflowTaskSlotLimiter SlotLimiter.