Allow dynamic batch scheduler to be disabled when autocompleting config

[OvervCW]

Is your feature request related to a problem? Please describe. I'm serving a model that supports batching (max_batch_size > 0) and I would like to use config autocomplete, but I don't want to use the dynamic batch scheduler.

The reason that we want to disable it is because there's currently a bug in the dynamic batch scheduler that we're running into (https://github.com/triton-inference-server/server/issues/4925).

The autocomplete code currently always selects the dynamic batch scheduler when batching is possible and no other scheduler has been specified:

https://github.com/triton-inference-server/onnxruntime_backend/blob/1e5dd03fd18992446fd169ead8bf208e8fc53686/src/onnxruntime.cc#L800-L814

This means that there is currently no way to use the default scheduler when autocompleting the config for a model that supports batching.

Describe the solution you'd like I would like there to be an option to disable the dynamic batch scheduler when using config autocomplete, possibly through a backend config parameter. For example, --backend-config=onnxruntime,autocomplete-scheduler=default-scheduler.

Describe alternatives you've considered As far as I can tell there is currently no way to disable this behavior except by disabling batching completely.

[dyastremsky]

Thanks for submitting this feature request! Looking at the code, you'd probably be able to disable it by setting the max_batch_size to 1. However, that's currently not documented and may not be the ideal solution. @nv-kmcgill53, what are your thoughts?

[OvervCW]

Setting max_batch_size to 1 has the unintended side effect of disabling client side batching as well, though.

[dyastremsky]

I see, thanks Alexander. Filing a ticket for this feature request.

[nv-kmcgill53]

The underlying issue of wanting to select a scheduler which you cannot currently select is the root of this problem. If we were to allow the default_scheduler as an option in config.pbtxt then the scheduler would be set and autocomplete would not override this choice. Would my solution work for you @OvervCW? With regard to this problem, autocomplete sees ambiguity when no scheduler is set; there is no way of knowing whether the user chose to omit the scheduler or if they want it filled in. I agree that a change needs to be made. cc @tanmayv25

[tanmayv25]

Adding default_scheduler option in oneof field for scheduling_choice looks good to me.

[OvervCW]

@nv-kmcgill53 Yeah, being able to specify the default scheduler in the config would be a perfect solution!