Feature Request: per connection batch "priority"

[brendanhoar]

The concept would be:

If you have more than one window/tab open, you can run a large batch in one tab and experiment with parameters in the other tab.

If there were a way to set priority per connection or as a parameter, one could continue to (somewhat more slowly) iterate through parameter changes while not having to stop the large batch if a "higher priority" request was received at the backend.

I could see the utility of this even if the initial release of the feature required that both windows/tabs used the same checkpoint model, as that avoids either slowness of swapping models or complex logic of which models to cache in ram to ensure short swap times. Long term, it would also be nice to be able to do this across checkpoints as well.

Yes, yes, I know, multiple backends (and GPUs) is the recommended approach here, but I dunno, supporting it on a single GPU would be nice.