[Question] How to make `max_batch_size` configurable?

[Yevgnen]

Hi,

I'm trying to avoid writing

        config = {
            max_batch_size: 32,
        };

and use

        config = {
            max_batch_size: max_batch_size,
        };

instead so that I can adjust the batch size dynamically or read the config from some variable/setting. But I got errors like

error[E0435]: attempt to use a non-constant value in a constant
  --> src/http/predict.rs:76:61
   |
70 |       let handler = batched_fn::batched_fn! {
   |  ___________________-
71 | |         handler = |batch: Vec<Input>, model: &SentimentModel| -> Vec<Output> {
72 | |             let predictions = model.predict(&batch.iter().map(String::as_str).collect::<Vec<&str>>().as_slice());
73 | |             predictions.iter().map(|x| Response { score: x.score }).collect()
...  |
76 | |             max_batch_size: if Cuda::cudnn_is_available() { batch_size } else { 1 },
   | |                                                             ^^^^^^^^^^ non-constant value
...  |
85 | |         };
86 | |     };
   | |_____- help: consider using `let` instead of `static`: `let BATCHED_FN`

For more information about this error, try `rustc --explain E0435`.
error: could not compile `axum-sst` due to previous error

Also it seems that the batch size can not be adjusted once the thread is started. Even so, how can I pass max_batch_size by a variable when initializing? Must the passed max_batch_size be static or const? e.g.

struct Config {
    max_batch_size: usize,
}

static PREDICT_CONFIG: Config = Config { max_batch_size: 32 }; // or `Config.from_file`...

...
        config = {
            max_batch_size: if Cuda::cudnn_is_available() { PREDICT_CONFIG.max_batch_size } else { 1 },
        };

Thanks.

[epwalsh]

Ah, I see what the issue is. Thanks for bringing that up. #21 should solve this, but requires a small breaking change:

Instead of:

let handler = batched_fn! { ... };
return handler(input).await;

You would have to change the last line to:

return handler.evaluate_in_batch(input).await;

[epwalsh]

Ugh, actually that won't work. There might be another way around this, but I'll have to put some more thought into it. I'll get back to you.

[epwalsh]

So, the reason configuration options need to be static is because the batched_fn! macro expands to define a BatchedFn instance and worker thread statically. These things need to be defined statically because batched_fn! is meant to be dropped into a function/method, and if they weren't declared static then they would be recreated every time the function/method is called.

So there is probably no way we could allow non-static expressions to be passed into batched_fn!. However, it would be possible to change the value of certain configuration options (like max_batch_size) on-the-fly. We'd just need to provide a Channel to communicate changes from the main/calling thread to the worker thread.