"Maximum batch size" and "Maximum batch time" are not overwritten through env variables

[saeid93]

trafficstars

if the two variable max_batch_time and max_batch_size are defined in the model-settings.json:

{
    "name": "node-1",
    "implementation": "models.NodeOne",
    "max_batch_size": 5,
    "max_batch_time": 1,
    "parameters": {
        "uri": "./fakeuri"
    }
}

Then they couldn't be overwritten by environment variables MLSERVER_MODEL_MAX_BATCH_SIZE and MLSERVER_MODEL_MAX_BATCH_TIME and mlserver will always revert back to the values set in the model-settings.json. However, it is expected to use the values in the json as defaults and env variable as user arguments for overwriting them.

[adriangonz]

Hey @saeid93 ,

Yeah, it's a known issue that, when using mlserver build, local setting files override environment variables. This is due to how they get injected during the container's startup (which is essentially by injecting them as env vars as well).

Having said that, there wasn't an issue for that yet, so thanks for opening one!

[saeid93]

Hey @adriangonz,

Sure, no problem. Since the setting.json is also injected by env vars why the same env var is not used for the initialization as well? so it will be overwritten by the user-defined var?

[adriangonz]

Hey @saeid93 ,

When you build a custom image and it has a settings.json present, the same issue should occur. As in, the values set up on that local settings.json will get injected through env vars, overriding whatever else comes from upstream.