nathanjacobiOXOS

I have my set as max_batch_size: 0 Any other size will yield: Internal: autofill failed for model 'Object_Detection': configuration specified max-batch 1 but TensorRT engine only supports max-batch 0

Hi, following up here @Tabrizian , any ideas on what to try?

Hey there @Tabrizian checking in again, any help would be greatly appreciated

Hi @Tabrizian, it's been about a month with no word, any sort of update would be appreciated

Hi @dyastremsky @Tabrizian Here is the verbose output: "name": "Object_Detection", "platform": "tensorrt_plan", "backend": "tensorrt", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1, "input": [ { "name": "input", "data_type":...

And the result I0221 16:47:44.410786 34238 server.cc:673] +------------------+---------+-------------------------------------------------------------------------------------------------------------------+ | Model | Version | Status | +------------------+---------+-------------------------------------------------------------------------------------------------------------------+ | Object_Detection | 1 | UNAVAILABLE: Invalid argument: unexpected vectorized dim is -1 for...

In the meantime, I have been exploring a custom built tensorrt engine, and have had no issues deserializing this exact same model.plan and inputing the HWC formatted data

Here is this as well if it is helpful 0221 16:47:42.293758 34238 model_config_utils.cc:1839] ModelConfig 64-bit fields: I0221 16:47:42.293848 34238 model_config_utils.cc:1841] ModelConfig::dynamic_batching::default_queue_policy::default_timeout_microseconds I0221 16:47:42.293896 34238 model_config_utils.cc:1841] ModelConfig::dynamic_batching::max_queue_delay_microseconds I0221 16:47:42.293938 34238 model_config_utils.cc:1841]...

Hi there, the issue stems from [this location](https://github.com/triton-inference-server/tensorrt_backend/blob/0f0c551b249978d93cd29e1519f85f22aab034d8/src/instance_state.cc#L3988) in tensorrt_backend. I have confirmed that the tensorRT plan is format kHWC `instance_->engine_->getBindingFormat(binding_index) == nvinfer1::TensorFormat::kHWC` will return true. I have not been...

There seems to be inconsistency in the handling of kHWC vs nvinfer documentation. From the documentation of nvinfer, kHWC is described as "Non-vectorized channel-last format." This [line](https://github.com/triton-inference-server/tensorrt_backend/blob/0f0c551b249978d93cd29e1519f85f22aab034d8/src/instance_state.cc#L3998) appears to assume...