onnx disabled optimizers for dropout

[zhaozhiming37]

My model includes Dropout module for inference, and when I run my model by onnxruntime locally, I will set disabled_optimizers=["EliminateDropout"]. And I want to know how can I do that by triton server? My code is like this:

session = ort.InferenceSession(
        onnxFile,
        disabled_optimizers=["EliminateDropout"],
        providers=[
            'TensorrtExecutionProvider',
            # 'CUDAExecutionProvider',
            # 'CPUExecutionProvider'
        ]
    )

[Tabrizian]

I don't know whether we expose this parameter in the model configuration. @pranavsharma Do you know whether it is possible to use this option when serving the models using onnxruntime backend?

[zhaozhiming37]

@pranavsharma Hi, is there an update on my question?

[zhaozhiming37]

@Tabrizian Hi, I wonder if there is an answer to the question or if you have plans to support?

[Tabrizian]

Hi @zhaozhiming37, sorry for the delayed resposne. ONNXRuntime backend is managed by the Microsoft team so they should be able to provide more info.

[pranavsharma]

This has not been exposed yet. The best way to do this is to create the session offline and serialize it and then use the serialized ONNX model in Triton.