Scott McKay

What's the reason for attempting to use the ORT file format in the GPU scenarios? ORT format is targeting mobile/edge scenarios where binary size is critical, so the current expected...

ONNX format files are supported on all platforms. It's just that the binary size of the ORT library will be bigger vs. a minimal build that only supports ORT format...

Example of reading bytes from file: https://github.com/microsoft/onnxruntime/blob/894fc828587c919d815918c4da6cde314e5d54ed/onnxruntime/test/shared_lib/test_model_loading.cc#L21-L31 The bytes are just passed directly when creating the inference session. https://github.com/microsoft/onnxruntime/blob/894fc828587c919d815918c4da6cde314e5d54ed/onnxruntime/test/shared_lib/test_model_loading.cc#L41 We'll look into the DML issue as it should be possible...

Regarding the DML support, the DML EP has two different ways of handling parts of the graph. One is with statically registered kernels, and one is with dynamically created kernels....

POC for adding support for DML when using an ORT format model: https://github.com/microsoft/onnxruntime/compare/skottmckay/ORT_model_support_with_DML_EP Technically we could create the ORT format model with just basic optimizations and DML disabled to not...

The DML EP makes some changes to the model during partitioning that are not really expected by ORT. Essentially it does a const_cast and steals initializers for memory usage reasons,...

@gedoensmax If you have a model with dynamic dimensions and want to make them fixed, you could use this tool: https://onnxruntime.ai/docs/reference/mobile/make-dynamic-shape-fixed.html I don't quite understand how model load time would...

Unfortunately ORT doesn't have a way to general way to save a compiled node. The TensorRT EP is doing that via TensorRT's ability to save but AFAIK that is the...

The underlying issue is that the installation instructions and folder setup for cudnn keep changing. The ORT cmake file needs to handle this somewhat insane number of combinations, as depending...

Should this be a separate build/package so a user has a choice of a base ORT package or a much larger one with extensions? Or will we see how big...