Whisper medium error with DirectML

[thewh1teagle]

When running whisper medium (int8 / normal) on DirectML provider it failed. on CPU it works. same build.

The error

Non-zero status code returned while running MemcpyToHost node

Reproduce

cmake -B build -DCMAKE_BUILD_TYPE=Release -DSHERPA_ONNX_ENABLE_DIRECTML=ON -DSHERPA_ONNX_ENABLE_PORTAUDIO=OFF -DSHERPA_ONNX_ENABLE_WEBSOCKET=OFF -DSHERPA_ONNX_ENABLE_TTS=OFF -DSHERPA_ONNX_ENABLE_BINARY=ON -DSHERPA_ONNX_BUILD_C_API_EXAMPLES=ON .

Change provider to DirectML in whiser-c-api and medium.int8

cmake --build build --config Release --target whisper-c-api

Result (failed)

Time taken for create recognizer: 13.49 seconds
2024-08-09 17:22:45.9754547 [E:onnxruntime:, sequential_executor.cc:494 onnxruntime::ExecuteKernel] Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_244' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2429)\onnxruntime.dll!00007FF9B54FE1D6: (caller: 00007FF9B5A25FC7) Exception(3) tid(2f08) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

D:\sherpa\sherpa-onnx\sherpa-onnx/csrc/offline-recognizer-whisper-impl.h:DecodeStream:176 

Caught exception:

Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_244' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2429)\onnxruntime.dll!00007FF9B54FE1D6: (caller: 00007FF9B5A25FC7) Exception(3) tid(2f08) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.


Return an empty result. Number of input frames: 663, Current tail paddings: 1000. If you see a lot of such exceptions, please consider using a larger --whisper-tail-paddings
Decoded text:
Time taken: 2.83 seconds
Decoded text:

Change provider to CPU in whisper-c-api and medium.int8

cmake --build build --config Release --target whisper-c-api

Result (success)

Time taken for create recognizer: 10.05 seconds
Decoded text:  After early nightfall the yellow lamps would light up here and there the squalid quarter of the brothels.
Time taken: 15.26 seconds
Decoded text:  After early nightfall the yellow lamps would light up here and there the squalid quarter of the brothels.

Change provider to DirectML and model to tiny.int8

Result (success)

Time taken for create recognizer: 1.39 seconds
Decoded text:  After early nightfall, the yellow lamps would light up here and there the squalid quarter of the brothels.
Time taken: 2.25 seconds
Decoded text:  After early nightfall, the yellow lamps would light up here and there the squalid quarter of the brothels.

Maybe related: https://github.com/dog-qiuqiu/Ultralight-SimplePose/issues/5 https://github.com/microsoft/onnxruntime/issues/3205 https://github.com/k2-fsa/sherpa-onnx/pull/1037

By the way I can that it loads some of the model into the GPU memory. like 2GB and then fail (I have 8GB GPU)

Happens also in test whisper script

Maybe a fix: https://github.com/onnx/models/issues/156#issuecomment-690847276