ROCmSoftwarePlatform
Embed onnx/tf files for api tests
Codecov Report
Attention: Patch coverage is 90.90909% with 1 line in your changes missing coverage. Please review.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/api/api.cpp | 87.50% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## develop #3905 +/- ##
===========================================
- Coverage 92.18% 92.12% -0.05%
===========================================
Files 528 528
Lines 24307 24308 +1
===========================================
- Hits 22405 22393 -12
- Misses 1902 1915 +13
| Files with missing lines | Coverage Δ | |
|---|---|---|
| src/api/include/migraphx/migraphx.hpp | 98.96% <100.00%> (-0.02%) |
:arrow_down: |
| src/api/api.cpp | 73.24% <87.50%> (-0.41%) |
:arrow_down: |
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Windows failures....
jom: C:\home\jenkins\agent\workspace\UIF2_AMDMIGraphX_PR-3905\AMDMIGraphX\build\WinRelease\test\api\CMakeFiles\test_api_custom_op.dir\build.make [bin\test_api_custom_op.exe] Error 1 -- | 33333 | 01:49:43 PM | jom: C:\home\jenkins\agent\workspace\UIF2_AMDMIGraphX_PR-3905\AMDMIGraphX\build\WinRelease\CMakeFiles\Makefile2 [test\api\CMakeFiles\test_api_custom_op.dir\all] Error 2 | 33334 | 01:49:44 PM | [100%] Linking CXX executable ..\..\bin\test_api_gpu.exe | 33335 | 01:49:44 PM | lld-link: error: duplicate symbol: class std::basic_string_view
| Test | Batch | Rate new e480e3 |
Rate old 1403f8 |
Diff | Compare |
|---|---|---|---|---|---|
| torchvision-resnet50 | 64 | 3,257.11 | 3,236.05 | 0.65% | :white_check_mark: |
| torchvision-resnet50_fp16 | 64 | 6,923.94 | 6,908.21 | 0.23% | :white_check_mark: |
| torchvision-densenet121 | 32 | 2,455.68 | 2,445.72 | 0.41% | :white_check_mark: |
| torchvision-densenet121_fp16 | 32 | 4,218.04 | 4,214.85 | 0.08% | :white_check_mark: |
| torchvision-inceptionv3 | 32 | 1,626.83 | 1,619.30 | 0.47% | :white_check_mark: |
| torchvision-inceptionv3_fp16 | 32 | 2,719.01 | 2,710.88 | 0.30% | :white_check_mark: |
| cadene-inceptionv4 | 16 | 760.79 | 756.76 | 0.53% | :white_check_mark: |
| cadene-resnext64x4 | 16 | 818.73 | 814.77 | 0.49% | :white_check_mark: |
| slim-mobilenet | 64 | 7,472.93 | 7,426.54 | 0.62% | :white_check_mark: |
| slim-nasnetalarge | 64 | 217.13 | 215.94 | 0.55% | :white_check_mark: |
| slim-resnet50v2 | 64 | 3,348.70 | 3,334.83 | 0.42% | :white_check_mark: |
| bert-mrpc-onnx | 8 | 1,151.86 | 1,141.57 | 0.90% | :white_check_mark: |
| bert-mrpc-tf | 1 | 457.94 | 458.12 | -0.04% | :white_check_mark: |
| pytorch-examples-wlang-gru | 1 | 350.79 | 346.96 | 1.11% | :white_check_mark: |
| pytorch-examples-wlang-lstm | 1 | 474.05 | 473.57 | 0.10% | :white_check_mark: |
| torchvision-resnet50_1 | 1 | 816.09 | 817.07 | -0.12% | :white_check_mark: |
| cadene-dpn92_1 | 1 | 431.38 | 433.49 | -0.49% | :white_check_mark: |
| cadene-resnext101_1 | 1 | 393.43 | 391.93 | 0.38% | :white_check_mark: |
| onnx-taau-downsample | 1 | 396.07 | 394.92 | 0.29% | :white_check_mark: |
| dlrm-criteoterabyte | 1 | 32.33 | 32.20 | 0.38% | :white_check_mark: |
| dlrm-criteoterabyte_fp16 | 1 | 51.27 | 51.15 | 0.23% | :white_check_mark: |
| agentmodel | 1 | 10,564.84 | 7,957.29 | 32.77% | :high_brightness: |
| unet_fp16 | 2 | 59.51 | 59.35 | 0.27% | :white_check_mark: |
| resnet50v1_fp16 | 1 | 1,076.84 | 1,089.23 | -1.14% | :white_check_mark: |
| resnet50v1_int8 | 1 | 1,051.47 | 1,067.66 | -1.52% | :white_check_mark: |
| bert_base_cased_fp16 | 64 | 1,170.48 | 1,163.27 | 0.62% | :white_check_mark: |
| bert_large_uncased_fp16 | 32 | 357.91 | 356.00 | 0.54% | :white_check_mark: |
| bert_large_fp16 | 1 | 200.78 | 199.24 | 0.77% | :white_check_mark: |
| distilgpt2_fp16 | 16 | 2,239.57 | 2,225.96 | 0.61% | :white_check_mark: |
| yolov5s | 1 | 536.84 | 541.46 | -0.85% | :white_check_mark: |
| tinyllama | 1 | 44.02 | 43.66 | 0.83% | :white_check_mark: |
| vicuna-fastchat | 1 | 45.05 | 44.89 | 0.36% | :white_check_mark: |
| whisper-tiny-encoder | 1 | 420.44 | 419.73 | 0.17% | :white_check_mark: |
| whisper-tiny-decoder | 1 | 412.46 | 411.32 | 0.28% | :white_check_mark: |
| llama2_7b | 1 | nan | nan | nan% | :x: |
| qwen1.5-7b | 1 | 23.54 | 23.45 | 0.40% | :white_check_mark: |
| phi3-3.8b | 1 | nan | nan | nan% | :x: |
| mask-rcnn | 1 | 12.77 | 12.82 | -0.43% | :white_check_mark: |
| llama3-8b | 1 | 21.77 | 21.68 | 0.39% | :white_check_mark: |
| whisper-large-encoder | 1 | 10.22 | 10.17 | 0.48% | :white_check_mark: |
| whisper-large-decoder | 1 | 99.81 | 100.07 | -0.26% | :white_check_mark: |
| mistral-7b | 1 | 23.74 | 23.65 | 0.38% | :white_check_mark: |
| FLUX.1-schnell | 1 | 916.89 | 913.14 | 0.41% | :white_check_mark: |
| nan | nan | nan | nan | nan% | :x: |
This build is not recommended to merge :red_circle:
:x:bert-mrpc-tf: ERROR - check error output
2025-04-28 01:16:40.500151: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1745821005.866684 163013 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 62973 MB memory: -> device: 0, name: AMD Instinct MI250X/MI250, pci bus id: 0000:b3:00.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1745821006.706937 163013 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2025-04-28 01:16:55.296610: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-28 01:16:55.296785: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-28 01:16:55.296835: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-28 01:16:55.296885: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-28 01:16:55.296941: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-28 01:16:55.296973: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-28 01:16:55.297026: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-28 01:16:55.297081: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
2025-04-28 01:16:55.298051: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-04-28 01:16:55.299159: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-04-28 01:16:55.299179: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
2025-04-28 01:16:55.299188: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
2025-04-28 01:16:55.299203: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 11217777527359497193
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1407, in _do_call
return fn(*args)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1390, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1483, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 324, in main
y_out = sess.run(y, feed_dict=tf_dict)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 977, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1220, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1400, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1426, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'::red_circle:unet: FAILED: MIGraphX is not within tolerance - check verbose output
:red_circle:bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output
:x:llama2_7b: ERROR - check error output
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:265: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/llama2_7b/decoder_model.onnx:x:qwen1.5-7b: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256:x:phi3-3.8b: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256:x:mask-rcnn: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: 3 800 800:x:#whisper-large-encoder: ERROR - check error output
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/include/migraphx/op/convolution.hpp:100: normalize_compute_shape: CONVOLUTION: mismatched channel numbers