AMDMIGraphX
AMDMIGraphX copied to clipboard
ROCmSoftwarePlatform
change (cerr -> cout) for consistency across all error messages
~~A few error messages are getting piped to cerr, while most go to cout. Thus, making it consistent.~~
Channeling warnings that stream to cout, instead into cerr.
(Ideally the verify failures and some other test exceptions should also go to cerr, but that isn't handled here.)
Codecov Report
Attention: Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/include/migraphx/op/pooling.hpp | 0.00% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## develop #3878 +/- ##
========================================
Coverage 92.10% 92.10%
========================================
Files 528 528
Lines 24329 24329
========================================
Hits 22407 22407
Misses 1922 1922
| Files with missing lines | Coverage Δ | |
|---|---|---|
| src/include/migraphx/op/scatternd_op.hpp | 97.96% <100.00%> (ø) |
|
| src/load_save.cpp | 97.14% <ø> (ø) |
|
| src/program.cpp | 70.43% <ø> (ø) |
|
| src/include/migraphx/op/pooling.hpp | 93.45% <0.00%> (ø) |
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Error messages should go to
cerrnotcout.
Currently, there is a mish-mash of these two approaches in MigraphX, with messages being piped mostly to cout; and yet, some directed to cerr. Warnings and Exceptions are being dumped on either stream as well.
This simple PR makes it all conformant.
Warnings and Exceptions are being dumped on either stream as well.
These should go to cerr unless they are being used in some kind of trace(like in MIGRAPHX_TRACE_ELIMINATE_CONTIGUOUS) or reporting test failures(like rms failure in verify_args).
| Test | Batch | Rate new af5462 |
Rate old 827407 |
Diff | Compare |
|---|---|---|---|---|---|
| torchvision-resnet50 | 64 | 3,256.42 | 3,257.01 | -0.02% | :white_check_mark: |
| torchvision-resnet50_fp16 | 64 | 6,920.77 | 6,920.80 | -0.00% | :white_check_mark: |
| torchvision-densenet121 | 32 | 2,454.63 | 2,455.34 | -0.03% | :white_check_mark: |
| torchvision-densenet121_fp16 | 32 | 4,220.54 | 4,219.88 | 0.02% | :white_check_mark: |
| torchvision-inceptionv3 | 32 | 1,628.60 | 1,629.21 | -0.04% | :white_check_mark: |
| torchvision-inceptionv3_fp16 | 32 | 2,719.56 | 2,719.40 | 0.01% | :white_check_mark: |
| cadene-inceptionv4 | 16 | 761.51 | 760.24 | 0.17% | :white_check_mark: |
| cadene-resnext64x4 | 16 | 818.59 | 820.18 | -0.19% | :white_check_mark: |
| slim-mobilenet | 64 | 7,473.57 | 7,471.68 | 0.03% | :white_check_mark: |
| slim-nasnetalarge | 64 | 211.01 | 210.92 | 0.04% | :white_check_mark: |
| slim-resnet50v2 | 64 | 3,349.35 | 3,349.61 | -0.01% | :white_check_mark: |
| bert-mrpc-onnx | 8 | 1,151.46 | 1,152.38 | -0.08% | :white_check_mark: |
| bert-mrpc-tf | 1 | 456.90 | 457.04 | -0.03% | :white_check_mark: |
| pytorch-examples-wlang-gru | 1 | 337.63 | 337.96 | -0.10% | :white_check_mark: |
| pytorch-examples-wlang-lstm | 1 | 476.68 | 476.12 | 0.12% | :white_check_mark: |
| torchvision-resnet50_1 | 1 | 814.78 | 809.78 | 0.62% | :white_check_mark: |
| cadene-dpn92_1 | 1 | 434.43 | 432.33 | 0.49% | :white_check_mark: |
| cadene-resnext101_1 | 1 | 393.18 | 393.50 | -0.08% | :white_check_mark: |
| onnx-taau-downsample | 1 | 395.92 | 395.62 | 0.08% | :white_check_mark: |
| dlrm-criteoterabyte | 1 | 32.33 | 32.32 | 0.03% | :white_check_mark: |
| dlrm-criteoterabyte_fp16 | 1 | 51.26 | 51.25 | 0.03% | :white_check_mark: |
| agentmodel | 1 | 10,162.56 | 10,070.70 | 0.91% | :white_check_mark: |
| unet_fp16 | 2 | 59.65 | 59.52 | 0.22% | :white_check_mark: |
| resnet50v1_fp16 | 1 | 1,056.39 | 1,081.95 | -2.36% | :white_check_mark: |
| resnet50v1_int8 | 1 | 1,062.07 | 1,052.00 | 0.96% | :white_check_mark: |
| bert_base_cased_fp16 | 64 | 1,175.11 | 1,174.88 | 0.02% | :white_check_mark: |
| bert_large_uncased_fp16 | 32 | 358.03 | 358.02 | 0.00% | :white_check_mark: |
| bert_large_fp16 | 1 | 200.09 | 199.38 | 0.35% | :white_check_mark: |
| distilgpt2_fp16 | 16 | 2,238.31 | 2,242.42 | -0.18% | :white_check_mark: |
| yolov5s | 1 | 540.02 | 534.10 | 1.11% | :white_check_mark: |
| tinyllama | 1 | 43.84 | 43.89 | -0.12% | :white_check_mark: |
| vicuna-fastchat | 1 | 45.10 | 45.13 | -0.07% | :white_check_mark: |
| whisper-tiny-encoder | 1 | 421.33 | 420.85 | 0.11% | :white_check_mark: |
| whisper-tiny-decoder | 1 | 412.13 | 412.21 | -0.02% | :white_check_mark: |
| llama2_7b | 1 | nan | nan | nan% | :x: |
| qwen1.5-7b | 1 | 23.55 | 23.55 | 0.01% | :white_check_mark: |
| phi3-3.8b | 1 | 26.64 | 26.64 | 0.00% | :white_check_mark: |
| mask-rcnn | 1 | 12.74 | 12.73 | 0.03% | :white_check_mark: |
| llama3-8b | 1 | 21.74 | 21.75 | -0.01% | :white_check_mark: |
| whisper-large-encoder | 1 | 10.22 | 10.22 | 0.00% | :white_check_mark: |
| whisper-large-decoder | 1 | 100.80 | 102.53 | -1.68% | :white_check_mark: |
| mistral-7b | 1 | 23.75 | 23.75 | 0.01% | :white_check_mark: |
| FLUX.1-schnell | 1 | 906.63 | 908.25 | -0.18% | :white_check_mark: |
| nan | nan | nan | nan | nan% | :x: |
This build is not recommended to merge :red_circle:
:x:bert-mrpc-tf: ERROR - check error output
2025-05-16 15:38:22.368627: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1747427907.729673 163740 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 62973 MB memory: -> device: 0, name: AMD Instinct MI250X/MI250, pci bus id: 0000:b3:00.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1747427908.582413 163740 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2025-05-16 15:38:37.929388: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-16 15:38:37.929429: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-16 15:38:37.929470: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-16 15:38:37.929516: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-16 15:38:37.929568: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-16 15:38:37.929618: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-16 15:38:37.929667: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-16 15:38:37.929715: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
2025-05-16 15:38:37.930724: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-05-16 15:38:37.931836: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-05-16 15:38:37.931856: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
2025-05-16 15:38:37.931868: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
2025-05-16 15:38:37.931886: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 11217777527359497193
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1407, in _do_call
return fn(*args)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1390, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1483, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 324, in main
y_out = sess.run(y, feed_dict=tf_dict)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 977, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1220, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1400, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1426, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'::red_circle:unet: FAILED: MIGraphX is not within tolerance - check verbose output
:red_circle:bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output
:x:llama2_7b: ERROR - check error output
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:265: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/llama2_7b/decoder_model.onnx:x:qwen1.5-7b: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256:x:phi3-3.8b: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256:x:mask-rcnn: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: 3 800 800:x:#whisper-large-encoder: ERROR - check error output
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/include/migraphx/op/convolution.hpp:100: normalize_compute_shape: CONVOLUTION: mismatched channel numbers