ROCmSoftwarePlatform
Adding MIGRAPHX_TIME_MATCHERS
Codecov Report
:x: Patch coverage is 42.85714% with 8 lines in your changes missing coverage. Please review.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/include/migraphx/matcher.hpp | 42.86% | 8 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## develop #3500 +/- ##
===========================================
- Coverage 92.12% 92.10% -0.02%
===========================================
Files 528 528
Lines 24310 24319 +9
===========================================
+ Hits 22395 22398 +3
- Misses 1915 1921 +6
| Files with missing lines | Coverage Δ | |
|---|---|---|
| src/include/migraphx/matcher.hpp | 94.25% <42.86%> (-1.80%) |
:arrow_down: |
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
- The time measurements should be done in the
find_matches_forfunction.
@pfultz2 does every single matcher go through find_matches_for? If we're going to add this feature to the develop branch then I suggest you add it for every matcher, not just the selected ones that Aarushi added the code to.
- There should also be an environment variable to enable it like
MIGRAPHX_TIME_MATCHERS.
This is a good way to globally enable/disable match timing. But maybe we also want to enable/disable timing programmatically for individual matchers? Aarushi, in your experience would this be helpful?
- Trace filter should probably be used to reduce down the amount of printouts.
I suggest wrapping the timer function in a macro so you can easily insert/remove it from the list in individual passes' find_matches() methods, eg.
void simplify_algebra::apply(module& m) const
{
// Run simplifications multiple times
m.repeat_while_changes(8, [&] {
match::find_matches(m,
LOG_MATCHER_TIMER(find_inner_broadcast{}), <===new macro
find_dot_broadcast{},
etc.
which would be less repetitive work than wrapping each matcher in curly brackets as Aarushi has done.
What do you think?
does every single matcher go through find_matches_for?
Yes.
This is a good way to globally enable/disable match timing. But maybe we also want to enable/disable timing programmatically for individual matchers?
The MIGRAPHX_TRACE_MATCHES_FOR lets you filter the matchers if we use it for timing as well.
What do you think?
I would like to avoid needing to change the source code to get timings. We dont do it for the passes. This will let us get timings on deployed systems so we dont need to rebuild migraphx.
I fixed the timing logic. Need doc update for the new environment variable.
| Test | Batch | Rate new f33515 |
Rate old 03922b |
Diff | Compare |
|---|---|---|---|---|---|
| torchvision-resnet50 | 64 | 3,255.09 | 3,257.40 | -0.07% | :white_check_mark: |
| torchvision-resnet50_fp16 | 64 | 6,936.68 | 6,937.63 | -0.01% | :white_check_mark: |
| torchvision-densenet121 | 32 | 2,456.21 | 2,454.67 | 0.06% | :white_check_mark: |
| torchvision-densenet121_fp16 | 32 | 4,222.76 | 4,234.91 | -0.29% | :white_check_mark: |
| torchvision-inceptionv3 | 32 | 1,628.11 | 1,627.74 | 0.02% | :white_check_mark: |
| torchvision-inceptionv3_fp16 | 32 | 2,722.99 | 2,720.93 | 0.08% | :white_check_mark: |
| cadene-inceptionv4 | 16 | 761.07 | 761.38 | -0.04% | :white_check_mark: |
| cadene-resnext64x4 | 16 | 819.22 | 819.11 | 0.01% | :white_check_mark: |
| slim-mobilenet | 64 | 7,476.07 | 7,472.34 | 0.05% | :white_check_mark: |
| slim-nasnetalarge | 64 | 217.11 | 217.12 | -0.00% | :white_check_mark: |
| slim-resnet50v2 | 64 | 3,349.63 | 3,349.86 | -0.01% | :white_check_mark: |
| bert-mrpc-onnx | 8 | 1,150.70 | 1,151.17 | -0.04% | :white_check_mark: |
| bert-mrpc-tf | 1 | 457.38 | 455.79 | 0.35% | :white_check_mark: |
| pytorch-examples-wlang-gru | 1 | 348.08 | 344.28 | 1.10% | :white_check_mark: |
| pytorch-examples-wlang-lstm | 1 | 476.74 | 481.34 | -0.96% | :white_check_mark: |
| torchvision-resnet50_1 | 1 | 813.21 | 814.48 | -0.16% | :white_check_mark: |
| cadene-dpn92_1 | 1 | 431.16 | 431.59 | -0.10% | :white_check_mark: |
| cadene-resnext101_1 | 1 | 393.24 | 393.45 | -0.05% | :white_check_mark: |
| onnx-taau-downsample | 1 | 397.04 | 395.60 | 0.36% | :white_check_mark: |
| dlrm-criteoterabyte | 1 | 32.32 | 32.32 | -0.01% | :white_check_mark: |
| dlrm-criteoterabyte_fp16 | 1 | 51.31 | 51.26 | 0.10% | :white_check_mark: |
| agentmodel | 1 | 10,434.51 | 10,714.81 | -2.62% | :white_check_mark: |
| unet_fp16 | 2 | 59.52 | 59.52 | 0.00% | :white_check_mark: |
| resnet50v1_fp16 | 1 | 1,082.88 | 1,076.03 | 0.64% | :white_check_mark: |
| resnet50v1_int8 | 1 | 1,049.48 | 1,049.50 | -0.00% | :white_check_mark: |
| bert_base_cased_fp16 | 64 | 1,170.88 | 1,170.45 | 0.04% | :white_check_mark: |
| bert_large_uncased_fp16 | 32 | 357.94 | 357.91 | 0.01% | :white_check_mark: |
| bert_large_fp16 | 1 | 200.25 | 200.20 | 0.03% | :white_check_mark: |
| distilgpt2_fp16 | 16 | 2,241.16 | 2,240.98 | 0.01% | :white_check_mark: |
| yolov5s | 1 | 546.06 | 542.66 | 0.63% | :white_check_mark: |
| tinyllama | 1 | 43.90 | 43.87 | 0.07% | :white_check_mark: |
| vicuna-fastchat | 1 | 45.07 | 45.26 | -0.42% | :white_check_mark: |
| whisper-tiny-encoder | 1 | 421.21 | 421.44 | -0.05% | :white_check_mark: |
| whisper-tiny-decoder | 1 | 412.54 | 413.06 | -0.13% | :white_check_mark: |
| llama2_7b | 1 | nan | nan | nan% | :x: |
| qwen1.5-7b | 1 | 23.56 | 23.55 | 0.04% | :white_check_mark: |
| phi3-3.8b | 1 | 26.64 | 26.64 | 0.00% | :white_check_mark: |
| mask-rcnn | 1 | 12.64 | 12.79 | -1.19% | :white_check_mark: |
| llama3-8b | 1 | 21.72 | 21.77 | -0.22% | :white_check_mark: |
| whisper-large-encoder | 1 | 10.22 | 10.22 | -0.04% | :white_check_mark: |
| whisper-large-decoder | 1 | 100.80 | 100.83 | -0.03% | :white_check_mark: |
| mistral-7b | 1 | 23.76 | 23.78 | -0.09% | :white_check_mark: |
| FLUX.1-schnell | 1 | 907.94 | 906.91 | 0.11% | :white_check_mark: |
| nan | nan | nan | nan | nan% | :x: |
This build is not recommended to merge :red_circle:
:x:bert-mrpc-tf: ERROR - check error output
2025-05-14 10:54:54.481307: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1747238099.932896 163328 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 62973 MB memory: -> device: 0, name: AMD Instinct MI250X/MI250, pci bus id: 0000:b3:00.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1747238100.779038 163328 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2025-05-14 10:55:10.508023: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-14 10:55:10.508078: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-14 10:55:10.508295: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-14 10:55:10.508345: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-14 10:55:10.508392: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-14 10:55:10.508420: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-14 10:55:10.508464: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-05-14 10:55:10.508516: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
2025-05-14 10:55:10.509511: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-05-14 10:55:10.510689: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-05-14 10:55:10.510709: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
2025-05-14 10:55:10.510720: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
2025-05-14 10:55:10.510735: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 11217777527359497193
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1407, in _do_call
return fn(*args)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1390, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1483, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 324, in main
y_out = sess.run(y, feed_dict=tf_dict)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 977, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1220, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1400, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1426, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'::red_circle:unet: FAILED: MIGraphX is not within tolerance - check verbose output
:red_circle:bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output
:x:llama2_7b: ERROR - check error output
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:265: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/llama2_7b/decoder_model.onnx:x:qwen1.5-7b: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256:x:phi3-3.8b: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256:x:mask-rcnn: ERROR - check error output
usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: 3 800 800:x:#whisper-large-encoder: ERROR - check error output
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/include/migraphx/op/convolution.hpp:100: normalize_compute_shape: CONVOLUTION: mismatched channel numbers