AMDMIGraphX Remove BF16 support for Navi21

https://ontrack-internal.amd.com/browse/SWDEV-518075?filter=-1

Apr 09 '25 23:04 richagadgil

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3943   +/-   ##
========================================
  Coverage    92.18%   92.18%           
========================================
  Files          528      528           
  Lines        24307    24307           
========================================
  Hits         22405    22405           
  Misses        1902     1902

:rocket: New features to boost your workflow:

:snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Apr 10 '25 01:04 codecov[bot]

Test Batch Rate new
496987 Rate old
0194f5 Diff Compare

torchvision-resnet50 64 3,255.90 3,256.25 -0.01% :white_check_mark:

torchvision-resnet50_fp16 64 6,938.38 6,942.21 -0.06% :white_check_mark:

torchvision-densenet121 32 2,456.63 2,456.08 0.02% :white_check_mark:

torchvision-densenet121_fp16 32 4,232.70 4,229.91 0.07% :white_check_mark:

torchvision-inceptionv3 32 1,630.44 1,626.68 0.23% :white_check_mark:

torchvision-inceptionv3_fp16 32 2,722.48 2,723.03 -0.02% :white_check_mark:

cadene-inceptionv4 16 761.22 761.75 -0.07% :white_check_mark:

cadene-resnext64x4 16 820.30 818.59 0.21% :white_check_mark:

slim-mobilenet 64 7,471.85 7,461.27 0.14% :white_check_mark:

slim-nasnetalarge 64 217.18 217.12 0.03% :white_check_mark:

slim-resnet50v2 64 3,352.79 3,350.96 0.05% :white_check_mark:

bert-mrpc-onnx 8 1,152.86 1,152.86 0.00% :white_check_mark:

bert-mrpc-tf 1 457.49 457.84 -0.08% :white_check_mark:

pytorch-examples-wlang-gru 1 344.13 372.15 -7.53% :red_circle:

pytorch-examples-wlang-lstm 1 476.24 476.18 0.01% :white_check_mark:

torchvision-resnet50_1 1 813.65 811.81 0.23% :white_check_mark:

cadene-dpn92_1 1 430.76 434.68 -0.90% :white_check_mark:

cadene-resnext101_1 1 393.14 393.04 0.03% :white_check_mark:

onnx-taau-downsample 1 395.50 396.12 -0.16% :white_check_mark:

dlrm-criteoterabyte 1 32.32 32.32 0.01% :white_check_mark:

dlrm-criteoterabyte_fp16 1 51.25 51.41 -0.31% :white_check_mark:

agentmodel 1 10,723.53 10,334.54 3.76% :high_brightness:

unet_fp16 2 59.52 59.58 -0.10% :white_check_mark:

resnet50v1_fp16 1 1,079.24 1,080.75 -0.14% :white_check_mark:

resnet50v1_int8 1 1,047.79 1,039.16 0.83% :white_check_mark:

bert_base_cased_fp16 64 1,170.74 1,170.58 0.01% :white_check_mark:

bert_large_uncased_fp16 32 357.94 357.89 0.01% :white_check_mark:

bert_large_fp16 1 200.91 200.03 0.44% :white_check_mark:

distilgpt2_fp16 16 2,238.23 2,242.13 -0.17% :white_check_mark:

yolov5s 1 538.93 538.00 0.17% :white_check_mark:

tinyllama 1 43.89 43.85 0.08% :white_check_mark:

vicuna-fastchat 1 45.11 45.08 0.07% :white_check_mark:

whisper-tiny-encoder 1 420.75 420.83 -0.02% :white_check_mark:

whisper-tiny-decoder 1 411.42 411.54 -0.03% :white_check_mark:

llama2_7b 1 nan nan nan% :x:

qwen1.5-7b 1 23.54 23.55 -0.01% :white_check_mark:

phi3-3.8b 1 nan nan nan% :x:

mask-rcnn 1 12.76 12.71 0.38% :white_check_mark:

llama3-8b 1 21.76 21.77 -0.01% :white_check_mark:

whisper-large-encoder 1 10.22 10.22 -0.02% :white_check_mark:

whisper-large-decoder 1 100.64 100.62 0.02% :white_check_mark:

mistral-7b 1 23.73 23.74 -0.03% :white_check_mark:

FLUX.1-schnell 1 910.90 907.92 0.33% :white_check_mark:

nan nan nan nan nan% :x:

Test	Batch	Rate new 496987	Rate old 0194f5	Diff	Compare
torchvision-resnet50	64	3,255.90	3,256.25	-0.01%	:white_check_mark:
torchvision-resnet50_fp16	64	6,938.38	6,942.21	-0.06%	:white_check_mark:
torchvision-densenet121	32	2,456.63	2,456.08	0.02%	:white_check_mark:
torchvision-densenet121_fp16	32	4,232.70	4,229.91	0.07%	:white_check_mark:
torchvision-inceptionv3	32	1,630.44	1,626.68	0.23%	:white_check_mark:
torchvision-inceptionv3_fp16	32	2,722.48	2,723.03	-0.02%	:white_check_mark:
cadene-inceptionv4	16	761.22	761.75	-0.07%	:white_check_mark:
cadene-resnext64x4	16	820.30	818.59	0.21%	:white_check_mark:
slim-mobilenet	64	7,471.85	7,461.27	0.14%	:white_check_mark:
slim-nasnetalarge	64	217.18	217.12	0.03%	:white_check_mark:
slim-resnet50v2	64	3,352.79	3,350.96	0.05%	:white_check_mark:
bert-mrpc-onnx	8	1,152.86	1,152.86	0.00%	:white_check_mark:
bert-mrpc-tf	1	457.49	457.84	-0.08%	:white_check_mark:
pytorch-examples-wlang-gru	1	344.13	372.15	-7.53%	:red_circle:
pytorch-examples-wlang-lstm	1	476.24	476.18	0.01%	:white_check_mark:
torchvision-resnet50_1	1	813.65	811.81	0.23%	:white_check_mark:
cadene-dpn92_1	1	430.76	434.68	-0.90%	:white_check_mark:
cadene-resnext101_1	1	393.14	393.04	0.03%	:white_check_mark:
onnx-taau-downsample	1	395.50	396.12	-0.16%	:white_check_mark:
dlrm-criteoterabyte	1	32.32	32.32	0.01%	:white_check_mark:
dlrm-criteoterabyte_fp16	1	51.25	51.41	-0.31%	:white_check_mark:
agentmodel	1	10,723.53	10,334.54	3.76%	:high_brightness:
unet_fp16	2	59.52	59.58	-0.10%	:white_check_mark:
resnet50v1_fp16	1	1,079.24	1,080.75	-0.14%	:white_check_mark:
resnet50v1_int8	1	1,047.79	1,039.16	0.83%	:white_check_mark:
bert_base_cased_fp16	64	1,170.74	1,170.58	0.01%	:white_check_mark:
bert_large_uncased_fp16	32	357.94	357.89	0.01%	:white_check_mark:
bert_large_fp16	1	200.91	200.03	0.44%	:white_check_mark:
distilgpt2_fp16	16	2,238.23	2,242.13	-0.17%	:white_check_mark:
yolov5s	1	538.93	538.00	0.17%	:white_check_mark:
tinyllama	1	43.89	43.85	0.08%	:white_check_mark:
vicuna-fastchat	1	45.11	45.08	0.07%	:white_check_mark:
whisper-tiny-encoder	1	420.75	420.83	-0.02%	:white_check_mark:
whisper-tiny-decoder	1	411.42	411.54	-0.03%	:white_check_mark:
llama2_7b	1	nan	nan	nan%	:x:
qwen1.5-7b	1	23.54	23.55	-0.01%	:white_check_mark:
phi3-3.8b	1	nan	nan	nan%	:x:
mask-rcnn	1	12.76	12.71	0.38%	:white_check_mark:
llama3-8b	1	21.76	21.77	-0.01%	:white_check_mark:
whisper-large-encoder	1	10.22	10.22	-0.02%	:white_check_mark:
whisper-large-decoder	1	100.64	100.62	0.02%	:white_check_mark:
mistral-7b	1	23.73	23.74	-0.03%	:white_check_mark:
FLUX.1-schnell	1	910.90	907.92	0.33%	:white_check_mark:
nan	nan	nan	nan	nan%	:x:

This build is not recommended to merge :red_circle:

Apr 26 '25 06:04 migraphx-bot

:white_check_mark: bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

:x:bert-mrpc-tf: ERROR - check error output

2025-04-26 00:47:37.822995: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1745646463.271733 163011 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 62973 MB memory: -> device: 0, name: AMD Instinct MI250X/MI250, pci bus id: 0000:b3:00.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1745646464.112062 163011 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2025-04-26 00:47:52.409067: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-26 00:47:52.409122: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-26 00:47:52.409171: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-26 00:47:52.409217: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-26 00:47:52.409266: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-26 00:47:52.409316: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-26 00:47:52.409364: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-04-26 00:47:52.409395: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
2025-04-26 00:47:52.410411: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-04-26 00:47:52.411562: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-04-26 00:47:52.411581: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
2025-04-26 00:47:52.411591: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
2025-04-26 00:47:52.411606: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 11217777527359497193
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1407, in _do_call
return fn(*args)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1390, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1483, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 324, in main
y_out = sess.run(y, feed_dict=tf_dict)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 977, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1220, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1400, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1426, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:

Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'import/bert/embeddings/LayerNorm/moments/SquaredDifference':

:white_check_mark: pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

:white_check_mark: pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

:white_check_mark: dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

:white_check_mark: agentmodel: PASSED: MIGraphX meets tolerance

:red_circle:unet: FAILED: MIGraphX is not within tolerance - check verbose output

:white_check_mark: resnet50v1: PASSED: MIGraphX meets tolerance

:white_check_mark: bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

:red_circle:bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

:white_check_mark: bert_large: PASSED: MIGraphX meets tolerance

:white_check_mark: yolov5s: PASSED: MIGraphX meets tolerance

:white_check_mark: tinyllama: PASSED: MIGraphX meets tolerance

:white_check_mark: vicuna-fastchat: PASSED: MIGraphX meets tolerance

:white_check_mark: whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

:white_check_mark: whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

:white_check_mark: distilgpt2_fp16: PASSED: MIGraphX meets tolerance

:x:llama2_7b: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:265: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/llama2_7b/decoder_model.onnx

:x:qwen1.5-7b: ERROR - check error output

usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256

:x:phi3-3.8b: ERROR - check error output

usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: input_ids attention_mask position_ids 1 256 @attention_mask 1 256 @position_ids 1 256

:x:mask-rcnn: ERROR - check error output

usage: accuracy_checker.py [-h] [--onnx ONNX] [--tf TF] [--provider PROVIDER]
[--batch BATCH] [--fill1] [--fill0] [--fp16]
[--argmax] [--verbose] [--tolerance TOLERANCE]
[--input-dim INPUT_DIM] [--target TARGET]
[--ort-run] [--ort-logging]
[--disable-offload-copy] [--disable-fast-math]
[--exhaustive_tune]
accuracy_checker.py: error: unrecognized arguments: 3 800 800

:white_check_mark: llama3-8b: PASSED: MIGraphX meets tolerance

:x:#whisper-large-encoder: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/include/migraphx/op/convolution.hpp:100: normalize_compute_shape: CONVOLUTION: mismatched channel numbers

:white_check_mark: whisper-large-decoder: PASSED: MIGraphX meets tolerance

:white_check_mark: mistral-7b: PASSED: MIGraphX meets tolerance

:white_check_mark: FLUX.1-schnell: PASSED: MIGraphX meets tolerance

Apr 26 '25 06:04 migraphx-bot

AMDMIGraphX AMDMIGraphX copied to clipboard

Remove BF16 support for Navi21

Codecov Report

AMDMIGraphX
AMDMIGraphX copied to clipboard