Add a trace flag to check CI intermittant failure

[lakhinderwalia]

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.80%. Comparing base (06eef05) to head (7d19efd).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3068   +/-   ##
========================================
  Coverage    91.80%   91.80%           
========================================
  Files          486      486           
  Lines        18867    18867           
========================================
  Hits         17320    17320           
  Misses        1547     1547           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[migraphx-bot]

Test	Batch	Rate new 7d19ef	Rate old bc6c79	Diff	Compare
torchvision-resnet50	64	2,955.92	2,950.97	0.17%	:white_check_mark:
torchvision-resnet50_fp16	64	6,563.77	6,566.23	-0.04%	:white_check_mark:
torchvision-densenet121	32	2,421.44	2,421.23	0.01%	:white_check_mark:
torchvision-densenet121_fp16	32	3,929.23	3,992.03	-1.57%	:white_check_mark:
torchvision-inceptionv3	32	1,661.22	1,659.26	0.12%	:white_check_mark:
torchvision-inceptionv3_fp16	32	2,595.42	2,598.65	-0.12%	:white_check_mark:
cadene-inceptionv4	16	776.39	776.84	-0.06%	:white_check_mark:
cadene-resnext64x4	16	740.86	740.86	0.00%	:white_check_mark:
slim-mobilenet	64	6,920.51	6,918.90	0.02%	:white_check_mark:
slim-nasnetalarge	64	177.11	177.18	-0.04%	:white_check_mark:
slim-resnet50v2	64	2,876.26	2,876.68	-0.01%	:white_check_mark:
bert-mrpc-onnx	8	1,064.17	1,064.43	-0.02%	:white_check_mark:
bert-mrpc-tf	1	493.52	511.61	-3.54%	:red_circle:
pytorch-examples-wlang-gru	1	368.88	371.38	-0.67%	:white_check_mark:
pytorch-examples-wlang-lstm	1	400.03	409.57	-2.33%	:white_check_mark:
torchvision-resnet50_1	1	778.49	790.15	-1.48%	:white_check_mark:
cadene-dpn92_1	1	440.96	394.00	11.92%	:high_brightness:
cadene-resnext101_1	1	367.35	362.65	1.29%	:white_check_mark:
onnx-taau-downsample	1	349.72	349.17	0.16%	:white_check_mark:
dlrm-criteoterabyte	1	33.44	33.45	-0.05%	:white_check_mark:
dlrm-criteoterabyte_fp16	1	56.75	56.69	0.10%	:white_check_mark:
agentmodel	1	7,395.36	7,460.35	-0.87%	:white_check_mark:
unet_fp16	2	57.26	57.31	-0.10%	:white_check_mark:
resnet50v1_fp16	1	950.55	869.63	9.31%	:high_brightness:
resnet50v1_int8	1	808.41	823.61	-1.85%	:white_check_mark:
bert_base_cased_fp16	64	1,012.08	1,012.93	-0.08%	:white_check_mark:
bert_large_uncased_fp16	32	316.48	316.62	-0.04%	:white_check_mark:
bert_large_fp16	1	nan	nan	nan%	:x:
distilgpt2_fp16	16	1,993.54	1,994.48	-0.05%	:white_check_mark:
yolov5s	1	515.24	501.26	2.79%	:white_check_mark:
tinyllama	1	45.03	44.99	0.08%	:white_check_mark:
vicuna-fastchat	1	174.37	178.18	-2.14%	:white_check_mark:
whisper-tiny-encoder	1	404.38	404.60	-0.06%	:white_check_mark:
whisper-tiny-decoder	1	426.49	424.85	0.39%	:white_check_mark:

This build is not recommended to merge :red_circle:

[migraphx-bot]

:x:bert-mrpc-onnx: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/huggingface-transformers/bert_mrpc1.onnx

     :white_check_mark: bert-mrpc-tf: PASSED: MIGraphX meets tolerance

     :white_check_mark: pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

     :white_check_mark: pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

     :white_check_mark: torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

     :white_check_mark: cadene-dpn92_1: PASSED: MIGraphX meets tolerance

:x:cadene-resnext101_1: ERROR - check error output

2024-05-10 20:37:39.354986278 [W:onnxruntime:, model.cc:183 Model] ONNX Runtime only guarantees support for models stamped with opset version 7 or above for opset domain 'ai.onnx'. Please upgrade your model to opset 7 or higher. For now, this opset 6 model may run depending upon legacy support of some older opset version operators.
2024-05-10 20:37:39.360937777 [W:onnxruntime:, transpose_optimizer.cc:28 ApplyImpl] Transpose optimizer failed: Unsupported ONNX opset: 6
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 267, in main
sess = ort.InferenceSession(model_name,
File "/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 463, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for BatchNormalization(6) node with name ''

     :white_check_mark: dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

     :white_check_mark: agentmodel: PASSED: MIGraphX meets tolerance

:x:unet: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 207, in main
model = migraphx.parse_onnx(model_name,
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/unet/model.onnx

     :white_check_mark: resnet50v1: PASSED: MIGraphX meets tolerance

:red_circle:bert_base_cased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

:red_circle:bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

:x:bert_large: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/bert/model.onnx

     :white_check_mark: yolov5s: PASSED: MIGraphX meets tolerance

     :white_check_mark: tinyllama: PASSED: MIGraphX meets tolerance

     :white_check_mark: vicuna-fastchat: PASSED: MIGraphX meets tolerance

     :white_check_mark: whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

     :white_check_mark: whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

     :white_check_mark: distilgpt2_fp16: PASSED: MIGraphX meets tolerance