AMDMIGraphX icon indicating copy to clipboard operation
AMDMIGraphX copied to clipboard

Bump ci rocm6.2

Open TedThemistokleous opened this issue 1 year ago • 4 comments

Updating CI to ROCm 6.2 and other appropriate dockers

TedThemistokleous avatar Aug 02 '24 22:08 TedThemistokleous

Still waiting info for torch sha-1 used dont see it on repo.radeaon under the usual manylinux link. Will let this run over weekend in the meantime.

TedThemistokleous avatar Aug 02 '24 22:08 TedThemistokleous

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.04%. Comparing base (7c2fdf5) to head (0bb51fc). Report is 174 commits behind head on develop.

Additional details and impacted files
@@            Coverage Diff            @@
##           develop    #3340    +/-   ##
=========================================
  Coverage    92.04%   92.04%            
=========================================
  Files          505      506     +1     
  Lines        20699    20872   +173     
=========================================
+ Hits         19052    19212   +160     
- Misses        1647     1660    +13     
Flag Coverage Δ
92.04% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Aug 02 '24 22:08 codecov[bot]

Looks good besides the CI failures.

ahsan-ca avatar Aug 03 '24 02:08 ahsan-ca

Looks good besides the CI failures.

Yeah. something I'll have to deal with I think. @pfultz, should I be including some of your changes to the CPP checks from the 6.1.3 CI update PR in CMakeLists.txt https://github.com/ROCm/AMDMIGraphX/pull/3335/commits/9291c014b94981b37e1a2241ff4d0630d0af9342

TedThemistokleous avatar Aug 03 '24 03:08 TedThemistokleous

Yeah. something I'll have to deal with I think. @pfultz, should I be including some of your changes to the CPP checks from the 6.1.3 CI update PR in CMakeLists.txt

There are lots of changes needed from there, but it seems like hip-config.cmake is broken in 6.2, so we probably need to do some patching of the files.

pfultz2 avatar Aug 05 '24 14:08 pfultz2

Confirmed items are upstream on repo.radeon finally for the Onnxruntime wheel

TedThemistokleous avatar Aug 08 '24 18:08 TedThemistokleous

Rebased off #3335

TedThemistokleous avatar Aug 13 '24 14:08 TedThemistokleous

Looks like somehow we're failing on an env variable due to something not setting hip_HIPCC_EXECUTABLE for this. Tryin to see if I modify the hip-clang docker if this will resolve the failures in tidy. I also updated the files the license check was complaining about. I'll check back in the morning

TedThemistokleous avatar Aug 22 '24 03:08 TedThemistokleous

Odd, micimaster all passed but I dont see this reflected in PR. Looks like its just tidy/cpp check warnings/errors now?

TedThemistokleous avatar Sep 09 '24 18:09 TedThemistokleous

The pass is from 20 days ago. I'll rekick. If you update the PR with a new commit it clears the Jenkins run information

causten avatar Sep 11 '24 17:09 causten

Got past the hipcc errors and the hip-config by modifying hip-clang.cmake usind sed to set the HIP_PLATFORM to "Amd"

Now hitting this

-- Generating embedding library 'migraphx_kernels'
-- Performing Test HAS_HIP_LAMBDA_HOST_DEVICE
-- Performing Test HAS_HIP_LAMBDA_HOST_DEVICE - Failed
-- Build with rocMLIR::rockCompiler 2.0.0
-- MIGraphX is using HIP Clang
CMake Error at cmake/TargetFlags.cmake:42 (get_target_property):
  get_target_property() called with non-existent target "hip::device".
Call Stack (most recent call first):
  cmake/TargetFlags.cmake:61 (get_target_property2)
  cmake/TargetFlags.cmake:100 (append_flags)
  src/targets/gpu/CMakeLists.txt:250 (target_flags)


-- Hip compiler flags: " "
-- Looking for miopenHiddenSetConvolutionFindMode in MIOpen
CMake Error at /data/build/CMakeFiles/CMakeTmp/CMakeLists.txt:20 (add_executable):
  Target "cmTC_cab74" links to target "hip::host" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?


CMake Error at /usr/share/cmake-3.22/Modules/CheckLibraryExists.cmake:72 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  src/targets/gpu/CMakeLists.txt:291 (check_library_exists)

TedThemistokleous avatar Sep 19 '24 02:09 TedThemistokleous

Before we close this out we should make issues for the following tidy warnings that I just stubbed out

  • [ ] bug-prone-empty-catch
  • [ ] performance-enum-size

The other readability ones may not need issues

  • [ ] readability-redundant-member-init
  • [ ] readability-avoid-nested-conditional-operator

TedThemistokleous avatar Sep 19 '24 12:09 TedThemistokleous

Test Batch Rate new
0bb51f
Rate old
32a84a
Diff Compare
torchvision-resnet50 64 3,251.39 3,250.93 0.01% :white_check_mark:
torchvision-resnet50_fp16 64 6,981.90 6,986.29 -0.06% :white_check_mark:
torchvision-densenet121 32 2,433.85 2,433.76 0.00% :white_check_mark:
torchvision-densenet121_fp16 32 4,111.85 4,110.78 0.03% :white_check_mark:
torchvision-inceptionv3 32 1,635.90 1,635.58 0.02% :white_check_mark:
torchvision-inceptionv3_fp16 32 2,738.41 2,737.29 0.04% :white_check_mark:
cadene-inceptionv4 16 776.74 775.61 0.15% :white_check_mark:
cadene-resnext64x4 16 808.50 808.58 -0.01% :white_check_mark:
slim-mobilenet 64 7,453.48 7,452.87 0.01% :white_check_mark:
slim-nasnetalarge 64 208.35 208.18 0.09% :white_check_mark:
slim-resnet50v2 64 3,434.90 3,434.69 0.01% :white_check_mark:
bert-mrpc-onnx 8 1,150.56 1,150.44 0.01% :white_check_mark:
bert-mrpc-tf 1 310.83 311.37 -0.17% :white_check_mark:
pytorch-examples-wlang-gru 1 428.30 426.16 0.50% :white_check_mark:
pytorch-examples-wlang-lstm 1 390.88 386.06 1.25% :white_check_mark:
torchvision-resnet50_1 1 773.02 816.95 -5.38% :red_circle:
cadene-dpn92_1 1 397.30 401.62 -1.08% :white_check_mark:
cadene-resnext101_1 1 381.22 380.63 0.16% :white_check_mark:
onnx-taau-downsample 1 344.52 345.07 -0.16% :white_check_mark:
dlrm-criteoterabyte 1 35.06 35.06 -0.00% :white_check_mark:
dlrm-criteoterabyte_fp16 1 58.05 58.01 0.07% :white_check_mark:
agentmodel 1 8,046.15 7,975.82 0.88% :white_check_mark:
unet_fp16 2 58.03 58.07 -0.07% :white_check_mark:
resnet50v1_fp16 1 934.55 921.74 1.39% :white_check_mark:
resnet50v1_int8 1 935.51 953.21 -1.86% :white_check_mark:
bert_base_cased_fp16 64 1,152.88 1,152.80 0.01% :white_check_mark:
bert_large_uncased_fp16 32 355.81 355.84 -0.01% :white_check_mark:
bert_large_fp16 1 211.62 211.95 -0.16% :white_check_mark:
distilgpt2_fp16 16 2,162.07 2,157.57 0.21% :white_check_mark:
yolov5s 1 536.73 528.76 1.51% :white_check_mark:
tinyllama 1 43.42 43.37 0.11% :white_check_mark:
vicuna-fastchat 1 171.25 168.85 1.42% :white_check_mark:
whisper-tiny-encoder 1 414.85 418.42 -0.85% :white_check_mark:
whisper-tiny-decoder 1 428.64 425.66 0.70% :white_check_mark:

This build is not recommended to merge :red_circle:

migraphx-bot avatar Sep 21 '24 00:09 migraphx-bot


     :white_check_mark: bert-mrpc-onnx: PASSED: MIGraphX meets tolerance
     :white_check_mark: bert-mrpc-tf: PASSED: MIGraphX meets tolerance
     :white_check_mark: pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance
     :white_check_mark: pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance
     :white_check_mark: torchvision-resnet50_1: PASSED: MIGraphX meets tolerance
     :white_check_mark: cadene-dpn92_1: PASSED: MIGraphX meets tolerance
     :white_check_mark: cadene-resnext101_1: PASSED: MIGraphX meets tolerance
     :white_check_mark: dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance
     :white_check_mark: agentmodel: PASSED: MIGraphX meets tolerance
     :white_check_mark: unet: PASSED: MIGraphX meets tolerance
     :white_check_mark: resnet50v1: PASSED: MIGraphX meets tolerance
     :white_check_mark: bert_base_cased_fp16: PASSED: MIGraphX meets tolerance
:red_circle:bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

     :white_check_mark: bert_large: PASSED: MIGraphX meets tolerance
     :white_check_mark: yolov5s: PASSED: MIGraphX meets tolerance
     :white_check_mark: tinyllama: PASSED: MIGraphX meets tolerance
     :white_check_mark: vicuna-fastchat: PASSED: MIGraphX meets tolerance
     :white_check_mark: whisper-tiny-encoder: PASSED: MIGraphX meets tolerance
     :white_check_mark: whisper-tiny-decoder: PASSED: MIGraphX meets tolerance
     :white_check_mark: distilgpt2_fp16: PASSED: MIGraphX meets tolerance

migraphx-bot avatar Sep 21 '24 00:09 migraphx-bot