ROCmSoftwarePlatform
Optimize docker image building in Jenkinsfile
Notes
- Restarting from a stage is impossible, since we have parallel stages, not sequential ones. Alternatively, we can make a RUN_STAGE_NAME parameter for each stage so that unnecessary stages can be turned off
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 92.02%. Comparing base (
9fb2f4e) to head (8ffe631). Report is 150 commits behind head on develop.
Additional details and impacted files
@@ Coverage Diff @@
## develop #2351 +/- ##
========================================
Coverage 92.02% 92.02%
========================================
Files 509 509
Lines 21005 21005
========================================
Hits 19330 19330
Misses 1675 1675
| Flag | Coverage Δ | |
|---|---|---|
92.02% <ø> (ø) |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
This PR stops docker images from being built on each server. This will reduce active time on each server since it eliminates the docker build step per server. It will therefore eliminate a certain set of network failures where random jobs died due to network failures.
This does introduce a problem were there will be an ever growing list of migraphx docker images on each test server. Additional work will be needed to remove older images on a schedule.
What I would like to get to is skipping the tests that have already passed at the particular commit hash. Skipping previously passed tests would significantly reduce overall server time
| Test | Batch | Rate new 8ffe63 |
Rate old 1ab830 |
Diff | Compare |
|---|---|---|---|---|---|
| torchvision-resnet50 | 64 | 3,254.86 | 3,258.24 | -0.10% | :white_check_mark: |
| torchvision-resnet50_fp16 | 64 | 6,962.02 | 6,986.28 | -0.35% | :white_check_mark: |
| torchvision-densenet121 | 32 | 2,433.39 | 2,435.48 | -0.09% | :white_check_mark: |
| torchvision-densenet121_fp16 | 32 | 4,050.45 | 4,094.65 | -1.08% | :white_check_mark: |
| torchvision-inceptionv3 | 32 | 1,638.02 | 1,636.30 | 0.10% | :white_check_mark: |
| torchvision-inceptionv3_fp16 | 32 | 2,755.34 | 2,745.30 | 0.37% | :white_check_mark: |
| cadene-inceptionv4 | 16 | 779.62 | 779.12 | 0.06% | :white_check_mark: |
| cadene-resnext64x4 | 16 | 807.61 | 808.04 | -0.05% | :white_check_mark: |
| slim-mobilenet | 64 | 7,469.09 | 7,462.28 | 0.09% | :white_check_mark: |
| slim-nasnetalarge | 64 | 208.48 | 208.07 | 0.19% | :white_check_mark: |
| slim-resnet50v2 | 64 | 3,435.45 | 3,438.00 | -0.07% | :white_check_mark: |
| bert-mrpc-onnx | 8 | 1,144.85 | 1,154.53 | -0.84% | :white_check_mark: |
| bert-mrpc-tf | 1 | 324.90 | 320.51 | 1.37% | :white_check_mark: |
| pytorch-examples-wlang-gru | 1 | 412.31 | 391.03 | 5.44% | :high_brightness: |
| pytorch-examples-wlang-lstm | 1 | 382.99 | 383.67 | -0.18% | :white_check_mark: |
| torchvision-resnet50_1 | 1 | 777.89 | 786.92 | -1.15% | :white_check_mark: |
| cadene-dpn92_1 | 1 | 435.23 | 400.89 | 8.57% | :high_brightness: |
| cadene-resnext101_1 | 1 | 379.64 | 382.29 | -0.69% | :white_check_mark: |
| onnx-taau-downsample | 1 | 366.55 | 343.92 | 6.58% | :high_brightness: |
| dlrm-criteoterabyte | 1 | 35.06 | 35.06 | -0.01% | :white_check_mark: |
| dlrm-criteoterabyte_fp16 | 1 | 58.22 | 58.15 | 0.11% | :white_check_mark: |
| agentmodel | 1 | 8,266.03 | 8,244.43 | 0.26% | :white_check_mark: |
| unet_fp16 | 2 | 58.90 | 58.03 | 1.50% | :white_check_mark: |
| resnet50v1_fp16 | 1 | 922.81 | 935.05 | -1.31% | :white_check_mark: |
| resnet50v1_int8 | 1 | 986.75 | 972.22 | 1.49% | :white_check_mark: |
| bert_base_cased_fp16 | 64 | 1,169.78 | 1,172.19 | -0.21% | :white_check_mark: |
| bert_large_uncased_fp16 | 32 | 363.06 | 362.83 | 0.06% | :white_check_mark: |
| bert_large_fp16 | 1 | 211.68 | 211.45 | 0.11% | :white_check_mark: |
| distilgpt2_fp16 | 16 | 2,204.49 | 2,204.90 | -0.02% | :white_check_mark: |
| yolov5s | 1 | 540.19 | 527.91 | 2.33% | :white_check_mark: |
| tinyllama | 1 | 43.37 | 43.44 | -0.15% | :white_check_mark: |
| vicuna-fastchat | 1 | 173.01 | 172.02 | 0.57% | :white_check_mark: |
| whisper-tiny-encoder | 1 | 418.39 | 415.69 | 0.65% | :white_check_mark: |
| whisper-tiny-decoder | 1 | 424.51 | 424.19 | 0.08% | :white_check_mark: |
Check results before merge :high_brightness:
:red_circle:bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output