Improve getDependency workflow

[llxia]

Currently, the pipeline downloads 3rd party jars twice.

• The first download (all jars): The pipeline downloads from the Jenkins job (pre-stage jars). https://ci.adoptium.net/job/test.getDependency/ https://openj9-jenkins.osuosl.org/job/test.getDependency/ It tries to download all 3rd party jars. Because we are at the Jenkins level, we do not know what we need.

• The second download (on demand in build.xml): The pipeline downloads from the 3rd party URL on demand.

Having large libraries (i.e., dacapo), the first download is very likely to fail.

00:03:56.638  downloading https://openj9-jenkins.osuosl.org/job/test.getDependency/lastSuccessfulBuild/artifact//dacapo.zip
00:04:57.358  ERROR: downloading https://openj9-jenkins.osuosl.org/job/test.getDependency/lastSuccessfulBuild/artifact//dacapo.zip failed, return code: 4608
00:04:57.358    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
00:04:57.358                                   Dload  Upload   Total   Spent    Left  Speed
00:04:57.358  
00:04:57.358    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
00:04:57.358    1 1368M    1 21.8M    0     0  27.2M      0  0:00:50 --:--:--  0:00:50 27.2M
...
00:04:57.358   75 1368M   75 1035M    0     0  19.4M      0  0:01:10  0:00:53  0:00:17 24.1M
00:04:57.358  curl: (18) transfer closed with 348610934 bytes remaining to read

As a result, not all jars are downloaded and we rely on the second download. However, the second download may not be stable due to 3rd party URL. See https://github.com/adoptium/aqa-tests/issues/6460

The getDependency workflow should be improved to download the required jars only. In summary, we do not download all jars at the Jenkins level. We only download on demand in build.xml. In the first try, download from the test.getDependency job. If the first one fails, use 3rd party URL in the second try.

[llxia]

@Amrutha-Kanhirathingal could you please work on this one? Thanks

[llxia]

Steps:

• Update https://github.com/adoptium/TKG/blob/master/scripts/getDependencies.pl to run twice. Once with customUrl. If it failed, retry with 3rd party URL.

• Only trigger getDependencies.pl when env.BUILD_LIST == 'system' https://github.com/adoptium/aqa-tests/blob/eecf48310a2d4b23332c6a15c76b748f210eaccb/buildenv/jenkins/JenkinsfileBase#L223

• Clean lib LIB_DIR on one machine (say, machineA) (ensure externalDependency/lib is removed on machineA)

• Rerun with the above PR on machineA and check for failures

  • sanity.functional
  • extended.functional
  • special.functional
  • sanity.perf
  • extended.perf