Azure test sometimes fails when downloading azure-cli

[jrusz]

Failed job: https://gitlab.com/osbuild/ci/image-builder/-/jobs/1354688824

Installing:
 azure-cli         x86_64         2.25.0-1.el7          azure-cli          43 M
Transaction Summary
================================================================================
Install  1 Package
Total download size: 43 M
Installed size: 557 M
Downloading Packages:
[MIRROR] azure-cli-2.25.0-1.el7.x86_64.rpm: Curl error (35): SSL connect error for https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.25.0-1.el7.x86_64.rpm [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to packages.microsoft.com:443 ]
[MIRROR] azure-cli-2.25.0-1.el7.x86_64.rpm: Curl error (28): Timeout was reached for https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.25.0-1.el7.x86_64.rpm [Connection timed out after 30001 milliseconds]
[MIRROR] azure-cli-2.25.0-1.el7.x86_64.rpm: Curl error (28): Timeout was reached for https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.25.0-1.el7.x86_64.rpm [Connection timed out after 30001 milliseconds]
[MIRROR] azure-cli-2.25.0-1.el7.x86_64.rpm: Curl error (35): SSL connect error for https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.25.0-1.el7.x86_64.rpm [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to packages.microsoft.com:443 ]
[FAILED] azure-cli-2.25.0-1.el7.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Failed to set locale, defaulting to C.UTF-8
Error: Error downloading packages:
  Cannot download azure-cli-2.25.0-1.el7.x86_64.rpm: All mirrors were tried

I've seen this several times, when trying manually or in osbuild-composer repository and I think it's the same issue. The way it's being installed is provided in official documentation so that should be fine. Usually a restart fixes it, so I guess adding some retry before failing the test could solve this easily...

[jrusz]

Update: restarted job failed with a timeout https://gitlab.com/osbuild/ci/image-builder/-/jobs/1355122426

Might be a bigger issue then...

[jrusz]

I found logs from another job in osbuild-composer repo, it was trying to get azure-cli for an hour before giving up... link: https://gitlab.com/osbuild/ci/osbuild-composer/-/jobs/1355193492

+ sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
+ echo -e '[azure-cli]
name=Azure CLI
baseurl=https://packages.microsoft.com/yumrepos/azure-cli
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc'
+ sudo tee /etc/yum.repos.d/azure-cli.repo
[azure-cli]
name=Azure CLI
baseurl=https://packages.microsoft.com/yumrepos/azure-cli
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc
+ sudo dnf install -y azure-cli
Azure CLI                                       3.8 kB/s |  14 MB     62:57    
Errors during downloading metadata for repository 'azure-cli':
  - Curl error (28): Timeout was reached for https://packages.microsoft.com/yumrepos/azure-cli/repodata/repomd.xml [Operation timed out after 30000 milliseconds with 0 out of 0 bytes received]
  - Curl error (28): Timeout was reached for https://packages.microsoft.com/yumrepos/azure-cli/repodata/daf6c96bd103b41c9dc9d1b2bd99a4de2554ff0803997186991b38388ea7a7df-filelists.xml.gz [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
Error: Failed to download metadata for repo 'azure-cli': Yum repo downloading error: Downloading error(s): repodata/daf6c96bd103b41c9dc9d1b2bd99a4de2554ff0803997186991b38388ea7a7df-filelists.xml.gz - Cannot download, all mirrors were already tried without success
Ignoring repositories: azure-cli
Last metadata expiration check: 0:03:03 ago on Thu Jun 17 12:25:02 2021.
No match for argument: azure-cli
Error: Unable to find a match: azure-cli

There are more azure jobs that failed on this step in the pipeline and this was around the same time this issue first appeared.

[jrusz]

I'm wondering how to best solve this issue, would introducing retry logic be enough? I think there was a conversation in the past, that making our own mirror for the azure-cli repo might be a good solution, would probably even increase the speed of download because it's terrible from the official repo. This applies for tests in all repositories that use azure-cli...

[ondrejbudai]

I guess we can snapshot it? https://www.osbuild.org/rpmrepo/