[CI] Reduce the number of Azure Pipeline jobs

[pitrou]

trafficstars

Describe the enhancement requested

Crossbow execution resources on Azure Pipelines are scarce, while they are quite ample on Github Actions. It would therefore be probably beneficial to move most Azure Pipeline jobs to Github Actions.

Component(s)

Continuous Integration

[pitrou]

@assignUser @raulcd @kou @nealrichardson

[kou]

Here is a list of jobs (and their template files) that use Azure Pipelines:

$ grep -B1 -A1 'ci: azure' dev/tasks/tasks.yml | grep -v 'ci: azure' | grep -v '^--'
  conda-clean:
    template: conda-recipes/azure.clean.yml
  conda-linux-x64-cpu-r41:
    template: conda-recipes/azure.linux.yml
  conda-linux-x64-cpu-r42:
    template: conda-recipes/azure.linux.yml
  conda-linux-x64-cpu-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-x64-cuda-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-aarch64-cpu-r41:
    template: conda-recipes/azure.linux.yml
  conda-linux-aarch64-cpu-r42:
    template: conda-recipes/azure.linux.yml
  conda-linux-aarch64-cpu-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-aarch64-cuda-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-ppc64le-cpu-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-ppc64le-cuda-py3:
    template: conda-recipes/azure.linux.yml
  conda-osx-x64-cpu-r41:
    template: conda-recipes/azure.osx.yml
  conda-osx-x64-cpu-r42:
    template: conda-recipes/azure.osx.yml
  conda-osx-x64-cpu-py3:
    template: conda-recipes/azure.osx.yml
  conda-osx-arm64-cpu-r41:
    template: conda-recipes/azure.osx.yml
  conda-osx-arm64-cpu-r42:
    template: conda-recipes/azure.osx.yml
  conda-osx-arm64-cpu-py3:
    template: conda-recipes/azure.osx.yml
  conda-win-x64-cpu-r41:
    template: conda-recipes/azure.win.yml
  conda-win-x64-cpu-py3:
    template: conda-recipes/azure.win.yml
  conda-win-x64-cuda-py3:
    template: conda-recipes/azure.win.yml
  test-conda-cpp-valgrind:
    template: docker-tests/azure.linux.yml
  test-debian-11-python-3:
    template: docker-tests/azure.linux.yml
  test-ubuntu-20.04-python-3:
    template: docker-tests/azure.linux.yml
  test-fedora-35-python-3:
    template: docker-tests/azure.linux.yml
  test-r-linux-valgrind:
    template: docker-tests/azure.linux.yml
  test-r-depsource-bundled:
    template: r/azure.linux.yml
  test-r-offline-minimal:
    template: r/azure.linux.yml
  test-r-rhub-debian-gcc-release-custom-ccache:
    template: r/azure.linux.yml
  test-r-{{ r_org }}-{{ r_image }}-{{ r_tag }}:
    template: r/azure.linux.yml
  test-r-rhub-debian-gcc-devel-lto-latest:
    template: r/azure.linux.yml
  test-r-rstudio-r-base-4.2-centos7-devtoolset-8:
    template: r/azure.linux.yml
  test-r-minimal-build:
    template: r/azure.linux.yml
  test-ubuntu-r-sanitizer:
    template: docker-tests/azure.linux.yml
  test-fedora-r-clang-sanitizer:
    template: docker-tests/azure.linux.yml
  test-debian-11-go-{{ go_version }}:
    template: docker-tests/azure.linux.yml

Only template files:

$ grep -B1 -A1 'ci: azure' dev/tasks/tasks.yml | grep -v 'ci: azure' | grep -v '^--' | grep template | sort | uniq
    template: conda-recipes/azure.clean.yml
    template: conda-recipes/azure.linux.yml
    template: conda-recipes/azure.osx.yml
    template: conda-recipes/azure.win.yml
    template: docker-tests/azure.linux.yml
    template: r/azure.linux.yml

• conda-recipes/*: We need to rewrite all files to use GitHub Actions instead of Azure Pipelines...
• docker-tests/azure.linux.yml: We can just switch to docker-tests/github.linux.yml and remove docker-tests/azure.linux.yml
• r/azure.linux.yml: We need to rewrite this to use GitHub Actions instead of Azure Pipelines.

[assignUser]

fyi: I removed the 10 r-arrow conda builds: #39243

[raulcd]

We have removed a lot of azure pipeline jobs, we currently only have some R nightly jobs on Azure, should we close this issue? We can open a new one if we want to migrate the R jobs from Azure, or to revisit the current R CI jobs @pitrou thoughts?

[pitrou]

Yes, I think we can probably close now. Thanks for doing this!