ARROW-15691: [Dev] Update archery to work with either master or main as default branch

[lafiona]

Overview

The goal of this pull request is to update archery to work with a repository default branch named master or main, as part of the effort to rename the Apache Arrow repository's default branch to main. The parent Jira ticket can be found here.

Implementation

• Update the language of the top level archery, crossbow, and docker command line interface code to reference the mainline development branch (default git branch) generically.
• Update comments that reference the master branch.
• Update the dev/archery/archery/docker/tests files that reference dask and pandas repositories' default branches. Both repositories currently use main as the default branch.
• Update the crossbow benchmarking examples to generically specify the <default-branch> rather than a hard-coded value.
• In .github/workflows/integration.yml, add an environment variable DEFAULT_BRANCH to the archery command in the "Execute Docker Build" step, so that archery can reliably access the default branch value.
• In .github/workflows/archery.yml, add an environment variable DEFAULT_BRANCH for all steps. This environment variable was already used by the Git Fixup step. It will also be used by the Archery Unittests step.
• Add a property, default_branch_name, to the Repo class in dev/archery/archery/crossbow/core.py for computing the default branch name.
  • If specified, the DEFAULT_BRANCH environment variable, takes precedent in determining the default branch name (this is for qualifying in CI).
  • Otherwise, pygit2 is used to get the default branch name via the Apache Arrow repository's origin remote HEAD reference. This is a heuristic, but in most cases, the HEAD reference of the remote points to the default branch.
• Add a cached property, default_branch to the Release class in dev/archery/archery/release/core.py for computing the default branch name. Similar to the default_branch_name property for Repo in archery/archery/crossbow/core.py:
  • If specified, the DEFAULT_BRANCH environment variable, takes precedent in determining the default branch name (this is for qualifying in CI).
  • Otherwise, similar to the previous step,GitPython is used to get the default branch name via the Apache Arrow repository's origin remote HEAD reference.

Out of scope:

• There are remaining instances of master in the test fixtures files in dev/archery/archery/test/fixtures. It appears that the data only refers to external repositories, such as ursa-labs/ursabot, which currently uses master, so these instances were not modified.

Testing

• Ran the archery and crossbow commands in local clones of both the mathworks/arrow and apache/arrow repositories.
• Confirmed that the GitHub CI jobs pass.
• We are unsure how to locally qualify the changes to the release component, but the release tests pass in CI.

Future Directions

1. Added Jira task to update the pull request merge script to work with both master and main (ARROW-17777)

Notes

Thank you @kevingurney for your help with this pull request!

[github-actions[bot]]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions[bot]]

https://issues.apache.org/jira/browse/ARROW-15691

[lafiona]

This change is ready for review and ready to be considered for running the CI workflows that are awaiting approval. Thank you for your help on this!

[pitrou]

Also @raulcd FYI

[lafiona]

Thanks @pitrou for your code review, I've addressed your feedback in the latest commits.

[pitrou]

@github-actions crossbow submit example-cpp-minimal-build-static

[github-actions[bot]]

Revision: 374f5401f062d0a0247154e6833baf309cee9947

Submitted crossbow builds: ursacomputing/crossbow @ actions-05d3f08a75

Task	Status
example-cpp-minimal-build-static

[pitrou]

Thank you @lafiona !

[ursabot]

Benchmark runs are scheduled for baseline = 21564cf3981a1da0662ccd495320a5127b693a1f and contender = 8861c0c8b2ac8193c5112e7b7c8ab0c2ea33a6ff. 8861c0c8b2ac8193c5112e7b7c8ab0c2ea33a6ff is a master commit associated with this PR. Results will be available as each benchmark for each run completes. Conbench compare runs links: [Finished :arrow_down:0.0% :arrow_up:0.0%] ec2-t3-xlarge-us-east-2 [Failed :arrow_down:0.0% :arrow_up:0.0%] test-mac-arm [Finished :arrow_down:0.0% :arrow_up:0.0%] ursa-i9-9960x [Finished :arrow_down:0.36% :arrow_up:0.0%] ursa-thinkcentre-m75q Buildkite builds: [Finished] 8861c0c8 ec2-t3-xlarge-us-east-2 [Failed] 8861c0c8 test-mac-arm [Finished] 8861c0c8 ursa-i9-9960x [Finished] 8861c0c8 ursa-thinkcentre-m75q [Finished] 21564cf3 ec2-t3-xlarge-us-east-2 [Failed] 21564cf3 test-mac-arm [Finished] 21564cf3 ursa-i9-9960x [Finished] 21564cf3 ursa-thinkcentre-m75q Supported benchmarks: ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True test-mac-arm: Supported benchmark langs: C++, Python, R ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java