Tracker for DevInfra Improvement

[ejguan]

🚀 The feature

This issue is used to track TODOs in my mind for DevInfra:

CI

• [x] Compare released git_version with the commit on the top of branch
• [x] Auto doc gen for RC release
• [x] Add conda upload
• [x] Make official release working on tag
• [x] Add bash script to help build release for both wheel and conda in case changing CI/CD system
• [x] Turn on nightly release
• [x] Turn on nightly doc (currently used main branch )
• [x] Make sure previous nightly releases are removed from conda
• [x] ~Add retry for both wheel and conda upload~ (Will be covered by DevInfra's Nova project)
• [x] ~Add retry for conda nightly to wait until new PyTorch nightly uploaded~ (Will be covered by DevInfra's Nova project)
• [x] ~Make RC release daily on release branch (Safer than manual)~ (Will be covered by DevInfra's Nova project)
• [x] Change torch from test channel not from nightly for CI tests on release branch
• [ ] Add GPU enabled runners (maybe via linux.8xlarge.nvidia.gpu) for CI
• [x] Add M1 enabled runners for releases
  • [ ] Change to use default conda from the runner rather than downloading.
• [x] ~Modify the logic here to determine PyTorch version per platform.~ (Will be covered by DevInfra's Nova project)
• [x] Update conda package to use corresponding pytorch nightly version in here
• [x] Update lint workflow to determine to use nightly pytorch or RC pytorch
• [ ] Add a mechanism to automatically disable a few tests.
• [x] Add python3.11 support
• [x] Remove python 3.7 support https://github.com/pytorch/data/issues/971
• [ ] Replace python setup.py bdist_wheel by pip wheel -w dist .
• [x] #1088
• [ ] Enforce hiding symbols from OpenSSL https://github.com/pytorch/data/issues/972
• [x] Add platform-agnostic releases: https://github.com/pytorch/data/issues/973

Release note:

• [ ] Update release note scripts to support gdoc rather than quip
• [ ] Add contributors' names to the release notes via git shortlog -s v0.6.0..v0.5.1 | cut -f2-

Others

After compilation matrix is enabled:

• [x] Turn on matrix when s3 is landed
• [x] Add artifacts upload and download. The wheels can be sent to a single job and uploaded at one time. (https://github.com/actions/upload-artifact, https://github.com/actions/download-artifact)
• [x] Make build workflow as the dependent job for test/doc_gen using needs
• [x] ~Add cache~
• [x] ~Trigger conda/wheel nightly at different time.~ (Will be covered by DevInfra's Nova project)
  • Conda release requires pytorch nightly being uploaded to conda nightly successfully, which is always impossible for a certain platforms like windows. See https://anaconda.org/pytorch-nightly/torchvision/files.
  • We need to wait for torchdata conda nightly until pytorch nightly releases are uploaded to conda successfully. However, for downstream libraries, they need wheel nightly ASAP for CI testing. Then, we have to trigger these two releases separately.
• [x] Append license from third_party to the license file for sdist_wheel or wheel
• [x] ~Revamp: Support auto-generation of workflows~ (Will be covered by DevInfra's Nova project)

Motivation, pitch

Improvement for releng

Alternatives

No response

Additional context

No response

[NivekT]

"Auto doc gen for RC release" - I believe we can add a GH action for that (i.e. whenever a push is made to the branch release/0.x.0, build a doc in the folder 0.x.0 in the gh-pages branch.

[ejguan]

"Auto doc gen for RC release" - I believe we can add a GH action for that (i.e. whenever a push is made to the branch release/0.x.0, build a doc in the folder 0.x.0 in the gh-pages branch.

I am working on it currently. It should be attached after the build_test_upload workflow for RC release.

[ejguan]

Nightly release is working for wheel. https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html But, it seems the conda token we have used to upload packages to pytorch channel cannot be used to upload them to pytorch-nightly channel. See: https://github.com/pytorch/data/runs/5653050846?check_suite_focus=true

[ejguan]

Another bug for wheel/conda, the nightly version currently relies on date https://github.com/pytorch/data/blob/76733df6a8903bf95b938cf611355f05f5c2f821/packaging/pkg_helpers.bash#L93

But, the date on CI macos machine is not correct. I need to do a quick test.