panamax icon indicating copy to clipboard operation
panamax copied to clipboard

Use S3 APIs to discover rustup-init architectures

Open k3d3 opened this issue 3 years ago • 1 comments

On Ubuntu, there's a package you can install called awscli that allows you to access many different APIs, including that of S3.

If you run

aws s3 ls --no-sign-request s3://static-rust-lang-org/rustup/dist/

Then you will receive a full list of rustup architectures, i.e.:

                           PRE aarch64-apple-darwin/
                           PRE aarch64-linux-android/
                           PRE aarch64-unknown-linux-gnu/
                           PRE aarch64-unknown-linux-musl/
                           PRE arm-linux-androideabi/
                           PRE arm-unknown-linux-gnueabi/
                           PRE arm-unknown-linux-gnueabihf/
                           PRE armv7-linux-androideabi/
                           PRE armv7-unknown-linux-gnueabihf/
                           PRE i686-apple-darwin/
                           PRE i686-linux-android/
                           PRE i686-pc-windows-gnu/
                           PRE i686-pc-windows-msvc/
                           PRE i686-unknown-linux-gnu/
                           PRE mips-unknown-linux-gnu/
                           PRE mips64-unknown-linux-gnuabi64/
                           PRE mips64el-unknown-linux-gnuabi64/
                           PRE mipsel-unknown-linux-gnu/
                           PRE powerpc-unknown-linux-gnu/
                           PRE powerpc64-unknown-linux-gnu/
                           PRE powerpc64le-unknown-linux-gnu/
                           PRE riscv64gc-unknown-linux-gnu/
                           PRE s390x-unknown-linux-gnu/
                           PRE x86_64-apple-darwin/
                           PRE x86_64-linux-android/
                           PRE x86_64-pc-windows-gnu/
                           PRE x86_64-pc-windows-msvc/
                           PRE x86_64-unknown-freebsd/
                           PRE x86_64-unknown-illumos/
                           PRE x86_64-unknown-linux-gnu/
                           PRE x86_64-unknown-linux-musl/
                           PRE x86_64-unknown-netbsd/

If we could perform a similar call when syncing rustup, we could use this list to either collect every single rustup-init, or use it to validate any filters that may be set in mirror.toml. Ideally, this would replace the hard-coded PLATFORMS and PLATFORMS_EXE lists that currently exist in src/rustup.rs.


The only unresolved question I have with regards to this, is what should be done if Panamax is grabbing from another Rust mirror, as opposed to the official repo. We probably can't assume said mirror would be placed on S3 or something with an S3-compatible API, so I think the best option is to not validate, and accept any architecture if rustup's source in mirror.toml is not set to https://static.rust-lang.org.

k3d3 avatar Apr 21 '21 14:04 k3d3

I've added a list-platforms command to Panamax, which will at least list out the available platforms. Additionally, changes have been made to allow for platforms not in that list.

k3d3 avatar Apr 02 '22 19:04 k3d3

Done. By default (if a list is not specified in mirror.toml), all architectures found on the rust-lang servers will be downloaded, rather than the hardcoded list previously in the Panamax source code.

k3d3 avatar Feb 04 '23 02:02 k3d3