Detect libstd dependencies and licenses

[repi]

Rust's libstd is also licensed under MIT OR Apache-2.0 and includes transitive dependencies that we need to scan and process the licenses for all of them to include in the attribution output.

https://github.com/rust-lang/rust/blob/master/src/libstd/Cargo.toml

Related:

• https://github.com/rust-lang/rust/issues/67014
• https://github.com/MaulingMonkey/lies/issues/1

[MaulingMonkey]

Scanning std as an end-user of cargo about is annoying - unless you're building std yourself (via cargo-xbuild, xargo, cargo -Z build-std, etc.) you may not have the Cargo.toml of std dependencies available, even with rust-src, which is necessary to detect license information for in a typical rust build. The crate graph may also be insufficient - rust-lang/rust/COPYRIGHT mentions all kinds of fun niche extra terms... and has also suffered plenty of bit rot.

This file is distributed by rustup in what appears to be an attempt at complying with the license:

• %USERPROFILE%\.rustup\toolchains\1.6.0-x86_64-pc-windows-msvc\share\doc\rust\{COPYRIGHT, LICENSE-APACHE, LICENSE-MIT}
• %USERPROFILE%\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\share\doc\rust\{COPYRIGHT, LICENSE-APACHE, LICENSE-MIT}
• ${rustup show home}\toolchains\${rustup show active-toolchain}\share\doc\rust\{COPYRIGHT, LICENSE-APACHE, LICENSE-MIT}

I think the most reasonable way forward on this is:

• [ ] rust-lang/rust cleans up COPYRIGHT upstream, and/or creates a separate file (COPYRIGHT-RUNTIME?), possibly extending
  rust-lang/rust/src/tools/tidy/src/deps.rs to autogenerate it. Since rust-lang developers tend to build std, they should have the dependencies available through their normal x.py build process.
• [ ] cargo about finds the above files in ${rustup show home}\toolchains\${rustup show active-toolchain}\share\doc\rust\
• [ ] cargo about perhaps warns about said bit rot / incompleteness of earlier versions of COPYRIGHT, or include_str!s a small database of corrected files for earlier versions of std.

[eddyb]

Btw, the easy way to get toolchain paths is e.g.: $(rustc --print=sysroot)/lib/rustlib/src/rust/library/std. This has the added benefit that it supports distro-installed Rust with no rustup (or $PATH entries that override the rustup wrappers and instead go straight to a custom toolchain etc.).

In terms of rust-src, if it is available, this is the kind of data that can be extracted:

bash -c "cd $(rustc --print=sysroot)/lib/rustlib/src/rust/library/std && cargo metadata --locked --format-version 1" |
  jq ".packages | group_by(.license) | map({key:.[0].license, value:map(.name)}) | from_entries"

{
  "Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT": [
    "wasi",
    "wasi"
  ],
  "MIT OR Apache-2.0": [
    "alloc",
    "cc",
    "core",
    "getrandom",
    "hashbrown",
    "libc",
    "panic_abort",
    "rand",
    "rand_chacha",
    "rand_core",
    "std",
    "std_detect",
    "unwind"
  ],
  "MIT/Apache-2.0": [
    "cfg-if",
    "compiler_builtins",
    "dlmalloc",
    "hermit-abi",
    "ppv-lite86",
    "rand_hc",
    "rustc-demangle",
    "rustc-std-workspace-alloc",
    "rustc-std-workspace-core"
  ],
  "MPL-2.0": [
    "fortanix-sgx-abi"
  ]
}

I assume crates using e.g. cargo_metadata (either directly or via e.g. krates::cm, like cargo-about), can just get this data the same way they query the user's Cargo workspace.

While it might not be accurate enough, it feels better than just ignoring it.

Oh and if the toolchain is from rustup (i.e. dirname $(rustc --print=sysroot) and $(rustup show home)/toolchains are equal), a tool could invoke rustup component add rust-src (perhaps after asking the user for permission?) to ensure it has access to the data.

(I will say, a lot more users likely have rust-src nowadays than used to, thanks to RA needing it, IIRC)

cc @Jake-Shadle (would cargo-deny also be able to use rust-src? I assume denying the MIT and Apache-2.0 licenses should be an error unless your code is #![no_core] and that's perma-unstable for now...)