Support testing non-native targets

[kkysen]

Fixes https://github.com/immunant/c2rust/issues/410.

Current Description

Added support for cross-compiling and emulation using qemu. The targets to install are determined in docker_build.sh from the target-tuple files and passed to the Dockerfile as an argument. There it installs the gcc-${c_target} apt packages for cross-compiling in provision_deb.sh (as well as qemu) and the rust targets in provision_rust.sh using rustup. Then it modifies test_translator.py to generate a .cargo/config.toml for every cross-compiling test. This sets the correct linker, rustflags, and runner (qemu-${arch} -L /usr/${c_target}, which means we don't need to mess with binfmt_misc either). It also sets the default target, so we don't need to always pass --target ${target} everywhere.

The existing arm tests, asm.arm and asm.aarch64, are currently failing, so they are temporarily fixed or skipped until we figure out how to actually fix them so that we don't cause CI failures. See https://github.com/immunant/c2rust/issues/431, https://github.com/immunant/c2rust/issues/430, and https://github.com/immunant/c2rust/issues/433 (now fixed by https://github.com/immunant/c2rust/pull/441) for the issues.

Since this PR also changes CI, the current CI tests that are passing don't really mean anything. I've tested the changes locally on ubuntu:focal and they pass. Once we're confident it won't break CI at all, these changes should be merged into feature/ci-dev and/or have ./scripts/docker_build.sh push-all be run.

Initial Description

Added support to scripts/test_translator.py for cross-compiling using cargo zigbuild and zig cc and for emulation using qemu.

More specifically, cargo zigbuild is used in place of cargo for building and running the test crates. It uses zig cc as the linker, which has seamless cross-compilation that just works.

qemu is used to emulate other targets and architectures. These are small test programs, so this should be fine, and doing it this way allows all tests to be run locally, which is very convenient.

I wasn't sure how dependencies should be managed, but for this, I just had them installed in test_translator.py itself, assuming it's running Ubuntu Linux. I assume there's a better way to do this that's more cross-platform, but I wasn't sure how to do that yet, so I just did it inline first to make things quick. We can fix that next. As is, it uses sudo and isn't going to work in CI, but it probably will work locally.

Specifically for dependencies,

• cargo-zigbuild is installed using cargo install
• zig cc is installed through pip install ziglang (added it to requirements.txt)
• rustc targets are installed through rustup target add
• qemu is installed through apt (qemu-user, qemu-user-static)
• qemu dynamic libraries are installed through apt (gcc-${triple})
• update-binfmts (for enabling qemu binfmt_misc support) is installed through apt (binfmt-support)
• qemu binfmt_misc entries are added through update-binfmts --enable

Note, however, that the asm.arm and asm.aarch64 tests, the ones I was trying to test, don't appear to work out of the box. I'm not sure if the errors in them are due to something wrong in the cross compilation and emulation, are they are simply errors in the tests themselves since I assume they weren't being tested as regularly. Specifically, asm.arm fails at runtime on an assertion (https://github.com/immunant/c2rust/issues/431), and asm.aarch64 fails to compile, saying an asm argument is unused (https://github.com/immunant/c2rust/issues/430).

[kkysen]

I moved the installation stuff to scripts/provision_cross_tests.sh, which I only run in scripts/provision_deb.sh now. If installs all the same stuff as before, but since it's not installed at use-site, it installs necessary things for all targets found in target-tuple files.

scripts/test_translator.py doesn't install anything anymore. If cargo-zigbuild is available, it uses that (which uses zig cc), otherwise just normal cargo. And it goes back to the logic of only running tests for targets if those (rustup) targets are already installed.

We can keep discussing whether to use zig cc or not, but I wanted to add these changes first, which are simpler, and see how it works in CI.

[kkysen]

Removed any dependencies on zig cc and cargo-zigbuild now. It now just uses the gcc-${c_target} apt packages for cross-compiling.

It does this by generating a .cargo/config.toml for every cross-compiling test. This sets the correct linker, rustflags, and runner (qemu-${arch}, which means we don't need to mess with binfmt_misc either). It also sets the default target, so we don't need to always pass --target ${target} everywhere.

provision_cross_tests.sh is also now split into just additions to provision_deb.sh and provision_rust.sh.

Also, the arm tests that are failing are temporarily fixed/skipped until we figure out how to actually fix them so that we don't cause CI failures. See https://github.com/immunant/c2rust/issues/431, https://github.com/immunant/c2rust/issues/430, and https://github.com/immunant/c2rust/issues/433.

Speaking of CI, these changes, once approved (@thedataking), need to also be pushed to feature/ci-dev and have ./scripts/docker_build.sh push-all be run.

[kkysen]

https://github.com/immunant/c2rust/issues/433 is now fixed by https://github.com/immunant/c2rust/pull/441, so that's down to 1/2 issues blocking the tests/asm.aarch64 test from working.