cargo
cargo copied to clipboard
Value too large for defined data type when cargo build on armv7 with qemu
Problem
Trying to build an app for armv7 in a docker container with qemu. The current result is explained here with the steps to reproduce it : https://github.com/jdrouet/docker-bug-value-too-long
Steps
- install docker
- install docker-buildx
- follow the steps here
Possible Solution(s) No idea.
Notes The guys from docker claim it's coming from cargo.
Output of cargo version
:
The one from the docker image rust:1-slim-buster
Thanks for the report. I think this is a duplicate of #7451. There is some sort of strange interaction of Linux Large File Support with a 64-bit host and a 32-bit docker image running qemu. There's some more discussion on that issue, but in particular see https://bugs.launchpad.net/qemu/+bug/1805913. I'm not sure there is anything Cargo can do about this. I suspect the underlying problem is that libgit2
uses size-agnostic functions like readdir
instead of explicitly using readdir64
. It may be possible to build libgit2 with -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
(or getconf LFS_CFLAGS
), but I have no idea if that will work or what the consequences would be (it seems like it could have portability issues). This may also just be an issue with qemu.
Hello all,
I have a similar use case i.e. (Qemu userspace emulation for armv7 on x86_64 arch). After chrooting inside the image, i tried building my project repository using that and this was the output.
[root@mnt ~]# cargo build --target=armv7-unknown-linux-gnueabihf
Updating crates.io index
warning: spurious network error (2 tries remaining): could not read directory '/root/.cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
warning: spurious network error (1 tries remaining): could not read directory '/root/.cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
error: failed to fetch `https://github.com/rust-lang/crates.io-index`
Caused by:
could not read directory '/root/.cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
[root@mnt ~]#
However, when fetching it manually using curl https://github.com/rust-lang/crates.io-index
from chroot, this worked exactly as it should have. Because of that, I also tested this on qemu aarch64 image and there it worked perfectly.
As suggested earlier by @ehuss, it seems the issue is still persistent to this date with qemu emulation of armv7. The cargo binaries are definitely not the culprit here, and more likely libgit2
seems to be causing this issue.
Internally, there is some incompatiblity going on, even after following the suggestion for building the library with parameters provided by getconf LFS_CFLAGS
, the issue remains the same.
Lastly, for this quick and simple test case, I would prefer to use command cargo install exa
inside an emulated armv7 image already installled with Rust, for any future testing regarding to its compatiblity.
EDIT: I also proposed a simple solution for anyone that is currenly facing this problem, and until its resolution. #9545
If someone is coming here and also needs a workaround for a docker build:
Put this in the beginning of the Dockerfile:
# syntax = docker/dockerfile:experimental
Then do something like this:
RUN --security=insecure mkdir -p /root/.cargo && chmod 777 /root/.cargo && mount -t tmpfs none /root/.cargo && some_command_that_uses_cargo
If you can't make the above Dockerfile-based workaround, what works as well is using tmpfs for the whole /var/lib/docker
on the build host. For GitHub actions, I've done that here: https://github.com/WeblateOrg/docker/pull/1274
Very nice @nijel! I used your example and made a reusable Action for those affected by this issue :)
If someone is coming here and also needs a workaround for a docker build: Put this in the beginning of the Dockerfile:
# syntax = docker/dockerfile:experimental
Then do something like this:RUN --security=insecure mkdir -p /root/.cargo && chmod 777 /root/.cargo && mount -t tmpfs none /root/.cargo && some_command_that_uses_cargo
If you are using buildx you also need to allow insecure-entitlement
when creating the node:
docker buildx create --driver docker-container --name local --buildkitd-flags '--allow-insecure-entitlement security.insecure' --use
https://github.com/docker/buildx/issues/220#issuecomment-574377254
and allow insecure things in the docker daemon:
# cat /etc/docker/daemon.json
{ "builder": {"Entitlements": {"security-insecure": true }}}
https://github.com/docker/buildx/issues/559#issuecomment-796430825
If you are using buildkit, there is a much cleaner workaround than using the --security=insecure
option. Just use RUN --mount
like this:
RUN --mount=type=tmpfs,target=/root/.cargo some_command_that_uses_cargo
If you are using buildkit, there is a much cleaner workaround than using the
--security=insecure
option. Just useRUN --mount
like this:RUN --mount=type=tmpfs,target=/root/.cargo some_command_that_uses_cargo
For me this worked, but I had to install rust within the same RUN layer:
RUN --mount=type=tmpfs,target=/root/.cargo curl https://sh.rustup.rs -sSf | bash -s -- -y && pip install cryptography==38.0.1
finally, to activate Buildkit, I had to run docker build
as:
DOCKER_BUILDKIT=1 docker build .
If you are using buildkit, there is a much cleaner workaround than using the
--security=insecure
option. Just useRUN --mount
like this:RUN --mount=type=tmpfs,target=/root/.cargo some_command_that_uses_cargo
I had to adapt this slightly to use a different directory for cargo to get it working with the rust base image to build a rust project.
RUN --mount=type=tmpfs,target=/.cargo CARGO_HOME=/.cargo cargo build --release
Doesn't the tmpfs workaround hurt caching a lot? Cargo has to download and build everything every time if CARGO_HOME is not stored on the filesystem.
Notes The guys from docker claim it's coming from cargo.
It's a problem with the virtualization layer. The error is not coming from cargo or docker, but from QEMU running 32-bit platforms on 64-bit hosts (I don't mean it's a bug in QEMU though). An alternative workaround could be to run a 32-bit QEMU process if you have a multilib set up. This would mean using the host kernel's 32-bit API, which does not suffer from the conversion problem.
Since cargo (stable) now supports the sparse registry protocol, the easiest workaround is to set the envinronment variable CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse
. This avoids the problematic call to libgit2 altogether.
You have to make sure the git binary is installed though, but that shouldn't be a problem.
@flo-at
CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse
works very well, many thanks. However, basically the same error happens on rustup self uninstall
and when temporary directories are attempted to be removed during the process:
info: syncing channel updates for '1.73.0-armv7-unknown-linux-gnueabihf'
info: latest update on 2023-10-05, rust version 1.73.0 (cc66ad468 2023-10-03)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: installing component 'cargo'
warning: could not delete temp directory: /root/.rustup/tmp/dq_vgjwah9211lz1_dir
info: installing component 'clippy'
warning: could not delete temp directory: /root/.rustup/tmp/_42_3qtwdsd2gjit_dir
info: installing component 'rust-std'
warning: could not delete temp directory: /root/.rustup/tmp/hg7ojeltqnog7xdr_dir
info: installing component 'rustc'
warning: could not delete temp directory: /root/.rustup/tmp/a10n8s4a7catp38h_dir
info: installing component 'rustfmt'
warning: could not delete temp directory: /root/.rustup/tmp/8fbaomir76h2yb8j_dir
Updating crates.io index
Downloading crates ...
...
# rustup self uninstall -y
info: removing rustup home
error: could not remove 'rustup_home' directory: '/root/.rustup': Value too large for defined data type (os error 75)
Are there similar options for cargo
and rustup
past the installation? Otherwise to me the tmpfs
workaround seems cleaner, to avoid possible surprises.
For anyone having this issue, in the end I went to the path of making a Dockerfile with multi-stage builds that will cross-compile for all the architectures I need on the architecture of the pipeline agent and scrapped using qemu for the cargo build, I only use it to run apt install and other small stuff at the end.
If you project allows for this route, you can follow my Dockerfile as guidance
https://github.com/madiele/vod2pod-rss/blob/cac3d27c415ed8341fa8da738e4604b7427c04ae/Dockerfile
I went form 2+ hours build times to 5-10 minutes build times, it was worth the effort.