Crosscompile (for Yocto 2.6 Thud) RedisJSON

[TopperBG]

I want to target compile RedisJSON https://github.com/RedisJSON/RedisJSON

Building machine OS: Ubuntu 18.04.5 LTS

Target MACHINE is var-som-mx6 (NXP i.MX6Q)

Added meta-rust-bin layer in build/conf/bblayers.conf On host OS build pass just fine with cargo build --release against git clone of RedisJSON

That's my Bitbake recipe:

SUMMARY = "Redis JSON"
DESCRIPTION = "Redis JSON plugin."
HOMEPAGE = "http://redis.io"
SECTION = "libs"
LICENSE = "BSD-3-Clause"
LIC_FILES_CHKSUM = "file://${WORKDIR}/git/LICENSE;md5=e75c8f72bbc9c3744174aaf54faf3bd4"

SRC_URI="git://github.com/RedisJSON/RedisJSON.git;protocol=git;"
SRCREV = "2d9ac841a3e6365016c68fe1b1b1f7762277bf6c"
#BB_STRICT_CHECKSUM = "0"
MIRRORS=""

inherit cargo

S = "${WORKDIR}/git"

#do_compile () {
#    cargo build --release
#}

#BBCLASSEXTEND = "native"

Compilation fail with next log:

Log data follows: | DEBUG: Executing shell function do_compile | NOTE: which rustc: /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot-native/usr/bin/rustc | NOTE: rustc --version rustc 1.54.0 (a178d0322 2021-07-26) | NOTE: which cargo: /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot-native/usr/bin/cargo | NOTE: cargo --version cargo 1.54.0 (5ae8d74b3 2021-06-22) | NOTE: cargo build --verbose --manifest-path /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/git/Cargo.toml --target=armv7-unknown-linux-gnueabihf --release | Fresh autocfg v1.0.1 | Fresh unicode-xid v0.2.2 | Fresh version_check v0.9.3 | Fresh glob v0.3.0 | Fresh cfg-if v1.0.0 | Fresh funty v1.1.0 | Fresh wyz v0.2.0 | Fresh tap v1.0.1 | Fresh regex-syntax v0.6.25 | Fresh unicode-width v0.1.8 | Fresh strsim v0.8.0 | Fresh vec_map v0.8.2 | Fresh ansi_term v0.11.0 | Fresh humantime v2.1.0 | Fresh termcolor v1.1.2 | Fresh lazycell v1.3.0 | Fresh hashbrown v0.11.2 | Fresh peeking_take_while v0.1.2 | Fresh lazy_static v1.4.0 | Fresh shlex v1.0.0 | Fresh rustc-hash v1.1.0 | Fresh unicode-xid v0.0.4 | Fresh unicode-segmentation v1.8.0 | Fresh itoa v0.4.8 | Fresh cc v1.0.69 | Fresh ppv-lite86 v0.2.10 | Fresh quote v0.3.15 | Fresh linked-hash-map v0.5.4 | Fresh hex v0.3.2 | Fresh bitflags v1.3.2 | Fresh md5 v0.6.1 | Fresh array_tool v1.0.3 | Fresh byteorder v1.4.3 | Fresh libloading v0.7.0 | Fresh textwrap v0.11.0 | Fresh libc v0.2.100 | Fresh synom v0.11.3 | Fresh heck v0.3.3 | Fresh proc-macro2 v1.0.28 | Fresh memchr v2.3.4 | Fresh radium v0.5.3 | Fresh ryu v1.0.5 | Fresh log v0.4.14 | Fresh quote v1.0.9 | Fresh aho-corasick v0.7.15 | Fresh bitvec v0.19.5 | Fresh atty v0.2.14 | Fresh which v3.1.1 | Fresh clang-sys v1.2.0 | Fresh getrandom v0.1.16 | Fresh indexmap v1.7.0 | Fresh num-traits v0.2.14 | Fresh syn v0.11.11 | Fresh time v0.1.44 | Fresh syn v1.0.75 | Fresh nom v6.2.1 | Fresh regex v1.4.6 | Fresh clap v2.33.3 | Fresh rand_core v0.5.1 | Fresh num-integer v0.1.44 | Fresh num-traits v0.1.43 | Fresh env_logger v0.8.4 | Fresh cexpr v0.5.0 | Fresh serde_derive v1.0.128 | Fresh rand_chacha v0.2.2 | Fresh enum-primitive-derive v0.1.2 | Fresh strum_macros v0.21.1 | Fresh chrono v0.4.19 | Fresh bindgen v0.59.1 | Fresh serde v1.0.128 | Fresh rand v0.7.3 | Fresh serde_json v1.0.66 | Compiling redis-module v0.23.0 | Fresh jsonpath_lib v0.2.6 (https://github.com/RedisJSON/jsonpath.git?branch=generic_json_path#14af7aa5) | Fresh bson v0.14.1 | Running /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/release/build/redis-module-96a8b1c787effaba/build-script-build | error: failed to run custom build command for redis-module v0.23.0 | | Caused by: | process didn't exit successfully: /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/release/build/redis-module-96a8b1c787effaba/build-script-build (exit status: 101) | --- stdout | TARGET = Some("armv7-unknown-linux-gnueabihf") | OPT_LEVEL = Some("3") | HOST = Some("x86_64-unknown-linux-gnu") | CC_armv7-unknown-linux-gnueabihf = None | CC_armv7_unknown_linux_gnueabihf = None | TARGET_CC = Some("/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/wrappers/cc-wrapper.sh") | CFLAGS_armv7-unknown-linux-gnueabihf = None | CFLAGS_armv7_unknown_linux_gnueabihf = None | TARGET_CFLAGS = Some(" -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0=/usr/src/debug/redisjson/1.0.8-r0 -fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot= -fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot-native= ") | CRATE_CC_NO_DEFAULTS = None | DEBUG = Some("true") | CARGO_CFG_TARGET_FEATURE = None | running: "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/wrappers/cc-wrapper.sh" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-march=armv7-a" "-O2" "-pipe" "-g" "-feliminate-unused-debug-types" "-fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0=/usr/src/debug/redisjson/1.0.8-r0" "-fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot=" "-fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot-native=" "-I" "src/include/" "-DREDISMODULE_EXPERIMENTAL_API" "-o" "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-ee71a5412677483f/out/src/redismodule.o" "-c" "src/redismodule.c" | exit status: 0 | AR_armv7-unknown-linux-gnueabihf = None | AR_armv7_unknown_linux_gnueabihf = None | TARGET_AR = None | AR = Some("arm-fslc-linux-gnueabi-ar") | running: "arm-fslc-linux-gnueabi-ar" "cq" "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-ee71a5412677483f/out/libredismodule.a" "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-ee71a5412677483f/out/src/redismodule.o" | exit status: 0 | running: "arm-fslc-linux-gnueabi-ar" "s" "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-ee71a5412677483f/out/libredismodule.a" | exit status: 0 | cargo:rustc-link-lib=static=redismodule | cargo:rustc-link-search=native=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-ee71a5412677483f/out | | --- stderr | /usr/include/sys/types.h:145:10: fatal error: 'stddef.h' file not found | /usr/include/sys/types.h:145:10: fatal error: 'stddef.h' file not found, err: true | thread 'main' panicked at 'error generating bindings: ()', /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/cargo_home/registry/src/github.com-1ecc6299db9ec823/redis-module-0.23.0/build.rs:76:10 | stack backtrace: | 0: rust_begin_unwind | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5 | 1: core::panicking::panic_fmt | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/panicking.rs:92:14 | 2: core::result::unwrap_failed | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/result.rs:1355:5 | 3: core::result::Result<T,E>::expect | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/result.rs:997:23 | 4: build_script_build::main | at ./build.rs:68:20 | 5: core::ops::function::FnOnce::call_once | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ops/function.rs:227:5 | note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. | WARNING: exit code 101 from a shell command.

NOTE: I do not want to install rustc and cargo into SYSROOT.

Any ideas from where failed compilation ?

[posborne]

Looks like it failed trying to do bindgen stuff for redis-modules. Bindgen has additional requirements (libclang, etc.) that will need to be modeled in your recipe for the build to have a chance, but I don't have direct experience sorting out this particular issue so you'll need to do your own digging but we would love a comment back here if you find a solution or believe that something is missing from this layer that is required or would help with the process significantly.

You'll probably want to drop into a devshell and poke around, adding more verbose logging, etc.

[posborne]

Looks like this relates to #96

[posborne]

For now, the workaround mentioned in #75 is worth a shot.

[TopperBG]

Thanks a lot, will look for linket tickets. Will infor here for eventual resolution. BTW yes, I have on host machine libclang-dev installed.

[TopperBG]

Not going right way. I've added in particular recipe for RedisJSON those rows:

export BINDGEN_EXTRA_CLANG_ARGS
BINDGEN_EXTRA_CLANG_ARGS = "--sysroot=${WORKDIR}/recipe-sysroot -I${WORKDIR}/recipe-sysroot/usr/include"

But error still exist even now is in use sysroot/types.h

Running `/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/release/build/redis-module-859adc16bceddc99/build-script-build`
| error: failed to run custom build command for `redis-module v0.23.0`
| 
| Caused by:
|   process didn't exit successfully: `/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/release/build/redis-module-859adc16bceddc99/build-script-build` (exit code: 101)
|   --- stdout
|   TARGET = Some("armv7-unknown-linux-gnueabihf")
|   OPT_LEVEL = Some("3")
|   HOST = Some("x86_64-unknown-linux-gnu")
|   CC_armv7-unknown-linux-gnueabihf = None
|   CC_armv7_unknown_linux_gnueabihf = None
|   TARGET_CC = Some("/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/wrappers/cc-wrapper.sh")
|   CFLAGS_armv7-unknown-linux-gnueabihf = None
|   CFLAGS_armv7_unknown_linux_gnueabihf = None
|   TARGET_CFLAGS = Some(" -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0=/usr/src/debug/redisjson/1.0.8-r0 -fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot= -fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot-native= ")
|   CRATE_CC_NO_DEFAULTS = None
|   DEBUG = Some("true")
|   CARGO_CFG_TARGET_FEATURE = None
|   running: "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/wrappers/cc-wrapper.sh" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-march=armv7-a" "-O2" "-pipe" "-g" "-feliminate-unused-debug-types" "-fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0=/usr/src/debug/redisjson/1.0.8-r0" "-fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot=" "-fdebug-prefix-map=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot-native=" "-I" "src/include/" "-DREDISMODULE_EXPERIMENTAL_API" "-o" "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-b694219d9a7506bb/out/src/redismodule.o" "-c" "src/redismodule.c"
|   exit code: 0
|   AR_armv7-unknown-linux-gnueabihf = None
|   AR_armv7_unknown_linux_gnueabihf = None
|   TARGET_AR = None
|   AR = Some("arm-fslc-linux-gnueabi-ar")
|   running: "arm-fslc-linux-gnueabi-ar" "cq" "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-b694219d9a7506bb/out/libredismodule.a" "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-b694219d9a7506bb/out/src/redismodule.o"
|   exit code: 0
|   running: "arm-fslc-linux-gnueabi-ar" "s" "/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-b694219d9a7506bb/out/libredismodule.a"
|   exit code: 0
|   cargo:rustc-link-lib=static=redismodule
|   cargo:rustc-link-search=native=/home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/target/armv7-unknown-linux-gnueabihf/release/build/redis-module-b694219d9a7506bb/out
| 
|   --- stderr
|   /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot/usr/include/sys/types.h:144:10: fatal error: 'stddef.h' file not found
|   /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/recipe-sysroot/usr/include/sys/types.h:144:10: fatal error: 'stddef.h' file not found, err: true
|   thread 'main' panicked at 'error generating bindings: ()', /home/optix/thud/build_fb/tmp/work/armv7at2hf-neon-fslc-linux-gnueabi/redisjson/1.0.8-r0/cargo_home/registry/src/github.com-1ecc6299db9ec823/redis-module-0.23.0/build.rs:76:10

That's more strange, the file exists:

/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */

#ifndef __always_inline #define __always_inline inline #endif

but it's called relative from types.h

#define	__need_size_t
#include <stddef.h>

#ifdef __USE_MISC

[TopperBG]

BTW I'm useing Zeus branch of meta-rust-bin, could be that be a problem ?

[samliddicott]

This could also be related to https://github.com/rust-lang/cc-rs/issues/82 especially this reply.

Otherwise for cross-compile, build-scripts compiled for host may fail to run in the case where the host has the same triple as the target, and then cargo assumes that the target tools can be used for the host, and target libc might be used.

For me, in a different project using host rust tools, this turned out to require:

+# enable nightly options +RUSTENV+=__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" +# CARGO_UNSTABLE_TARGET_APPLIES_TO_HOST enables CARGO_TARGET_APPLIES_TO_HOST in non-nightly builds (https://github.com/rust-lang/cc-rs/issues/82#issuecomment-1310510272) +RUSTENV+=CARGO_UNSTABLE_TARGET_APPLIES_TO_HOST="true" CARGO_TARGET_APPLIES_TO_HOST="false" +RUSTENV+=CARGO_TARGET_LINKER='$(CC)' TARGET_LINKER='$(CC)' TARGET_LD='$(CC)' TARGET_LDFLAGS= LDFLAGS= +RUSTENV+=CARGO_TARGET_CC='$(CC)' TARGET_CC='$(CC)' TARGET_CFLAGS=

+# CARGO_UNSTABLE_HOST_CONFIG="true" enables CARGO_HOST_LINKER="gcc" and other host tools configurations +RUSTENV+=CARGO_UNSTABLE_HOST_CONFIG="true" +RUSTENV+=CARGO_HOST_LINKER=gcc HOST_LINKER=gcc HOST_LD=gcc HOST_LDFLAGS= +RUSTENV+=CARGO_HOST_CC=gcc HOST_CC=gcc HOST_CFLAGS= + +RUSTENV+=RUSTFLAGS='$(RUSTFLAGS)' + +CARGO=env $(RUSTENV) cargo

I'm trying to use meta-rust-bin instead of host tools, but I'm not sure how those changes above would be transposed into this project.

[samliddicott]

This seems to be the equivalent fix in Kirkstone so that build/sciprt/build will have the correct linker so it can run on the host

https://git.openembedded.org/openembedded-core/commit/meta/recipes-devtools/rust?id=02133f92c56c90a5365985b515953e4136e112aa

[samliddicott]

I still find that cross-compiled code with a different triple produces build-script like this, using the loader in the uninative sysroot, and it executes fine:

$ file /build/arm/build_output/work/aarch64-soniccorex-linux/cbindgen/0.24.3-r0/target/release/build/log-6481e63a95fe6178/build-script-build /build/arm/build_output/work/aarch64-soniccorex-linux/cbindgen/0.24.3-r0/target/release/build/log-6481e63a95fe6178/build-script-build: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /build/arm/build_output/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2, BuildID[sha1]=e2c81d808dc3867ea982e9d293ae9ee378c67568, for GNU/Linux 3.2.0, with debug_info, not stripped

But when cross compiled to the same triple as the host, it has the leading path to the loader stripped, to refer to plain /lib/... and won't run (file not found) because that loader isn't there on the host:

file /build/x86/build_output/work/corei7-64-soniccorex-linux/cbindgen/0.24.3-r0/target/release/build/log-6481e63a95fe6178/build-script-build /build/x86/build_output/work/corei7-64-soniccorex-linux/cbindgen/0.24.3-r0/target/release/build/log-6481e63a95fe6178/build-script-build: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-x86-64.so.2, BuildID[sha1]=1f4ae69a4bd659a5844d8edf12c637772593ea87, for GNU/Linux 3.2.0, with debug_info, not stripped

So I'm not sure why, given all the precautions, the "interpreter" is different when building the build-script while cross compiling

[EDIT] the cause is that the linker-native-wrapper is not being invoked at all when the host and target triple match, so we need CARGO_TARGET_APPLIES_TO_HOST="false"

[samliddicott]

If you want to try this branch: https://github.com/samliddicott/meta-rust-bin/commits/master then redis-json will build as far as do_package_qa:

ERROR: redis-json-git-r0 do_package_qa: QA Issue: -dev package redis-json-dev contains non-symlink .so '/usr/lib/librejson.so' [dev-elf]
ERROR: redis-json-git-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: Logfile of failure stored in: /build/S40/nsa4700/build_output/work/corei7-64-soniccorex-linux/redis-json/git-r0/temp/log.do_package_qa.2850937
ERROR: Task (/build/S40/nsa4700/../projects/lithium/sonicgit/soniccore/componentsx/meta-rust-bin/recipes-devtools/rust/redis-json_git.bb:do_package_qa) failed with exit code '1'

That branch has @rich-g's fix https://github.com/rust-embedded/meta-rust-bin/issues/128 as well as my fix based on guidance from @jameshilliard here https://github.com/rust-lang/cc-rs/issues/82#issuecomment-1310233674

I tested it cross-compiling on x86 for both arm and for a different x86

[samliddicott]

I see that @banditopazzo has a similar fix as a pull-request as of two weeks ago

[posborne]

Should be resolved as of #146