proxy
proxy copied to clipboard
cilium
cilium-envoy failing on RPi4 + RaspbianOS
While trying to deploy cilium in my RPi4, cilium-envoy crashes wih:
external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:625] MmapAligned() failed - unable to allocate with tag (hint, size, alignment) - is something limiting address placement? 0x15acc0000000 1073741824 1073741824 @ 0x555ab304c4 0x555ab2c848 0x555ab2c1d4 0x555ab0ba40 0x555ab29ec8 0x555ab29c9c 0x555ab01208 0x555aa1a8e0 0x555aa19614 0x7fb577754c
external/com_github_google_tcmalloc/tcmalloc/arena.cc:58] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size); is something preventing mmap from succeeding (sandbox, VSS limitations)? 131072 632 @ 0x555ab30834 0x555ab0bad0 0x555ab29ec8 0x555ab29c9c 0x555ab01208 0x555aa1a8e0 0x555aa19614 0x7fb577754c
I still have available RAM on the system. Further debugging might indicate that i am missing some Kernel enabled configs:
cat config-6.6.62+rpt-rpi-* | grep -E 'CONFIG_BPF_JIT|CONFIG_NET_SCH_INGRESS|CONFIG_CRYPTO_USER_API_HASH'
CONFIG_BPF_JIT=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_BPF_JIT=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_CRYPTO_USER_API_HASH=m
From page https://docs.cilium.io/en/latest/operations/system_requirements/#base-requirements. Cant really install linux-modules-extra-raspi from raspios unless i download and install the packages from the ubuntu which might be compatible. Otherwise would need to rebuild the kernel?(which i never did) and might not persist changes between future updates./
Not exactly sure on the best course of action here:
cilium was deployed with cilium-cli downloaded directly on the RPi node with:
cilium install --helm-set ipam.operator.clusterPoolIPv6MaskSize=120 --helm-set ipv6.enabled=true --helm-set ipv4.enabled=false --helm-set routingMode=native --helm-set autoDirectNodeRoutes=true --helm-set ipv6NativeRoutingCIDR=bipbop::/64 --helm-set kubeProxyReplacement=true --helm-set ipam.operator.clusterPoolIPv6PodCIDRList="fd00:10:244::/104" --helm-set nodeinit.enabled=true --helm-set k8s.requireIPv6PodCIDR=true --helm-set k8sServiceHost=bipbop --helm-set k8sServicePort=6443 --helm-set hubble.enabled=true --helm-set hubble.relay.enabled=true
👋
We have one workaround with gperftools, can you give it a try ?
https://github.com/cilium/proxy?tab=readme-ov-file#building-for-the-raspberry-pi-kernelc
👋
We have one workaround with gperftools, can you give it a try ?
https://github.com/cilium/proxy?tab=readme-ov-file#building-for-the-raspberry-pi-kernelc
Should be able to cross compile yeah? Having some errors thrown out, need to check with more attention
Very cryptic failure, not sure what. Seems like I can build fine without using the extra build argument
BAZEL_BUILD_OPTS="--define tcmalloc=gperftools"
#17 8.241 [1 / 10] [Prepa] BazelWorkspaceStatusAction stable-status.txt
#17 9.682 [3,622 / 9,640] Foreign Cc - Configure: Building gperftools_build; 0s remote-cache, processwrapper-sandbox ... (4 actions, 2 running)
#17 10.73 [6,852 / 9,646] Foreign Cc - Configure: Building gperftools_build; 1s remote-cache, processwrapper-sandbox ... (29 actions, 28 running)
#17 11.74 [6,852 / 9,646] Foreign Cc - Configure: Building gperftools_build; 2s remote-cache, processwrapper-sandbox ... (30 actions running)
#17 11.79 ERROR: /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/external/envoy/bazel/foreign_cc/BUILD:32:15: Foreign Cc - Configure: Building gperftools_build failed: (Exit 77): bash failed: error executing command (from target @envoy//bazel/foreign_cc:gperftools_build)
#17 11.79 (cd /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium && \
#17 11.79 exec env - \
#17 11.79 BAZEL_COMPILER=clang \
#17 11.79 BAZEL_LINKLIBS=-l%:libstdc++.a \
#17 11.79 BAZEL_LINKOPTS=-lm \
#17 11.79 CC=/usr/lib/llvm-17/bin/clang \
#17 11.79 CXX=/usr/lib/llvm-17/bin/clang++ \
#17 11.79 LLVM_CONFIG=/usr/lib/llvm-17/bin/llvm-config \
#17 11.79 PATH=/usr/lib/llvm-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
#17 11.79 /bin/bash -c bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/wrapper_build_script.sh)
#17 11.79 # Configuration: d13fb2b1db2396dd0d63c26115f21fb2fb530ca15c43b13b95e8be6f91e06e6a
#17 11.79 # Execution platform: @local_config_platform//:host
#17 11.79
#17 11.79 Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
#17 11.79 rules_foreign_cc: Build failed!
#17 11.79 rules_foreign_cc: Keeping temp build directory and dependencies directory for debug.
#17 11.79 rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify --sandbox_debug Bazel command line flag.
#17 11.79 rules_foreign_cc: Printing build logs:
#17 11.79 _____ BEGIN BUILD LOGS _____
#17 11.79
#17 11.79 Bazel external C/C++ Rules. Building library gperftools_build
#17 11.79
#17 11.79 Environment:______________
#17 11.79 BUILD_SCRIPT=bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/build_script.sh
#17 11.79 EXT_BUILD_ROOT=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium
#17 11.79 BUILD_LOG=bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/Configure.log
#17 11.79 PWD=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium
#17 11.79 LLVM_CONFIG=/usr/lib/llvm-17/bin/llvm-config
#17 11.79 CXX=/usr/lib/llvm-17/bin/clang++
#17 11.79 BUILD_WRAPPER_SCRIPT=bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/wrapper_build_script.sh
#17 11.79 BAZEL_LINKOPTS=-lm
#17 11.79 TMPDIR=/tmp
#17 11.79 EXT_BUILD_DEPS=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.ext_build_deps
#17 11.79 BAZEL_LINKLIBS=-l%:libstdc++.a
#17 11.79 BUILD_TMPDIR=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.build_tmpdir
#17 11.79 SHLVL=2
#17 11.79 INSTALLDIR=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build
#17 11.79 PATH=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium:/usr/lib/llvm-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#17 11.79 BAZEL_COMPILER=clang
#17 11.79 CC=/usr/lib/llvm-17/bin/clang
#17 11.79 _=/usr/bin/env
#17 11.79 __________________________
#17 11.79 + mkdir -p /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.build_tmpdir/gperftools_build
#17 11.79 + ARFLAGS=rcsD
#17 11.79 + AR_FLAGS=rcsD
#17 11.79 + ASFLAGS='--target=aarch64-unknown-linux-gnu -fuse-ld=lld -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -Wno-unused-command-line-argument -Wno-deprecated-builtins -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted -DABSL_MIN_LOG_LEVEL=4 -fdebug-types-section -fPIC -Wno-deprecated-declarations -fexceptions'
#17 11.79 + CFLAGS='--target=aarch64-unknown-linux-gnu -fuse-ld=lld -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -Wno-unused-command-line-argument -Wno-deprecated-builtins -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted -DABSL_MIN_LOG_LEVEL=4 -fdebug-types-section -fPIC -Wno-deprecated-declarations -fexceptions'
#17 11.79 + CXXFLAGS='--target=aarch64-unknown-linux-gnu -fuse-ld=lld -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -Wno-unused-command-line-argument -Wno-deprecated-builtins -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -std=c++0x -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted -DABSL_MIN_LOG_LEVEL=4 -fdebug-types-section -fPIC -Wno-deprecated-declarations -std=c++17'
#17 11.79 + LDFLAGS='--target=aarch64-unknown-linux-gnu -fuse-ld=lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -lm -Wl,--gc-sections -l:libstdc++.a -L/usr/lib/llvm-17/lib -Wl,-rpath,/usr/lib/llvm-17/lib -Wl,--strip-all -lpthread'
#17 11.79 + AR=/usr/bin/llvm-ar-17
#17 11.79 + CC=/usr/bin/clang-17
#17 11.79 + CXX=/usr/bin/clang-17
#17 11.79 + RANLIB=:
#17 11.79 + CPPFLAGS=
#17 11.79 + /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/external/com_github_gperftools_gperftools/configure --prefix=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.build_tmpdir/gperftools_build --enable-shared=no --enable-frame-pointers --disable-libunwind
#17 11.79 checking build system type... x86_64-pc-linux-gnu
#17 11.79 checking host system type... x86_64-pc-linux-gnu
#17 11.79 checking for a BSD-compatible install... /usr/bin/install -c
#17 11.79 checking whether build environment is sane... yes
#17 11.79 checking for a race-free mkdir -p... /usr/bin/mkdir -p
#17 11.79 checking for gawk... no
#17 11.79 checking for mawk... mawk
#17 11.79 checking whether /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make/bin/make sets $(MAKE)... yes
#17 11.79 checking whether /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make/bin/make supports nested variables... yes
#17 11.79 checking whether to enable maintainer-specific portions of Makefiles... no
#17 11.79 checking for git... /usr/bin/git
#17 11.79 checking whether /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make/bin/make supports the include directive... yes (GNU style)
#17 11.79 checking for gcc... /usr/bin/clang-17
#17 11.79 checking whether the C compiler works... yes
#17 11.79 checking for C compiler default output file name... a.out
#17 11.79 checking for suffix of executables...
#17 11.79 checking whether we are cross compiling... configure: error: in `/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.build_tmpdir':
#17 11.79 configure: error: cannot run C compiled programs.
#17 11.79 If you meant to cross compile, use `--host'.
#17 11.79 See `config.log' for more details
#17 11.79 _____ END BUILD LOGS _____
#17 11.79 rules_foreign_cc: Build wrapper script location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/wrapper_build_script.sh
#17 11.79 rules_foreign_cc: Build script location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/build_script.sh
#17 11.79 rules_foreign_cc: Build log location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/Configure.log
#17 11.79
#17 12.04 Target //:cilium-envoy failed to build
#17 12.05 INFO: Elapsed time: 8.387s, Critical Path: 3.38s
#17 12.05 INFO: 16 processes: 16 internal.
#17 12.05 FAILED: Build did NOT complete successfully
#17 12.05 make: *** [Makefile:121: bazel-bin/cilium-envoy] Error 1
#17 ERROR: process "/bin/sh -c if [ -f /tmp/bazel-cache/ENVOY_VERSION ]; then CACHE_ENVOY_VERSION=`cat /tmp/bazel-cache/ENVOY_VERSION`; ENVOY_VERSION=`cat ENVOY_VERSION`; if [ \"${CACHE_ENVOY_VERSION}\" != \"${ENVOY_VERSION}\" ]; then echo \"Building Envoy ${ENVOY_VERSION} with bazel archive from different Envoy version (${CACHE_ENVOY_VERSION})\"; else echo \"Building Envoy ${ENVOY_VERSION} with bazel cache of the same version\"; fi; else echo \"Bazel cache has no ENVOY_VERSION, it may be empty.\"; fi && touch /tmp/bazel-cache/permissions-check && if [ -n \"${COPY_CACHE_EXT}\" ]; then PKG_BUILD=1 make BUILD_DEP_HASHES; if [ -f /tmp/bazel-cache/BUILD_DEP_HASHES ] && ! diff BUILD_DEP_HASHES /tmp/bazel-cache/BUILD_DEP_HASHES; then echo \"Build dependencies have changed, clearing bazel cache\"; rm -rf /tmp/bazel-cache/*; rm -rf /cilium/proxy/.cache/*; fi ; cp BUILD_DEP_HASHES ENVOY_VERSION /tmp/bazel-cache; fi && BAZEL_BUILD_OPTS=\"${BAZEL_BUILD_OPTS} --disk_cache=/tmp/bazel-cache\" PKG_BUILD=1 V=$V DEBUG=$DEBUG DESTDIR=/tmp/install make install && if [ -n \"${COPY_CACHE_EXT}\" ]; then cp -ra /tmp/bazel-cache /tmp/bazel-cache${COPY_CACHE_EXT}; ls -la /tmp/bazel-cache${COPY_CACHE_EXT}; fi" did not complete successfully: exit code: 2
------
> [builder 5/6] RUN --mount=mode=0777,uid=1337,gid=1337,target=/cilium/proxy/.cache,type=cache,id=arm64,sharing=private --mount=target=/tmp/bazel-cache,source=/tmp/bazel-cache,from=builder-cache,rw if [ -f /tmp/bazel-cache/ENVOY_VERSION ]; then CACHE_ENVOY_VERSION=`cat /tmp/bazel-cache/ENVOY_VERSION`; ENVOY_VERSION=`cat ENVOY_VERSION`; if [ "${CACHE_ENVOY_VERSION}" != "${ENVOY_VERSION}" ]; then echo "Building Envoy ${ENVOY_VERSION} with bazel archive from different Envoy version (${CACHE_ENVOY_VERSION})"; else echo "Building Envoy ${ENVOY_VERSION} with bazel cache of the same version"; fi; else echo "Bazel cache has no ENVOY_VERSION, it may be empty."; fi && touch /tmp/bazel-cache/permissions-check && if [ -n "${COPY_CACHE_EXT}" ]; then PKG_BUILD=1 make BUILD_DEP_HASHES; if [ -f /tmp/bazel-cache/BUILD_DEP_HASHES ] && ! diff BUILD_DEP_HASHES /tmp/bazel-cache/BUILD_DEP_HASHES; then echo "Build dependencies have changed, clearing bazel cache"; rm -rf /tmp/bazel-cache/*; rm -rf /cilium/proxy/.cache/*; fi ; cp BUILD_DEP_HASHES ENVOY_VERSION /tmp/bazel-cache; fi && BAZEL_BUILD_OPTS="--define tcmalloc=gperftools --verbose_failures --disk_cache=/tmp/bazel-cache" PKG_BUILD=1 V=$V DEBUG=$DEBUG DESTDIR=/tmp/install make install && if [ -n "${COPY_CACHE_EXT}" ]; then cp -ra /tmp/bazel-cache /tmp/bazel-cache${COPY_CACHE_EXT}; ls -la /tmp/bazel-cache${COPY_CACHE_EXT}; fi:
11.79 _____ END BUILD LOGS _____
11.79 rules_foreign_cc: Build wrapper script location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/wrapper_build_script.sh
11.79 rules_foreign_cc: Build script location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/build_script.sh
11.79 rules_foreign_cc: Build log location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/Configure.log
11.79
12.04 Target //:cilium-envoy failed to build
12.05 INFO: Elapsed time: 8.387s, Critical Path: 3.38s
12.05 INFO: 16 processes: 16 internal.
12.05 FAILED: Build did NOT complete successfully
12.05 make: *** [Makefile:121: bazel-bin/cilium-envoy] Error 1
------
Dockerfile:69
--------------------
68 | RUN ./bazel/get_workspace_status
69 | >>> RUN --mount=mode=0777,uid=1337,gid=1337,target=/cilium/proxy/.cache,type=cache,id=$TARGETARCH,sharing=private \
70 | >>> --mount=target=/tmp/bazel-cache,source=/tmp/bazel-cache,from=builder-cache,rw \
71 | >>> if [ -f /tmp/bazel-cache/ENVOY_VERSION ]; then CACHE_ENVOY_VERSION=`cat /tmp/bazel-cache/ENVOY_VERSION`; ENVOY_VERSION=`cat ENVOY_VERSION`; if [ "${CACHE_ENVOY_VERSION}" != "${ENVOY_VERSION}" ]; then echo "Building Envoy ${ENVOY_VERSION} with bazel archive from different Envoy version (${CACHE_ENVOY_VERSION})"; else echo "Building Envoy ${ENVOY_VERSION} with bazel cache of the same version"; fi; else echo "Bazel cache has no ENVOY_VERSION, it may be empty."; fi && \
72 | >>> touch /tmp/bazel-cache/permissions-check && \
73 | >>> if [ -n "${COPY_CACHE_EXT}" ]; then PKG_BUILD=1 make BUILD_DEP_HASHES; if [ -f /tmp/bazel-cache/BUILD_DEP_HASHES ] && ! diff BUILD_DEP_HASHES /tmp/bazel-cache/BUILD_DEP_HASHES; then echo "Build dependencies have changed, clearing bazel cache"; rm -rf /tmp/bazel-cache/*; rm -rf /cilium/proxy/.cache/*; fi ; cp BUILD_DEP_HASHES ENVOY_VERSION /tmp/bazel-cache; fi && \
74 | >>> BAZEL_BUILD_OPTS="${BAZEL_BUILD_OPTS} --disk_cache=/tmp/bazel-cache" PKG_BUILD=1 V=$V DEBUG=$DEBUG DESTDIR=/tmp/install make install && \
75 | >>> if [ -n "${COPY_CACHE_EXT}" ]; then cp -ra /tmp/bazel-cache /tmp/bazel-cache${COPY_CACHE_EXT}; ls -la /tmp/bazel-cache${COPY_CACHE_EXT}; fi
76 | #
--------------------
ERROR: failed to solve: process "/bin/sh -c if [ -f /tmp/bazel-cache/ENVOY_VERSION ]; then CACHE_ENVOY_VERSION=`cat /tmp/bazel-cache/ENVOY_VERSION`; ENVOY_VERSION=`cat ENVOY_VERSION`; if [ \"${CACHE_ENVOY_VERSION}\" != \"${ENVOY_VERSION}\" ]; then echo \"Building Envoy ${ENVOY_VERSION} with bazel archive from different Envoy version (${CACHE_ENVOY_VERSION})\"; else echo \"Building Envoy ${ENVOY_VERSION} with bazel cache of the same version\"; fi; else echo \"Bazel cache has no ENVOY_VERSION, it may be empty.\"; fi && touch /tmp/bazel-cache/permissions-check && if [ -n \"${COPY_CACHE_EXT}\" ]; then PKG_BUILD=1 make BUILD_DEP_HASHES; if [ -f /tmp/bazel-cache/BUILD_DEP_HASHES ] && ! diff BUILD_DEP_HASHES /tmp/bazel-cache/BUILD_DEP_HASHES; then echo \"Build dependencies have changed, clearing bazel cache\"; rm -rf /tmp/bazel-cache/*; rm -rf /cilium/proxy/.cache/*; fi ; cp BUILD_DEP_HASHES ENVOY_VERSION /tmp/bazel-cache; fi && BAZEL_BUILD_OPTS=\"${BAZEL_BUILD_OPTS} --disk_cache=/tmp/bazel-cache\" PKG_BUILD=1 V=$V DEBUG=$DEBUG DESTDIR=/tmp/install make install && if [ -n \"${COPY_CACHE_EXT}\" ]; then cp -ra /tmp/bazel-cache /tmp/bazel-cache${COPY_CACHE_EXT}; ls -la /tmp/bazel-cache${COPY_CACHE_EXT}; fi" did not complete successfully: exit code: 2
make: *** [Makefile.docker:221: docker-image-envoy] Error 1
After analyzing this a bit. It seems that there is something happening with cross-compile and this gperftools options.
From the output it looks like when cross-compiling I need to use a --host flag
#17 11.79 configure: error: cannot run C compiled programs.
#17 11.79 If you meant to cross compile, use `--host'.
#17 11.79 See `config.log' for more details
I dont seem to be able to find this config.log file (host or buildx container) and also trying out to include --host in bazel configs throws an error. This time i changed the options to clear the cache, to make sure something weird was not happening:
NO_CACHE=1 NO_ARCHIVE=1 DOCKER_BUILD_OPTS="--output=out" ARCH=multi BAZEL_BUILD_OPTS="--define tcmalloc=gperftools --verbose_failures --sandbox_debug" sudo -E make docker-image-envoy
For reference, i'm trying to compile for v1.29, commit ID 2c13e1e06c848aeae988ba8e14b2a6608df9b2a5. Also noticed that new versions are breaking do to missing dependent containers
My current process is to try and build the image(only arm) in the RPI4 directly, which will take some time...
BUILDKIT_PROGRESS=plain NO_CACHE=1 NO_ARCHIVE=1 DOCKER_BUILD_OPTS="--output=out" ARCH=arm64 BAZEL_BUILD_OPTS="--define tcmalloc=gperftools --verbose_failures --sandbox_debug" sudo -E make docker-image-envoy > build.log
Will then build the amd one and stich them with buildx or something if it works. 🤞 Edit: after writting this, dont think it really makes sense the arm only on arm processor build would work because the virtualization of buildx... Yeah... so kinda stuck until someone can help...
I personally rely on cross compilation, however, sometime local testing can be done with the below command directly.
If it helps, might need to bundle binary to docker image.
$ bazel build cilium-envoy --cpu aarch64
I personally rely on cross compilation, however, sometime local testing can be done with the below command directly.
If it helps, might need to bundle binary to docker image.
$ bazel build cilium-envoy --cpu aarch64
Finally build after couple days, will give it a try in the next days
ARCH=arm64 BAZEL_BUILD_OPTS="--define tcmalloc=disabled" make docker-image-envoy this should do.
Is this a recent development? I think i tried changing/disabling the tcmalloc and no luck.
I found it easier to recompile the linux kernel following this blogpost (can't find it) But I had this on my notes:
tmux new -s session-name
sudo apt install git bc bison flex libssl-dev make ncurses-devel libncurses-dev
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux
KERNEL=kernel8
make bcm2711_defconfig
vim .config
CONFIG_ARM64_VA_BITS_48=y
CONFIG_ARM64_VA_BITS=48
# drop anything related to memory management 39 bits
# EDIT FROM THE FUTURE:
# The linux repo used here needs to enable the IPv6 COMPONENT? options on the config file (if you use those)
make -j4 Image.gz modules dtbs
sudo make modules_install
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/
sudo cp arch/arm64/boot/Image.gz /boot/firmware/kernel8-48bit.img
Add the following line to your /boot/firmware/config.txt
kernel=kernel8-48bit.img
I tried to recompile cillium a while ago and didnt try again. Would more easily change the OS than trying to understand how to fix the compilation scripts/makefiles
0864395884b263913eac200ee2048fd985f8e626 I'm on cilium v1.15.18 though. You can find hash for yours here https://github.com/cilium/cilium/blob/94ced16a7bafa24642f3f4d08ea30afdacaf9a08/images/cilium/Dockerfile#L9
i can reproduce the tmalloc errors on a fresh k3s.io cluster per https://docs.k3s.io/networking/basic-network-options?cni=Cilium#custom-cni and https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-cilium (k3s tab) with cilium install --version 1.18.2
that's on Raspberry Pi OS Lite 64-bit (trixie): https://www.raspberrypi.com/software/operating-systems/
