Gdn failed to run on ubuntu bionic

[xtremerui]

We noticed in Concourse CI testing, workers failed to start by error

gdn: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by gdn)
gdn: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by gdn)

it is running on ubuntu bionic.

version: 1.21.0

[MarcPaquette]

Hi @xtremerui ,

Are you in a position where you can install glibc?

Thanks, Marc

[xtremerui]

yes we control the dependency installation for the env where we build our binary.

[MarcPaquette]

@xtremerui After installing glibc does the problem resolve?

[xtremerui]

Do i need to install glibc by a specific version like 2.32 or 2.34 in this case?

[xtremerui]

@MarcPaquette i stand corrected here. For the docker image we build, we have control. But for the test that our binary runs on google cloud, we dont have control over the OS image that google provides (which is a standard google jammy jellyfish OS image family).

[MarcPaquette]

Hi @xtremerui

We just released Garden-runc-release v1.22.2, which statically compiles gdn.

Can you test it out and see if it fixes your issue?

Thanks, @MarcPaquette

[xtremerui]

@MarcPaquette Thx for the effort.

Now we are seeing

/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)

in a GCP VM where gdn is ran

ruiya@smoke-flexible-primate:/home/concourse$ ldd --version
ldd (Ubuntu GLIBC 2.27-3ubuntu1.6) 2.27
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
ruiya@smoke-flexible-primate:/home/concourse$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
ruiya@smoke-flexible-primate:/home/concourse$ 

[MarcPaquette]

Hi @xtremerui Can you confirm that the latest release resolves this issue for you?

[xtremerui]

@MarcPaquette nope we are still seeing the same error

[schindlersebastian]

Hi all, any news on this? I'm still having the same issues as @xtremerui.

/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)

GLIBC is expected to be at least 2.28, what I find in bionic/1.150 is 2.27:

ldd --version
ldd (Ubuntu GLIBC 2.27-3ubuntu1.6) 2.27
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[MarcPaquette]

We have this story in our internal backlog: https://www.pivotaltracker.com/story/show/183466259

I've been out of sabbatical for a couple months. let me ping the team to see where we have it prioritized.

CC @ameowlia

[geofffranks]

We started making the garden job use the system iptables (/sbin) by default in v1.22.0.

What version of garden + the stemcell are you seeing this on? What is your garden.iptables_bin_dir property set to? Where is this error appearing for you?

[MarcPaquette]

Hi @xtremerui

Is this issue still outstanding for you? It's been fairly quite for a bit of time and I'm wondering if we can close it out?

[xtremerui]

@geofffranks @MarcPaquette we are still getting tickets from Customer about this issue.

What version of garden + the stemcell are you seeing this on?

1.22.7 that released with Concourse v7.9.0

What is your garden.iptables_bin_dir property set to?

we don't set this property in Concourse

Where is this error appearing for you?

The above errors (both in mine and @schindlersebastian comments) show up when Concourse runs gdn binary here. It seems like gdn binary doesn't equiped with needed GLIBC so it still looks for that lib from OS.

[MarcPaquette]

@xtremerui and @schindlersebastian

Can you provide us with a reproduction steps?

FWIW, Bionic support ends in April.

[winkingturtle-vmw]

@xtremerui @schindlersebastian 1.23.0 version of the gdn is now shipped without dependency on GLIBC and build with musl. Please try them on bionic and let us know if that will solve this issue.

[winkingturtle-vmw]

I am going to close this issue, please re-open if you are still having a problem.

[xtremerui]

We have built the latest Concourse image with gdn 1.23 and it still fails on GCP VM with ubuntu-1804-lts os image, please find the detailed error below.

PS: I can't reopen this issue. Should I create a new issue instead?

error log

```console

Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: {"timestamp":"2023-02-09T22:08:38.287387443Z","level":"error","source":"guardian","message":"guardian.starting-guardian-backend","data":{"error":"bulk starter: setting up default chains: iptables: setup-global-chains: + set -o nounset\n+ set -o errexit\n+ shopt -s nullglob\n+ filter_input_chain=w--input\n+ filter_forward_chain=w--forward\n+ filter_default_chain=w--default\n+ filter_instance_prefix=w--instance-\n+ nat_prerouting_chain=w--prerouting\n+ nat_postrouting_chain=w--postrouting\n+ nat_instance_prefix=w--instance-\n+ iptables_bin=/var/gdn/assets/linux/sbin/iptables\n+ case "${ACTION}" in\n+ setup_filter\n+ teardown_filter\n+ teardown_deprecated_rules\n++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT\n+ rules=\n+ true\n+ echo ''\n+ grep ' -j garden-dispatch'\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD\n+ rules=\n+ true\n+ grep ' -j garden-dispatch'\n+ echo ''\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ /var/gdn/assets/linux/sbin/iptables -w -F garden-dispatch\n+ true\n+ /var/gdn/assets/linux/sbin/iptables -w -X garden-dispatch\n+ true\n++ /var/gdn/assets/linux/sbin/iptables -w -S w--forward\n+ rules=\n+ true\n+ echo ''\n+ grep '\-g w--instance-'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n++ /var/gdn/assets/linux/sbin/iptables -w -S\n+ rules=\n+ true\n+ echo ''\n+ grep '^-A w--instance-'\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n++ /var/gdn/assets/linux/sbin/iptables -w -S\n+ rules=\n+ true\n+ echo ''\n+ grep '^-N w--instance-'\n+ sed -e s/-N/-X/ -e 's/\s\+$//'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD\n+ rules=\n+ true\n+ echo ''\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ grep ' -j w--forward'\n+ /var/gdn/assets/linux/sbin/iptables -w -F w--forward\n+ true\n+ /var/gdn/assets/linux/sbin/iptables -w -F w--default\n+ true\n++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT\n+ rules=\n+ true\n+ echo ''\n+ grep ' -j w--input'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n+ /var/gdn/assets/linux/sbin/iptables -w -F w--input\n+ true\n+ /var/gdn/assets/linux/sbin/iptables -w -X w--input\n+ true\n++ ip route show\n++ grep default\n++ head -1\n++ cut '-d ' -f5\n+ default_interface=ens4\n+ /var/gdn/assets/linux/sbin/iptables -w -N w--input\n+ /var/gdn/assets/linux/sbin/iptables -w -F w--input\n/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables)\n/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)\n/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables)\n/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)\n"}} Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: bulk starter: setting up default chains: iptables: setup-global-chains: + set -o nounset Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + set -o errexit Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + shopt -s nullglob Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_input_chain=w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_forward_chain=w--forward Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_default_chain=w--default Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_instance_prefix=w--instance- Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_prerouting_chain=w--prerouting Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_postrouting_chain=w--postrouting Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_instance_prefix=w--instance- Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + iptables_bin=/var/gdn/assets/linux/sbin/iptables Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + case "${ACTION}" in Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + setup_filter Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + teardown_filter Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + teardown_deprecated_rules Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j garden-dispatch' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j garden-dispatch' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F garden-dispatch Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -X garden-dispatch Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S w--forward Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '-g w--instance-' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '^-A w--instance-' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '^-N w--instance-' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-N/-X/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j w--forward' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--forward Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--default Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j w--input' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -X w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ ip route show Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ grep default Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ head -1 Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ cut '-d ' -f5 Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + default_interface=ens4 Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -N w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: bulk starter: setting up default chains: iptables: setup-global-chains: + set -o nounset Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + set -o errexit Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + shopt -s nullglob Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_input_chain=w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_forward_chain=w--forward Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_default_chain=w--default Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_instance_prefix=w--instance- Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_prerouting_chain=w--prerouting Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_postrouting_chain=w--postrouting Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_instance_prefix=w--instance- Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + iptables_bin=/var/gdn/assets/linux/sbin/iptables Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + case "${ACTION}" in Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + setup_filter Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + teardown_filter Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + teardown_deprecated_rules Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j garden-dispatch' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j garden-dispatch' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F garden-dispatch Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -X garden-dispatch Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S w--forward Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '-g w--instance-' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '^-A w--instance-' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '^-N w--instance-' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-N/-X/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j w--forward' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--forward Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--default Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules= Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo '' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j w--input' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//' Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -X w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ ip route show Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ grep default Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ head -1 Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ cut '-d ' -f5 Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + default_interface=ens4 Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -N w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--input Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: {"timestamp":"2023-02-09T22:08:38.292747265Z","level":"error","source":"worker","message":"worker.garden.gdn-runner.logging-runner-exited","data":{"error":"exit status 1","session":"1.2"}}

</details> 

[winkingturtle-vmw]

@xtremerui i am going to re-open this issue instead. I think the problem is that we need to build iptables with musl as well.

[winkingturtle-vmw]

@xtremerui With the above commit, we are now building iptables with musl as well, which should solve the issue you were seeing last. That said, we are using x86_64 libraries to build these binaries and that probably means that the arm64 binaries for (cloudfoundry/garden-runc-release#238) wouldn't work as expected. We have no way of verifying arm64 binaries for gdn. If it's broken, we are happy to take them off publishing list so that only x86_64 is published.

[xtremerui]

@winkingturtle-vmw thank you so much! We will soon update the CI to build arm64 binary for Concourse. Once that is done we should be able to verify it.

[winkingturtle-vmw]

@xtremerui We just released v1.25.0 with the fix for this issue. Please let us know if that solves the issues.

[xtremerui]

@winkingturtle-vmw we are now seeing this error with latest gdn release

concourse-worker-1  | {"timestamp":"2023-02-22T23:41:59.829149215Z","level":"error","source":"guardian","message":"guardian.starting-guardian-backend","data":{"error":"bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted"}}
concourse-worker-1  | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse-worker-1  | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse-worker-1  | {"timestamp":"2023-02-22T23:41:59.831115448Z","level":"error","source":"worker","message":"worker.garden.gdn-runner.logging-runner-exited","data":{"error":"exit status 1","session":"1.2"}}

Here is the gdn command if it helps:

concourse-worker-1  | gdn args: [server --bind-ip 0.0.0.0 --bind-port 7777 --depot /worker-state/depot --properties-path /worker-state/garden-properties.json --time-format rfc3339 --no-image-plugin --max-containers 250 --network-pool 10.80.0.0/16]

[winkingturtle-vmw]

@xtremerui Is this error happening now with both bionic and jammy? Is there a way for us to test out the gdn binary manually in a concourse worker?

[xtremerui]

@winkingturtle-vmw if you follow this https://github.com/concourse/concourse#quick-start in a bionic or jammy, it should pulling the latest concourse/dev docker image that built with gdn 1.25.

After your local concourse is up, you can docker exec the worker container (if it doesn't started you can exec on the web container since they are using the same base concourse/dev image).

[winkingturtle-vmw]

@xtremerui Is this error happening now with both bionic and jammy?

[xtremerui]

I have only tested in Jammy as our CI haven't reached to the phase to fan out on bionic OS testing.

[winkingturtle-vmw]

@xtremerui We ended up building the iptables on bionic so that it will continue to work on both Jammy and Bionic. I've verified that the resulting gdn binary works on a bionic workstation, and it will be release in the next version of garden-runc

@winkingturtle-vmw we are now seeing this error with latest gdn release

concourse-worker-1  | {"timestamp":"2023-02-22T23:41:59.829149215Z","level":"error","source":"guardian","message":"guardian.starting-guardian-backend","data":{"error":"bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted"}}
concourse-worker-1  | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse-worker-1  | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse-worker-1  | {"timestamp":"2023-02-22T23:41:59.831115448Z","level":"error","source":"worker","message":"worker.garden.gdn-runner.logging-runner-exited","data":{"error":"exit status 1","session":"1.2"}}

Here is the gdn command if it helps:

concourse-worker-1  | gdn args: [server --bind-ip 0.0.0.0 --bind-port 7777 --depot /worker-state/depot --properties-path /worker-state/garden-properties.json --time-format rfc3339 --no-image-plugin --max-containers 250 --network-pool 10.80.0.0/16]

In regards to this issue that you referred to. I believe this is the result of Jammy docker images using cgroups-v2 by default. In Jammy bosh stemcells, we actually turn off cfgroups-v2 and use cgroups-v1 and that's why guardian still works. At this time, there hasn't been a plan to support cgroups-v2 yet. We hope to introduce this support soon.

[xtremerui]

@winkingturtle-vmw is there an ETA on next runc release? Thank you!

[winkingturtle-vmw]

@xtremerui v1.26.0 is released now.