etcd-io
fatal error: fuse3/fuse.h: No such file or directory
Which Github Action / Prow Jobs are flaking?
-- Build files have been written to: /tmp/lazyfs/lazyfs/build
gmake[1]: Entering directory '/tmp/lazyfs/lazyfs/build'
gmake[2]: Entering directory '/tmp/lazyfs/lazyfs/build'
gmake[3]: Entering directory '/tmp/lazyfs/lazyfs/build'
gmake[3]: Leaving directory '/tmp/lazyfs/lazyfs/build'
gmake[3]: Entering directory '/tmp/lazyfs/lazyfs/build'
[ 25%] Building CXX object CMakeFiles/lazyfs.dir/include/lazyfs/fusepp/Fuse.cpp.o
In file included from /tmp/lazyfs/lazyfs/include/lazyfs/fusepp/Fuse.cpp:26:
/tmp/lazyfs/lazyfs/include/lazyfs/fusepp/Fuse.h:36:10: fatal error: fuse3/fuse.h: No such file or directory
36 | #include <fuse3/fuse.h>
| ^~~~~~~~~~~~~~
compilation terminated.
gmake[3]: *** [CMakeFiles/lazyfs.dir/build.make:76: CMakeFiles/lazyfs.dir/include/lazyfs/fusepp/Fuse.cpp.o] Error 1
gmake[3]: Leaving directory '/tmp/lazyfs/lazyfs/build'
gmake[2]: *** [CMakeFiles/Makefile2:83: CMakeFiles/lazyfs.dir/all] Error 2
gmake[2]: Leaving directory '/tmp/lazyfs/lazyfs/build'
gmake[1]: *** [Makefile:91: all] Error 2
gmake[1]: Leaving directory '/tmp/lazyfs/lazyfs/build'
make: *** [Makefile:179: bin/lazyfs] Error 2
Which tests are flaking?
Robustness test
Github Action / Prow Job link
https://github.com/etcd-io/etcd/actions/runs/8815562613/job/24197769264?pr=17868
Reason for failure (if possible)
No response
Anything else we need to know?
No response
Github is down
Run sudo apt update && sudo apt-get --yes install cmake libfuse3-dev libfuse3-3 fuse3
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [142 B]
Hit:2 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
Get:3 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Get:5 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
Err:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Get:7 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1610 kB]
Get:[8](https://github.com/etcd-io/etcd/actions/runs/8814353606/job/24194021762?pr=17861#step:8:9) http://azure.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB]
Get:[9](https://github.com/etcd-io/etcd/actions/runs/8814353606/job/24194021762?pr=17861#step:8:10) http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1830 kB]
Get:[10](https://github.com/etcd-io/etcd/actions/runs/8814353606/job/24194021762?pr=17861#step:8:11) http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [3[11](https://github.com/etcd-io/etcd/actions/runs/8814353606/job/24194021762?pr=17861#step:8:12) kB]
Get:11 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1070 kB]
Get:[12](https://github.com/etcd-io/etcd/actions/runs/8814353606/job/24194021762?pr=17861#step:8:13) http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB]
Get:13 http://azure.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB]
Get:14 http://azure.archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB]
Get:15 http://azure.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB]
Get:16 http://azure.archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB]
Hit:17 https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu jammy InRelease
Get:18 http://azure.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [[13](https://github.com/etcd-io/etcd/actions/runs/8814353606/job/24194021762?pr=17861#step:8:14)92 kB]
Get:19 http://azure.archive.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB]
Get:20 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1768 kB]
Get:21 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB]
Get:22 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB]
Get:23 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [[16](https://github.com/etcd-io/etcd/actions/runs/8814353606/job/24194021762?pr=17861#step:8:17)3 kB]
Reading package lists...
E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: The repository 'https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease' is no longer signed.
One thing that is weird is that the failure of apt-get doesn't cause the job to fail, it fails much later when it tries to reference libfuse3-dev that was not installed.
Discussed during sig-etcd triage meeting, @serathius did you want this issue to investigate the lack of failure at the apt install step? Or do you think we should just close this given we know failure will occur in this scenario just further down than apt.
Discussed during sig-etcd triage meeting, @serathius did you want this issue to investigate the lack of failure at the
aptinstall step? Or do you think we should just close this given we know failure will occur in this scenario just further down thanapt.
Yes, I think it's worth to investigate. It would be more readable if the error was reported from the place it happened, not much later.
I simulated apt update and apt-get update with a network issue (I ran a container and turned off the network in the middle of the execution). The exit code is 0, which is why it tried to continue. After research, I confirmed that apt-get update -o APT::Update::Error-Mode=any returns an exit code >0.
root@729489dce213:/# apt-get update
Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1798 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2308 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1082 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [44.7 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2069 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [51.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1374 kB]
Ign:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages
...
Ign:18 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages
Err:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages
Connection timed out [IP: 91.189.91.83 80]
Err:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages
Temporary failure resolving 'archive.ubuntu.com' [IP: 91.189.91.83 80]
Err:17 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages
Temporary failure resolving 'archive.ubuntu.com' [IP: 91.189.91.83 80]
Err:18 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages
Temporary failure resolving 'archive.ubuntu.com' [IP: 91.189.91.83 80]
Fetched 27.7 MB in 4min 34s (101 kB/s)
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-amd64/Packages Connection timed out [IP: 91.189.91.83 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-amd64/Packages Temporary failure resolving 'archive.ubuntu.com' [IP: 91.189.91.83 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/main/binary-amd64/Packages Temporary failure resolving 'archive.ubuntu.com' [IP: 91.189.91.83 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/universe/binary-amd64/Packages Temporary failure resolving 'archive.ubuntu.com' [IP: 91.189.91.83 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@729489dce213:/# echo $?
0
root@729489dce213:/# apt-get update -o APT::Update::Error-Mode=any
Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Could not resolve 'archive.ubuntu.com'
Err:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Could not resolve 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Could not resolve 'archive.ubuntu.com'
Reading package lists... Done
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease Could not resolve 'archive.ubuntu.com'
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease Could not resolve 'archive.ubuntu.com'
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease Could not resolve 'archive.ubuntu.com'
E: Some index files failed to download. They have been ignored, or old ones used instead.
root@729489dce213:/# echo $?
100
Ref: https://unix.stackexchange.com/a/653377
We could make this a good first issue/help wanted ticket. It seems easy enough for a new contributor.
Many thanks for the investigation @ivanvc - Agree this would be a good first issue.
For any contributor interested we need to update any github actions file that performs an apt install to add the optional configuration for strict error enforcement mentioned above -o APT::Update::Error-Mode=any.
Re-open as I just realised we need to make a second pr here: https://github.com/kubernetes/test-infra/blob/master/config/jobs/etcd/etcd-periodics.yaml#L84
@jimmy-bro Would you have time to do this second pr? Sorry for not making it clear from the outset.