coreos-assembler icon indicating copy to clipboard operation
coreos-assembler copied to clipboard
verified publisher icon coreos

gangplank: sometimes exits with incorrect exit code

Open dustymabe opened this issue 3 years ago • 1 comments

We've seen it happen several times recently where gangplank exits with the wrong exit code. There is a failure but then jenkins doesn't stop because it doesn't get the bad exit code.

There are two examples back to back that ran and failed in the same way, but one of them continued because (we think) gangplank returned a successful exit code, even though it reported that it was going to return a bad exit code.

  • build-arch#232 ✔️ - Properly detected exit code. Jenkins marked as red.
  • build-arch#230 ❌ - Improperly continued the run. Jenkins marked as green.

Here's the logs from build-arch#232✔️ :

[2022-04-29T14:46:16.857Z] === RUN   ext.config.kdump.crash
[2022-04-29T14:57:19.743Z] ��������������������������������--- FAIL: ext.config.kdump.crash (663.93s)
[2022-04-29T14:57:19.743Z]         harness.go:103: TIMEOUT[10m0s]: ssh: sudo ./kolet run-test-unit kola-runext.service
[2022-04-29T14:57:19.743Z]         harness.go:103: TIMEOUT[10m0s]: ssh: journalctl -t kola-runext-crash
[2022-04-29T14:57:19.743Z] FAIL, output in tmp/kola/rerun
[2022-04-29T14:57:19.743Z] Error: harness: test suite failed
[2022-04-29T14:57:19.743Z] 2022-04-29T14:57:18Z cli: harness: test suite failed
[2022-04-29T14:57:19.743Z] [36mINFO[0m[5632] Executing rendered script                     [36mcmd[0m=/bin/bash [36mrendered[0m=/tmp/rendered875699139
[2022-04-29T14:57:19.743Z] + cosa compress --compressor xz
[2022-04-29T14:57:19.743Z] Targeting build: 37.20220429.91.0
[2022-04-29T14:57:19.743Z] Compressing: builds/37.20220429.91.0/x86_64
[2022-04-29T14:57:19.743Z] No meta.json exists for builds/37.20220429.91.0/x86_64.. Skipping
[2022-04-29T14:57:19.743Z] All builds already compressed
[2022-04-29T14:57:19.743Z] Compressing: builds/37.20220429.91.0/aarch64
[2022-04-29T14:57:19.743Z] + xz -c9 -T0 builds/37.20220429.91.0/aarch64/fedora-coreos-37.20220429.91.0-qemu.aarch64.qcow2
[2022-04-29T15:00:28.123Z] ���������Compressed: fedora-coreos-37.20220429.91.0-qemu.aarch64.qcow2.xz
[2022-04-29T15:00:28.123Z] Skipped artifacts: ostree
[2022-04-29T15:00:28.123Z] Updated: builds/37.20220429.91.0/aarch64/meta.json
[2022-04-29T15:00:28.123Z] + tar --xz -cf tmp/kola.tar.xz tmp/kola-basic tmp/kola tmp/kola-metal tmp/kola-metal4k
[2022-04-29T15:01:16.911Z] ��tar: tmp/kola-metal: Cannot stat: No such file or directory
[2022-04-29T15:01:16.911Z] tar: tmp/kola-metal4k: Cannot stat: No such file or directory
[2022-04-29T15:01:16.911Z] tar: Exiting with failure status due to previous errors
[2022-04-29T15:01:16.911Z] + true
[2022-04-29T15:01:16.911Z] [36mINFO[0m[5869] Script complete                               [36mscript[0m=/tmp/stages028496320/post.sh
[2022-04-29T15:01:16.911Z] [31mERRO[0m[5869] failed stage execution                        [31mbuild artifacts[0m="[]" [31mcommands[0m="[cosa buildextend-qemu cosa kola run --rerun --basic-qemu-scenarios --output-dir tmp/kola-basic cosa kola run --rerun --parallel 4 --output-dir tmp/kola cosa buildextend-metal cosa buildextend-metal4k cosa buildextend-live kola testiso -S --output-dir tmp/kola-metal kola testiso -SP --qemu-native-4k --scenarios iso-install --output-dir tmp/kola-metal4k cosa buildextend-openstack echo 'ZGlmZiAtLWdpdCBhL3Vzci9saWIvY29yZW9zLWFzc2VtYmxlci9nZi1wbGF0Zm9ybWlkIGIvdXNyL2xpYi9jb3Jlb3MtYXNzZW1ibGVyL2dmLXBsYXRmb3JtaWQKaW5kZXggNDI5Y2ExYmViLi44MTIzNTk0NjkgMTAwNzU1Ci0tLSBhL3Vzci9saWIvY29yZW9zLWFzc2VtYmxlci9nZi1wbGF0Zm9ybWlkCisrKyBiL3Vzci9saWIvY29yZW9zLWFzc2VtYmxlci9nZi1wbGF0Zm9ybWlkCkBAIC00Niw3ICs0NiwxMSBAQCBibHNjZmdfcGF0aD0kKGNvcmVvc19nZiBnbG9iLWV4cGFuZCAvYm9vdC9sb2FkZXIvZW50cmllcy9vc3RyZWUtKi5jb25mKQogY29yZW9zX2dmIGRvd25sb2FkICIke2Jsc2NmZ19wYXRofSIgIiR7dG1wZH0iL2Jscy5jb25mCiAjIFJlbW92ZSBhbnkgcGxhdGZvcm1pZCBjdXJyZW50bHkgdGhlcmUKIHNlZCAtaSAtZSAncywgaWduaXRpb24ucGxhdGZvcm0uaWQ9W2EtekEtWjAtOV0qLCxnJyAiJHt0bXBkfSIvYmxzLmNvbmYKLXNlZCAtaSAtZSAnL15vcHRpb25zIC8gcywkLCBpZ25pdGlvbi5wbGF0Zm9ybS5pZD0nIiR7cGxhdGZvcm1pZH0iJywnICIke3RtcGR9Ii9ibHMuY29uZgoraWYgWyAiJHtwbGF0Zm9ybWlkfSIgPT0gJ2F3cycgXTsgdGhlbgorICAgIHNlZCAtaSAtZSAnc3xeXChvcHRpb25zIC4qXCl8XDEgaWduaXRpb24ucGxhdGZvcm0uaWQ9JyIke3BsYXRmb3JtaWR9IicgY29uc29sZT10dHlTMCwxMTUyMDBuOHwnICIke3RtcGR9Ii9ibHMuY29uZgorZWxzZQorICAgIHNlZCAtaSAtZSAnL15vcHRpb25zIC8gcywkLCBpZ25pdGlvbi5wbGF0Zm9ybS5pZD0nIiR7cGxhdGZvcm1pZH0iJywnICIke3RtcGR9Ii9ibHMuY29uZgorZmkKIGNvcmVvc19nZiB1cGxvYWQgIiR7dG1wZH0iL2Jscy5jb25mICIke2Jsc2NmZ19wYXRofSIKIAogaWYgWyAiJGJhc2VhcmNoIiA9ICJzMzkweCIgXSA7IHRoZW4K' | base64 --decode | sudo patch /usr/lib/coreos-assembler/gf-platformid cosa buildextend-aws]" [31merror[0m="Script exited with return code exit status 1" [31moptional artifacts[0m="[]" [31mrequired artifacts[0m="[ostree]" [31mreturn files[0m="[]" [31mstage id[0m="ExecOrder 1 Stage"
[2022-04-29T15:01:16.911Z] [36mINFO[0m[5869] Uploaded                                      [36mbucket[0m=builds [36metag[0m=af582a4f64815c8e2724b59bd9ecaf7e-2 [36mfrom[0m=/srv/tmp/kola.tar.xz [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651244474918749182 [36mobject[0m=37.20220429.91.0/aarch64/logs/kola.tar.xz [36mremote size[0m=22592260 [36msize[0m=22592260
[2022-04-29T15:01:16.911Z] [36mINFO[0m[5869] Uploaded                                      [36mbucket[0m=builds [36metag[0m=4e33f9e680a7af8365aabba39f3f39d6 [36mfrom[0m=/srv/builds/builds.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651244474954633371 [36mobject[0m=builds.json [36mremote size[0m=27266 [36msize[0m=27266
[2022-04-29T15:01:16.911Z] [36mINFO[0m[5869] Uploaded                                      [36mbucket[0m=builds [36metag[0m=c442b461b9dcbd9873473b78e6f4c85c [36mfrom[0m=/srv/builds/37.20220429.91.0/aarch64/commitmeta.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651244474988717258 [36mobject[0m=37.20220429.91.0/aarch64/commitmeta.json [36mremote size[0m=38527 [36msize[0m=38527
[2022-04-29T15:01:16.911Z] [36mINFO[0m[5869] Uploaded                                      [36mbucket[0m=builds [36metag[0m=b8251178891f8ea29b141a52b0eb7d80 [36mfrom[0m=/srv/builds/37.20220429.91.0/aarch64/coreos-assembler-config-git.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651244475023242074 [36mobject[0m=37.20220429.91.0/aarch64/coreos-assembler-config-git.json [36mremote size[0m=490 [36msize[0m=490
[2022-04-29T15:01:22.142Z] [36mINFO[0m[5875] Uploaded                                      [36mbucket[0m=builds [36metag[0m=36e57aafdf01d0629878895ec77997e3-40 [36mfrom[0m=/srv/builds/37.20220429.91.0/aarch64/fedora-coreos-37.20220429.91.0-qemu.aarch64.qcow2.xz [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651244480992649879 [36mobject[0m=37.20220429.91.0/aarch64/fedora-coreos-37.20220429.91.0-qemu.aarch64.qcow2.xz [36mremote size[0m=668747220 [36msize[0m=668747220
[2022-04-29T15:01:22.142Z] [36mINFO[0m[5875] Uploaded                                      [36mbucket[0m=builds [36metag[0m=402e4b5421f5b562dd582071aa9c441a [36mfrom[0m=/srv/builds/37.20220429.91.0/aarch64/manifest-lock.generated.aarch64.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651244481028451280 [36mobject[0m=37.20220429.91.0/aarch64/manifest-lock.generated.aarch64.json [36mremote size[0m=27021 [36msize[0m=27021
[2022-04-29T15:01:22.142Z] [36mINFO[0m[5875] Uploaded                                      [36mbucket[0m=builds [36metag[0m=aa7f45d0ff445a97733816e4ea8179c0 [36mfrom[0m=/srv/builds/37.20220429.91.0/aarch64/meta.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651244481069265386 [36mobject[0m=37.20220429.91.0/aarch64/meta.json [36mremote size[0m=8200 [36msize[0m=8200
[2022-04-29T15:01:22.142Z] [36mINFO[0m[5875] Uploaded                                      [36mbucket[0m=builds [36metag[0m=c4aeec53fcae5c606dff2d4a0e8afca5 [36mfrom[0m=/srv/builds/37.20220429.91.0/aarch64/ostree-commit-object [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651244481136951168 [36mobject[0m=37.20220429.91.0/aarch64/ostree-commit-object [36mremote size[0m=17820 [36msize[0m=17820
[2022-04-29T15:01:22.142Z] [36mINFO[0m[5875] Processed Uploads                             [36merror[0m="<nil>"
[2022-04-29T15:01:22.142Z] [31mFATA[0m[5875] Failed to prepare environment.                [31merr[0m="Script exited with return code exit status 1"
[2022-04-29T15:01:22.142Z] time="2022-04-29T15:01:21Z" level=debug msg="DoRequest Method: POST URI: http://d/v3.4.3/libpod/containers/27caf1fca3ac3963ad5ae81b00c00bc1e765c03be84a4e356906aad9fb875607/wait"
[2022-04-29T15:01:23.065Z] time="2022-04-29T15:01:22Z" level=debug msg="DoRequest Method: DELETE URI: http://d/v3.4.3/libpod/containers/27caf1fca3ac3963ad5ae81b00c00bc1e765c03be84a4e356906aad9fb875607"
[2022-04-29T15:01:24.239Z] �time="2022-04-29T15:01:23Z" level=debug msg="DoRequest Method: DELETE URI: http://d/v3.4.3/libpod/volumes/cosa-3f89a969-13a2-4399-95fa-9fb7e3ef14dd-worker-0"
[2022-04-29T15:01:27.485Z] time="2022-04-29T15:01:26Z" level=debug msg="execution done"
[2022-04-29T15:01:27.485Z] time="2022-04-29T15:01:26Z" level=error msg="Stage sent error" error="ExecOrder 1 Stage failed: work pod failed: return code 1"
[2022-04-29T15:01:27.485Z] time="2022-04-29T15:01:26Z" level=debug msg="done with execution group" execution group=1
[2022-04-29T15:01:27.485Z] time="2022-04-29T15:01:26Z" level=info msg="Shutting down ssh forwarding" port=22 remote host=**** remote user=builder
[2022-04-29T15:01:27.485Z] time="2022-04-29T15:01:27Z" level=fatal msg="failed to execute CI builder: process failed"
script returned exit code 1[2022-04-29T15:01:27.525Z] hudson.AbortException: script returned exit code 1
Error running gangplank.[2022-04-29T15:01:27.650Z] Archiving artifacts
[2022-04-29T15:01:29.635Z] + set -xeuo pipefail
[2022-04-29T15:01:29.635Z] ++ umask
[2022-04-29T15:01:29.635Z] + '[' 0000 = 0000 ']'
[2022-04-29T15:01:29.635Z] + umask 0022
[2022-04-29T15:01:29.635Z] + rm -rf builds/37.20220429.91.0/aarch64/logs
[2022-04-29T15:01:29.734Z] :fcos: :trashfire: [rawhide][aarch64] <https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build-arch/232/|232> - FAILURE (37.20220429.91.0)

And here they are from build-arch#230❌:

[2022-04-28T14:46:24.878Z] ======== Re-running failed tests (flake detection) ========
[2022-04-28T14:46:24.878Z] 
[2022-04-28T14:46:24.878Z] ������  Skipping kola test pattern "fcos.internet":
[2022-04-28T14:46:24.878Z]   ���� https://github.com/coreos/coreos-assembler/pull/1478
[2022-04-28T14:46:24.878Z] ������  Skipping kola test pattern "podman.workflow":
[2022-04-28T14:46:24.878Z]   ���� https://github.com/coreos/coreos-assembler/pull/1478
[2022-04-28T14:46:24.878Z] === RUN   ext.config.kdump.crash
[2022-04-28T14:57:29.708Z] ��������������������������������--- FAIL: ext.config.kdump.crash (663.57s)
[2022-04-28T14:57:29.708Z]         harness.go:103: TIMEOUT[10m0s]: ssh: sudo ./kolet run-test-unit kola-runext.service
[2022-04-28T14:57:29.708Z]         harness.go:103: TIMEOUT[10m0s]: ssh: journalctl -t kola-runext-crash
[2022-04-28T14:57:29.708Z] FAIL, output in tmp/kola/rerun
[2022-04-28T14:57:29.708Z] Error: harness: test suite failed
[2022-04-28T14:57:29.708Z] 2022-04-28T14:57:28Z cli: harness: test suite failed
[2022-04-28T14:57:29.708Z] [36mINFO[0m[5759] Executing rendered script                     [36mcmd[0m=/bin/bash [36mrendered[0m=/tmp/rendered550784625
[2022-04-28T14:57:29.708Z] + cosa compress --compressor xz
[2022-04-28T14:57:29.708Z] Targeting build: 37.20220428.91.0
[2022-04-28T14:57:29.708Z] Compressing: builds/37.20220428.91.0/x86_64
[2022-04-28T14:57:29.708Z] No meta.json exists for builds/37.20220428.91.0/x86_64.. Skipping
[2022-04-28T14:57:29.708Z] All builds already compressed
[2022-04-28T14:57:29.708Z] Compressing: builds/37.20220428.91.0/aarch64
[2022-04-28T14:57:29.708Z] + xz -c9 -T0 builds/37.20220428.91.0/aarch64/fedora-coreos-37.20220428.91.0-qemu.aarch64.qcow2
[2022-04-28T15:00:37.995Z] ���������Compressed: fedora-coreos-37.20220428.91.0-qemu.aarch64.qcow2.xz
[2022-04-28T15:00:37.995Z] Skipped artifacts: ostree
[2022-04-28T15:00:37.995Z] Updated: builds/37.20220428.91.0/aarch64/meta.json
[2022-04-28T15:00:37.995Z] + tar --xz -cf tmp/kola.tar.xz tmp/kola-basic tmp/kola tmp/kola-metal tmp/kola-metal4k
[2022-04-28T15:01:26.779Z] ��tar: tmp/kola-metal: Cannot stat: No such file or directory
[2022-04-28T15:01:26.779Z] tar: tmp/kola-metal4k: Cannot stat: No such file or directory
[2022-04-28T15:01:26.779Z] tar: Exiting with failure status due to previous errors
[2022-04-28T15:01:26.779Z] + true
[2022-04-28T15:01:26.779Z] [36mINFO[0m[5996] Script complete                               [36mscript[0m=/tmp/stages909226054/post.sh
[2022-04-28T15:01:26.779Z] [31mERRO[0m[5996] failed stage execution                        [31mbuild artifacts[0m="[]" [31mcommands[0m="[cosa buildextend-qemu cosa kola run --rerun --basic-qemu-scenarios --output-dir tmp/kola-basic cosa kola run --rerun --parallel 4 --output-dir tmp/kola cosa buildextend-metal cosa buildextend-metal4k cosa buildextend-live kola testiso -S --output-dir tmp/kola-metal kola testiso -SP --qemu-native-4k --scenarios iso-install --output-dir tmp/kola-metal4k cosa buildextend-openstack echo 'ZGlmZiAtLWdpdCBhL3Vzci9saWIvY29yZW9zLWFzc2VtYmxlci9nZi1wbGF0Zm9ybWlkIGIvdXNyL2xpYi9jb3Jlb3MtYXNzZW1ibGVyL2dmLXBsYXRmb3JtaWQKaW5kZXggNDI5Y2ExYmViLi44MTIzNTk0NjkgMTAwNzU1Ci0tLSBhL3Vzci9saWIvY29yZW9zLWFzc2VtYmxlci9nZi1wbGF0Zm9ybWlkCisrKyBiL3Vzci9saWIvY29yZW9zLWFzc2VtYmxlci9nZi1wbGF0Zm9ybWlkCkBAIC00Niw3ICs0NiwxMSBAQCBibHNjZmdfcGF0aD0kKGNvcmVvc19nZiBnbG9iLWV4cGFuZCAvYm9vdC9sb2FkZXIvZW50cmllcy9vc3RyZWUtKi5jb25mKQogY29yZW9zX2dmIGRvd25sb2FkICIke2Jsc2NmZ19wYXRofSIgIiR7dG1wZH0iL2Jscy5jb25mCiAjIFJlbW92ZSBhbnkgcGxhdGZvcm1pZCBjdXJyZW50bHkgdGhlcmUKIHNlZCAtaSAtZSAncywgaWduaXRpb24ucGxhdGZvcm0uaWQ9W2EtekEtWjAtOV0qLCxnJyAiJHt0bXBkfSIvYmxzLmNvbmYKLXNlZCAtaSAtZSAnL15vcHRpb25zIC8gcywkLCBpZ25pdGlvbi5wbGF0Zm9ybS5pZD0nIiR7cGxhdGZvcm1pZH0iJywnICIke3RtcGR9Ii9ibHMuY29uZgoraWYgWyAiJHtwbGF0Zm9ybWlkfSIgPT0gJ2F3cycgXTsgdGhlbgorICAgIHNlZCAtaSAtZSAnc3xeXChvcHRpb25zIC4qXCl8XDEgaWduaXRpb24ucGxhdGZvcm0uaWQ9JyIke3BsYXRmb3JtaWR9IicgY29uc29sZT10dHlTMCwxMTUyMDBuOHwnICIke3RtcGR9Ii9ibHMuY29uZgorZWxzZQorICAgIHNlZCAtaSAtZSAnL15vcHRpb25zIC8gcywkLCBpZ25pdGlvbi5wbGF0Zm9ybS5pZD0nIiR7cGxhdGZvcm1pZH0iJywnICIke3RtcGR9Ii9ibHMuY29uZgorZmkKIGNvcmVvc19nZiB1cGxvYWQgIiR7dG1wZH0iL2Jscy5jb25mICIke2Jsc2NmZ19wYXRofSIKIAogaWYgWyAiJGJhc2VhcmNoIiA9ICJzMzkweCIgXSA7IHRoZW4K' | base64 --decode | sudo patch /usr/lib/coreos-assembler/gf-platformid cosa buildextend-aws]" [31merror[0m="Script exited with return code exit status 1" [31moptional artifacts[0m="[]" [31mrequired artifacts[0m="[ostree]" [31mreturn files[0m="[]" [31mstage id[0m="ExecOrder 1 Stage"
[2022-04-28T15:01:26.779Z] [36mINFO[0m[5996] Uploaded                                      [36mbucket[0m=builds [36metag[0m=928bea9733e35807e429896d8a1cc0e7 [36mfrom[0m=/srv/builds/37.20220428.91.0/aarch64/meta.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651158085351343536 [36mobject[0m=37.20220428.91.0/aarch64/meta.json [36mremote size[0m=15139 [36msize[0m=15139
[2022-04-28T15:01:26.779Z] [36mINFO[0m[5996] Uploaded                                      [36mbucket[0m=builds [36metag[0m=85b18ed36c06985a3721dc32e30ca9a5 [36mfrom[0m=/srv/builds/37.20220428.91.0/aarch64/ostree-commit-object [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651158085404699613 [36mobject[0m=37.20220428.91.0/aarch64/ostree-commit-object [36mremote size[0m=17820 [36msize[0m=17820
[2022-04-28T15:01:26.779Z] [36mINFO[0m[5997] Uploaded                                      [36mbucket[0m=builds [36metag[0m=094a6a49dd3b46a40dbde028c3e46467-2 [36mfrom[0m=/srv/tmp/kola.tar.xz [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651158085732680485 [36mobject[0m=37.20220428.91.0/aarch64/logs/kola.tar.xz [36mremote size[0m=22657588 [36msize[0m=22657588
[2022-04-28T15:01:26.779Z] [36mINFO[0m[5997] Uploaded                                      [36mbucket[0m=builds [36metag[0m=7f87ce8aa89d3387d685569470e5acfc [36mfrom[0m=/srv/builds/builds.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651158085766263593 [36mobject[0m=builds.json [36mremote size[0m=27117 [36msize[0m=27117
[2022-04-28T15:01:26.779Z] [36mINFO[0m[5997] Uploaded                                      [36mbucket[0m=builds [36metag[0m=9c1c06be34a9bc4625e83fa815b161ac [36mfrom[0m=/srv/builds/37.20220428.91.0/aarch64/commitmeta.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651158085800115982 [36mobject[0m=37.20220428.91.0/aarch64/commitmeta.json [36mremote size[0m=38531 [36msize[0m=38531
[2022-04-28T15:01:26.779Z] [36mINFO[0m[5997] Uploaded                                      [36mbucket[0m=builds [36metag[0m=fe096454e2baa8c750a9162065b03273 [36mfrom[0m=/srv/builds/37.20220428.91.0/aarch64/coreos-assembler-config-git.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651158085834308955 [36mobject[0m=37.20220428.91.0/aarch64/coreos-assembler-config-git.json [36mremote size[0m=490 [36msize[0m=490
[2022-04-28T15:01:32.065Z] [36mINFO[0m[6002] Uploaded                                      [36mbucket[0m=builds [36metag[0m=f66064ac5c059cf364aea24c38c8ea85-40 [36mfrom[0m=/srv/builds/37.20220428.91.0/aarch64/fedora-coreos-37.20220428.91.0-qemu.aarch64.qcow2.xz [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651158091350331326 [36mobject[0m=37.20220428.91.0/aarch64/fedora-coreos-37.20220428.91.0-qemu.aarch64.qcow2.xz [36mremote size[0m=668972728 [36msize[0m=668972728
[2022-04-28T15:01:32.065Z] [36mINFO[0m[6002] Uploaded                                      [36mbucket[0m=builds [36metag[0m=b356c2e5109e83978814f023d4069afb [36mfrom[0m=/srv/builds/37.20220428.91.0/aarch64/manifest-lock.generated.aarch64.json [36mfunc[0m=putter [36mgangplank.coreos.com/cosa/stamp[0m=1651158091466628285 [36mobject[0m=37.20220428.91.0/aarch64/manifest-lock.generated.aarch64.json [36mremote size[0m=27025 [36msize[0m=27025
[2022-04-28T15:01:32.065Z] [36mINFO[0m[6002] Processed Uploads                             [36merror[0m="<nil>"
[2022-04-28T15:01:32.065Z] [31mFATA[0m[6002] Failed to prepare environment.                [31merr[0m="Script exited with return code exit status 1"
[2022-04-28T15:01:32.065Z] time="2022-04-28T15:01:32Z" level=debug msg="DoRequest Method: POST URI: http://d/v3.4.3/libpod/containers/b5415ab48a62726d1fb285bd64b3966101d96a1d5c1ddb617721c263010c95a6/wait"
[2022-04-28T15:01:33.565Z] �time="2022-04-28T15:01:33Z" level=debug msg="DoRequest Method: DELETE URI: http://d/v3.4.3/libpod/containers/b5415ab48a62726d1fb285bd64b3966101d96a1d5c1ddb617721c263010c95a6"
[2022-04-28T15:01:33.830Z] time="2022-04-28T15:01:33Z" level=debug msg="DoRequest Method: DELETE URI: http://d/v3.4.3/libpod/volumes/cosa-04e75eb7-4fc7-4ace-8e50-1ac9c26b2a96-worker-0"
[2022-04-28T15:01:36.393Z] time="2022-04-28T15:01:36Z" level=debug msg="execution done"
[2022-04-28T15:01:36.393Z] time="2022-04-28T15:01:36Z" level=debug msg="done with execution group" execution group=1
[2022-04-28T15:01:36.393Z] time="2022-04-28T15:01:36Z" level=error msg="Stage sent error" error="ExecOrder 1 Stage failed: work pod failed: return code 1"
[2022-04-28T15:01:36.393Z] time="2022-04-28T15:01:36Z" level=info msg="Shutting down ssh forwarding" port=22 remote host=**** remote user=builder
[2022-04-28T15:01:36.999Z] Gangplank command finished successfully!

The only real difference I see is the last few messages are out of order between the two. i.e. Stage sent error is before/after done with execution group.

We need to dig into this because we don't want false positives in our pipelines.

dustymabe avatar Apr 29 '22 19:04 dustymabe

Also looks like the bad one is missing "failed to execute CI builder: process failed"

dustymabe avatar Apr 29 '22 20:04 dustymabe