go
go copied to clipboard
x/build/cmd/release: buildlets inexplicably got stuck
The below image basically summarizes it:

Running releasebot -mode=prepare goX.YbetaN resulted in some builders getting stuck like this. Nothing in the output appears to indicate a problem (except for a darwin/amd64 issue, filed separately):
~ $ releasebot -mode=prepare go1.18beta2
2022/01/28 12:29:24 Loading data from log *maintner.netMutSource ...
HTTP: 12:29:24.983 + https://maintner.golang.org/logs
HTTP: 12:29:25.322 - /logs 200 OK (0.339s)
2022/01/28 12:29:25 Downloading 2519 bytes of https://maintner.golang.org/logs/128 ...
HTTP: 12:29:25.349 + https://maintner.golang.org/logs/128
HTTP: 12:29:25.401 - /128 206 Partial Content (0.052s)
2022/01/28 12:29:25 wrote /Users/mknyszek/Library/Caches/golang-maintner/0128.growing.mutlog
2022/01/28 12:29:57 Reloaded data from log *maintner.netMutSource.
HTTP: 12:29:57.196 + https://oauth2.googleapis.com/token
HTTP: 12:29:57.352 - /token 200 OK (0.156s)
2022/01/28 12:29:57 starting
2022/01/28 12:29:57 working in /Users/mknyszek/go-releasebot-work/go1.18beta2
2022/01/28 12:29:57 $ git fetch origin master
2022/01/28 12:29:58 $ git fetch origin master
2022/01/28 12:30:09 $ git clone --reference /Users/mknyszek/go-releasebot-work/go1.18beta2/gitmirror -b master https://go.googlesource.com/go /Users/mknyszek/go-releasebot-work/go1.18beta2/gitwork
2022/01/28 12:30:13 $ git codereview change relwork
2022/01/28 12:30:14 $ git config gc.auto 0
2022/01/28 12:30:14 an associated security release branch "master-security" does not exist; assuming it has been merged and deleted, so proceeding as usual
2022/01/28 12:30:14 $ git rev-parse go1.18beta2
2022/01/28 12:30:14 Release status issue title: "all: Go 1.18beta2 release status"
HTTP: 12:30:14.925 + https://api.github.com/repos/golang/go/issues?per_page=100&state=all
HTTP: 12:30:16.042 - /issues 200 OK (1.117s)
HTTP: 12:30:16.154 + https://api.github.com/repos/golang/go/issues
HTTP: 12:30:16.571 - /issues 201 Created (0.418s)
2022/01/28 12:30:16 Release status issue: https://golang.org/issue/50886
2022/01/28 12:30:16 $ go clean -modcache
2022/01/28 12:30:16 $ go install golang.org/x/build/cmd/release@latest
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-386
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 src
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-arm64
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-amd64-longtest
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-amd64-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-armv6l
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-armv6l -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target src -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 darwin-amd64
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-386
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 freebsd-386
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target freebsd-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-arm64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 darwin-arm64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-s390x
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-s390x -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-ppc64le
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-ppc64le -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-amd64-longtest
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 freebsd-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target freebsd-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:52 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-amd64-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:52 BUILDRELEASE go1.18beta2 linux-386-longtest
2022/01/28 12:30:52 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-386-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:31:48 release -target="src": build succeeded (after 0 retries)
2022/01/28 12:31:48 DONE BUILDRELEASE go1.18beta2 src
2022/01/28 12:35:34 release -target="linux-ppc64le": build succeeded (after 0 retries)
2022/01/28 12:35:34 DONE BUILDRELEASE go1.18beta2 linux-ppc64le
2022/01/28 12:36:17 release -target="linux-s390x": build succeeded (after 0 retries)
2022/01/28 12:36:17 DONE BUILDRELEASE go1.18beta2 linux-s390x
2022/01/28 12:40:25 release -target="darwin-amd64" did not produce expected output files [go1.18beta2.darwin-amd64.tar.gz]:
error from cmd/release binary = <nil>
output from cmd/release binary:
2022/01/28 12:30:53 darwin-amd64: Start.
2022/01/28 12:30:53 darwin-amd64: Creating buildlet.
2022/01/28 12:30:55 darwin-amd64: Pushing source to buildlet.
2022/01/28 12:31:03 darwin-amd64: Writing VERSION file.
2022/01/28 12:31:03 darwin-amd64: Cleaning goroot (pre-build).
2022/01/28 12:31:03 darwin-amd64: Building (make.bash only).
2022/01/28 12:33:02 darwin-amd64: Building race detector.
2022/01/28 12:33:21 darwin-amd64: Cleaning goroot (post-build).
2022/01/28 12:33:21 darwin-amd64: Pushing and running releaselet.
2022/01/28 12:33:22 darwin-amd64: Cleaning workdir.
2022/01/28 12:33:24 darwin-amd64: Downloading tarball.
2022/01/28 12:33:45 darwin-amd64: Wrote "/Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821/go1.18beta2.darwin-amd64.tar.gz.untested".
2022/01/28 12:33:45 darwin-amd64: Building (all.bash to ensure tests pass).
2022/01/28 12:40:25 darwin-amd64: Error: Build failed: exit status 1
Output:
Building Go cmd/dist using /Users/gopher/goboot. (go1.17.3 darwin/amd64)
Building Go toolchain1 using /Users/gopher/goboot.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
##### Testing packages.
ok archive/tar 0.300s
ok archive/zip 0.743s
ok bufio 0.303s
ok bytes 3.697s
ok compress/bzip2 0.335s
ok compress/flate 1.637s
ok compress/gzip 0.545s
ok compress/lzw 0.492s
ok compress/zlib 1.328s
ok constraints 0.298s
ok container/heap 0.173s
ok container/list 0.385s
ok container/ring 0.137s
ok context 0.184s
ok crypto 0.261s
ok crypto/aes 0.196s
ok crypto/cipher 0.340s
ok crypto/des 0.196s
ok crypto/dsa 0.482s
ok crypto/ecdsa 0.218s
ok crypto/ed25519 0.274s
ok crypto/ed25519/internal/edwards25519 8.250s
ok crypto/ed25519/internal/edwards25519/field 2.614s
ok crypto/elliptic 0.237s
ok crypto/elliptic/internal/fiat 0.181s [no tests to run]
ok crypto/elliptic/internal/nistec 1.356s
ok crypto/hmac 0.262s
ok crypto/internal/subtle 0.186s
ok crypto/md5 0.211s
ok crypto/rand 0.272s
ok crypto/rc4 0.241s
ok crypto/rsa 0.325s
ok crypto/sha1 0.246s
ok crypto/sha256 0.246s
ok crypto/sha512 0.185s
ok crypto/subtle 0.205s
ok crypto/tls 1.807s
ok crypto/x509 0.932s
ok database/sql 0.844s
ok database/sql/driver 0.195s
ok debug/buildinfo 0.160s
ok debug/dwarf 0.208s
ok debug/elf 0.192s
ok debug/gosym 0.194s
ok debug/macho 0.317s
ok debug/pe 0.383s
ok debug/plan9obj 0.568s
ok embed 0.291s [no tests to run]
ok embed/internal/embedtest 0.192s
ok encoding/ascii85 0.428s
ok encoding/asn1 0.250s
ok encoding/base32 0.367s
ok encoding/base64 0.394s
ok encoding/binary 0.170s
ok encoding/csv 0.179s
ok encoding/gob 0.272s
ok encoding/hex 0.235s
ok encoding/json 0.785s
ok encoding/pem 0.286s
ok encoding/xml 0.465s
ok errors 0.262s
ok expvar 0.304s
ok flag 0.266s
ok fmt 0.380s
ok go/ast 0.187s
ok go/build 3.269s
ok go/build/constraint 0.158s
ok go/constant 0.430s
ok go/doc 0.439s
ok go/format 0.348s
ok go/importer 0.350s
ok go/internal/gccgoimporter 0.185s
ok go/internal/gcimporter 11.260s
ok go/internal/srcimporter 15.459s
ok go/parser 0.290s
ok go/printer 0.408s
ok go/scanner 0.218s
ok go/token 0.182s
ok go/types 24.239s
ok hash 0.170s
ok hash/adler32 0.163s
ok hash/crc32 0.218s
ok hash/crc64 0.226s
ok hash/fnv 0.324s
ok hash/maphash 0.215s
ok html 0.208s
ok html/template 0.315s
ok image 0.224s
ok image/color 0.176s
ok image/draw 0.216s
ok image/gif 21.135s
ok image/jpeg 3.667s
ok image/png 0.336s
ok index/suffixarray 0.277s
ok internal/abi 0.233s
ok internal/buildcfg 0.148s
ok internal/cpu 0.219s
ok internal/execabs 0.148s
ok internal/fmtsort 0.191s
ok internal/fuzz 0.183s
ok internal/godebug 0.253s
ok internal/intern 0.672s
ok internal/itoa 0.207s
ok internal/poll 0.415s
ok internal/profile 0.144s
ok internal/reflectlite 0.412s
ok internal/singleflight 0.188s
ok internal/trace 0.216s
ok internal/unsafeheader 0.187s
ok internal/xcoff 0.270s
ok io 0.209s
ok io/fs 0.191s
ok io/ioutil 0.197s
ok log 0.199s
ok log/syslog 1.529s
ok math 0.182s
ok math/big 2.180s
ok math/bits 0.406s
ok math/cmplx 0.287s
ok math/rand 0.453s
ok mime 0.166s
ok mime/multipart 0.411s
ok mime/quotedprintable 0.148s
ok net 13.619s
ok net/http 9.803s
ok net/http/cgi 1.356s
ok net/http/cookiejar 0.229s
ok net/http/fcgi 0.495s
ok net/http/httptest 0.251s
ok net/http/httptrace 0.185s
ok net/http/httputil 0.828s
ok net/http/internal 0.158s
ok net/http/internal/ascii 0.156s
ok net/http/pprof 5.285s
ok net/internal/socktest 0.328s
ok net/mail 0.167s
ok net/netip 0.855s
ok net/rpc 0.220s
ok net/rpc/jsonrpc 0.171s
ok net/smtp 0.171s
ok net/textproto 0.160s
ok net/url 0.187s
ok os 2.097s
ok os/exec 0.959s
ok os/exec/internal/fdtest 0.218s
ok os/signal 2.536s
ok os/user 0.157s
ok path 0.184s
ok path/filepath 0.261s
ok plugin 0.341s
ok reflect 0.912s
ok regexp 0.323s
ok regexp/syntax 0.717s
ok runtime 47.848s
ok runtime/cgo 0.256s
ok runtime/debug 0.254s
ok runtime/internal/atomic 0.481s
ok runtime/internal/math 0.195s
ok runtime/internal/sys 0.507s
ok runtime/metrics 0.555s
ok runtime/pprof 11.277s
ok runtime/trace 1.926s
ok sort 0.260s
ok strconv 0.991s
ok strings 3.548s
ok sync 0.509s
ok sync/atomic 5.571s
ok syscall 1.570s
ok testing 1.638s
ok testing/fstest 0.154s
ok testing/iotest 0.174s
ok testing/quick 0.272s
ok text/scanner 0.160s
ok text/tabwriter 0.332s
ok text/template 0.406s
ok text/template/parse 0.171s
ok time 2.074s
ok unicode 0.197s
ok unicode/utf16 0.198s
ok unicode/utf8 0.186s
ok cmd/addr2line 2.633s
ok cmd/api 24.757s
ok cmd/asm/internal/asm 1.264s
ok cmd/asm/internal/lex 0.268s
ok cmd/compile/internal/amd64 11.869s
ok cmd/compile/internal/dwarfgen 0.838s
ok cmd/compile/internal/importer 6.028s
ok cmd/compile/internal/ir 0.195s
ok cmd/compile/internal/logopt 0.802s
ok cmd/compile/internal/noder 0.283s
ok cmd/compile/internal/ssa 47.879s
ok cmd/compile/internal/syntax 0.181s
ok cmd/compile/internal/test 42.507s
ok cmd/compile/internal/typecheck 1.149s
ok cmd/compile/internal/types 0.151s
ok cmd/compile/internal/types2 10.706s
ok cmd/cover 4.252s
ok cmd/dist 0.209s
ok cmd/doc 0.737s
ok cmd/fix 6.169s
go test proxy running at GOPROXY=http://127.0.0.1:51589/mod
--- FAIL: TestScript (0.03s)
--- FAIL: TestScript/cgo_stale_precompiled (0.38s)
script_test.go:257:
# Regression test for https://go.dev/issue/47215 and https://go.dev/issue/50183:
# A mismatched $GOROOT_FINAL or missing $CC caused the C dependencies of the net
# package to appear stale, and it could not be rebuilt due to a missing $CC. (0.000s)
# Control case: net must not already be stale. (0.374s)
> ! stale net
FAIL: testdata/script/cgo_stale_precompiled.txt:8: net is unexpectedly stale
FAIL
FAIL cmd/go 124.028s
ok cmd/go/internal/auth 0.182s
ok cmd/go/internal/cache 0.849s
ok cmd/go/internal/fsys 0.321s
ok cmd/go/internal/generate 0.296s
ok cmd/go/internal/get 0.487s
ok cmd/go/internal/imports 0.466s
ok cmd/go/internal/load 0.224s
ok cmd/go/internal/lockedfile 0.257s
ok cmd/go/internal/lockedfile/internal/filelock 0.208s
ok cmd/go/internal/modconv 0.463s
ok cmd/go/internal/modfetch 0.477s
ok cmd/go/internal/modfetch/codehost 0.457s
ok cmd/go/internal/modfetch/zip_sum_test 0.741s
ok cmd/go/internal/modload 0.405s
ok cmd/go/internal/mvs 0.200s
ok cmd/go/internal/par 0.303s
ok cmd/go/internal/search 0.180s
ok cmd/go/internal/str 0.252s
ok cmd/go/internal/test 0.408s
ok cmd/go/internal/vcs 0.297s
ok cmd/go/internal/web 0.607s
ok cmd/go/internal/work 0.396s
ok cmd/gofmt 0.279s
ok cmd/internal/archive 2.720s
ok cmd/internal/buildid 0.828s
ok cmd/internal/dwarf 0.273s
ok cmd/internal/edit 0.193s
ok cmd/internal/goobj 0.209s
ok cmd/internal/moddeps 8.080s
ok cmd/internal/obj 0.207s
ok cmd/internal/obj/ppc64 0.476s
ok cmd/internal/obj/riscv 0.240s
ok cmd/internal/obj/s390x 0.292s
ok cmd/internal/obj/x86 14.339s
ok cmd/internal/objabi 0.182s
ok cmd/internal/pkgpath 0.318s
ok cmd/internal/quoted 0.461s
ok cmd/internal/src 0.446s
ok cmd/internal/sys 0.337s
ok cmd/internal/test2json 0.463s
ok cmd/link 60.705s
ok cmd/link/internal/benchmark 0.655s
ok cmd/link/internal/ld 34.839s
ok cmd/link/internal/loader 0.248s
ok cmd/nm 9.810s
ok cmd/objdump 12.877s
ok cmd/pack 9.122s
ok cmd/pprof 9.377s
ok cmd/trace 0.463s
ok cmd/vet 19.342s
FAIL
go tool dist: Failed: exit status 1
2022/01/28 12:40:25 release -target="darwin-amd64": waiting a bit and trying again
2022/01/28 12:41:15 release -target="darwin-arm64": build succeeded (after 0 retries)
2022/01/28 12:41:15 DONE BUILDRELEASE go1.18beta2 darwin-arm64
2022/01/28 12:41:25 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:43:00 release -target="linux-arm64": build succeeded (after 0 retries)
2022/01/28 12:43:00 DONE BUILDRELEASE go1.18beta2 linux-arm64
2022/01/28 12:45:12 release -target="linux-armv6l": build succeeded (after 0 retries)
2022/01/28 12:45:12 DONE BUILDRELEASE go1.18beta2 linux-armv6l
2022/01/28 12:46:20 release -target="freebsd-amd64": build succeeded (after 0 retries)
2022/01/28 12:46:20 DONE BUILDRELEASE go1.18beta2 freebsd-amd64
2022/01/28 12:49:11 release -target="windows-arm64": build succeeded (after 0 retries)
2022/01/28 12:49:11 DONE BUILDRELEASE go1.18beta2 windows-arm64
2022/01/28 12:50:40 release -target="darwin-amd64" did not produce expected output files [go1.18beta2.darwin-amd64.tar.gz]:
error from cmd/release binary = <nil>
output from cmd/release binary:
2022/01/28 12:41:25 darwin-amd64: Start.
2022/01/28 12:41:25 darwin-amd64: Creating buildlet.
2022/01/28 12:41:26 darwin-amd64: Pushing source to buildlet.
2022/01/28 12:41:32 darwin-amd64: Writing VERSION file.
2022/01/28 12:41:32 darwin-amd64: Cleaning goroot (pre-build).
2022/01/28 12:41:32 darwin-amd64: Building (make.bash only).
2022/01/28 12:43:28 darwin-amd64: Building race detector.
2022/01/28 12:43:47 darwin-amd64: Cleaning goroot (post-build).
2022/01/28 12:43:47 darwin-amd64: Pushing and running releaselet.
2022/01/28 12:43:48 darwin-amd64: Cleaning workdir.
2022/01/28 12:43:50 darwin-amd64: Downloading tarball.
2022/01/28 12:44:22 darwin-amd64: Wrote "/Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821/go1.18beta2.darwin-amd64.tar.gz.untested".
2022/01/28 12:44:22 darwin-amd64: Building (all.bash to ensure tests pass).
2022/01/28 12:50:40 darwin-amd64: Error: Build failed: exit status 1
Output:
Building Go cmd/dist using /Users/gopher/goboot. (go1.17.3 darwin/amd64)
Building Go toolchain1 using /Users/gopher/goboot.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
##### Testing packages.
ok archive/tar 0.439s
ok archive/zip 0.764s
ok bufio 0.233s
ok bytes 3.091s
ok compress/bzip2 0.340s
ok compress/flate 1.985s
ok compress/gzip 0.616s
ok compress/lzw 0.241s
ok compress/zlib 1.280s
ok constraints 0.561s
ok container/heap 0.180s
ok container/list 0.180s
ok container/ring 0.339s
ok context 0.180s
ok crypto 0.431s
ok crypto/aes 0.184s
ok crypto/cipher 0.474s
ok crypto/des 0.189s
ok crypto/dsa 0.210s
ok crypto/ecdsa 0.240s
ok crypto/ed25519 0.213s
ok crypto/ed25519/internal/edwards25519 8.097s
ok crypto/ed25519/internal/edwards25519/field 2.873s
ok crypto/elliptic 0.166s
ok crypto/elliptic/internal/fiat 0.213s [no tests to run]
ok crypto/elliptic/internal/nistec 1.343s
ok crypto/hmac 0.352s
ok crypto/internal/subtle 0.308s
ok crypto/md5 0.216s
ok crypto/rand 0.295s
ok crypto/rc4 0.341s
ok crypto/rsa 0.413s
ok crypto/sha1 0.243s
ok crypto/sha256 0.175s
ok crypto/sha512 0.338s
ok crypto/subtle 0.181s
ok crypto/tls 1.756s
ok crypto/x509 0.936s
ok database/sql 0.809s
ok database/sql/driver 0.178s
ok debug/buildinfo 0.197s
ok debug/dwarf 0.180s
ok debug/elf 0.172s
ok debug/gosym 0.321s
ok debug/macho 0.233s
ok debug/pe 0.335s
ok debug/plan9obj 0.448s
ok embed 0.423s [no tests to run]
ok embed/internal/embedtest 0.225s
ok encoding/ascii85 0.236s
ok encoding/asn1 0.267s
ok encoding/base32 0.490s
ok encoding/base64 0.447s
ok encoding/binary 0.167s
ok encoding/csv 0.160s
ok encoding/gob 0.245s
ok encoding/hex 0.303s
ok encoding/json 0.723s
ok encoding/pem 0.208s
ok encoding/xml 0.309s
ok errors 0.147s
ok expvar 0.289s
ok flag 0.263s
ok fmt 0.340s
ok go/ast 0.263s
ok go/build 3.170s
ok go/build/constraint 0.189s
ok go/constant 0.392s
ok go/doc 0.460s
ok go/format 0.388s
ok go/importer 0.477s
ok go/internal/gccgoimporter 0.201s
ok go/internal/gcimporter 10.991s
ok go/internal/srcimporter 14.778s
ok go/parser 0.258s
ok go/printer 0.359s
ok go/scanner 0.168s
ok go/token 0.237s
ok go/types 23.112s
ok hash 0.145s
ok hash/adler32 0.247s
ok hash/crc32 0.220s
ok hash/crc64 0.183s
ok hash/fnv 0.171s
ok hash/maphash 0.231s
ok html 0.185s
ok html/template 0.257s
ok image 0.244s
ok image/color 0.225s
ok image/draw 0.250s
ok image/gif 21.088s
ok image/jpeg 3.566s
ok image/png 0.322s
ok index/suffixarray 0.280s
ok internal/abi 0.204s
ok internal/buildcfg 0.150s
ok internal/cpu 0.155s
ok internal/execabs 0.159s
ok internal/fmtsort 0.139s
ok internal/fuzz 0.172s
ok internal/godebug 0.151s
ok internal/intern 0.978s
ok internal/itoa 0.210s
ok internal/poll 0.289s
ok internal/profile 0.168s
ok internal/reflectlite 0.435s
ok internal/singleflight 0.181s
ok internal/trace 0.281s
ok internal/unsafeheader 0.162s
ok internal/xcoff 0.199s
ok io 0.170s
ok io/fs 0.156s
ok io/ioutil 0.253s
ok log 0.176s
ok log/syslog 1.397s
ok math 0.165s
ok math/big 2.063s
ok math/bits 0.183s
ok math/cmplx 0.201s
ok math/rand 0.444s
ok mime 0.175s
ok mime/multipart 0.415s
ok mime/quotedprintable 0.308s
ok net 11.038s
ok net/http 8.477s
ok net/http/cgi 1.274s
ok net/http/cookiejar 0.236s
ok net/http/fcgi 0.279s
ok net/http/httptest 0.332s
ok net/http/httptrace 0.293s
ok net/http/httputil 0.843s
ok net/http/internal 0.153s
ok net/http/internal/ascii 0.169s
ok net/http/pprof 5.306s
ok net/internal/socktest 0.167s
ok net/mail 0.162s
ok net/netip 0.913s
ok net/rpc 0.243s
ok net/rpc/jsonrpc 0.193s
ok net/smtp 0.277s
ok net/textproto 0.188s
ok net/url 0.191s
ok os 1.618s
ok os/exec 1.055s
ok os/exec/internal/fdtest 0.205s
ok os/signal 2.316s
ok os/user 0.151s
ok path 0.175s
ok path/filepath 0.244s
ok plugin 0.193s
ok reflect 0.597s
ok regexp 0.221s
ok regexp/syntax 0.745s
ok runtime 40.162s
ok runtime/cgo 0.312s
ok runtime/debug 0.236s
ok runtime/internal/atomic 0.269s
ok runtime/internal/math 0.287s
ok runtime/internal/sys 0.302s
ok runtime/metrics 0.168s
ok runtime/pprof 8.854s
ok runtime/trace 0.999s
ok sort 0.188s
ok strconv 0.767s
ok strings 2.575s
ok sync 0.463s
ok sync/atomic 3.157s
ok syscall 0.822s
ok testing 0.924s
ok testing/fstest 0.154s
ok testing/iotest 0.152s
ok testing/quick 0.177s
ok text/scanner 0.240s
ok text/tabwriter 0.180s
ok text/template 0.258s
ok text/template/parse 0.152s
ok time 2.054s
ok unicode 0.227s
ok unicode/utf16 0.154s
ok unicode/utf8 0.165s
ok cmd/addr2line 1.738s
ok cmd/api 22.207s
ok cmd/asm/internal/asm 0.946s
ok cmd/asm/internal/lex 0.145s
ok cmd/compile/internal/amd64 11.589s
ok cmd/compile/internal/dwarfgen 0.737s
ok cmd/compile/internal/importer 5.659s
ok cmd/compile/internal/ir 0.207s
ok cmd/compile/internal/logopt 0.595s
ok cmd/compile/internal/noder 0.210s
ok cmd/compile/internal/ssa 41.322s
ok cmd/compile/internal/syntax 0.214s
ok cmd/compile/internal/test 36.106s
ok cmd/compile/internal/typecheck 0.845s
ok cmd/compile/internal/types 0.158s
ok cmd/compile/internal/types2 8.383s
ok cmd/cover 4.041s
ok cmd/dist 0.210s
ok cmd/doc 0.559s
ok cmd/fix 5.303s
go test proxy running at GOPROXY=http://127.0.0.1:51540/mod
--- FAIL: TestScript (0.02s)
--- FAIL: TestScript/cgo_stale_precompiled (0.66s)
script_test.go:257:
# Regression test for https://go.dev/issue/47215 and https://go.dev/issue/50183:
# A mismatched $GOROOT_FINAL or missing $CC caused the C dependencies of the net
# package to appear stale, and it could not be rebuilt due to a missing $CC. (0.000s)
# Control case: net must not already be stale. (0.654s)
> ! stale net
FAIL: testdata/script/cgo_stale_precompiled.txt:8: net is unexpectedly stale
FAIL
FAIL cmd/go 117.938s
ok cmd/go/internal/auth 0.219s
ok cmd/go/internal/cache 0.785s
ok cmd/go/internal/fsys 0.396s
ok cmd/go/internal/generate 0.301s
ok cmd/go/internal/get 0.386s
ok cmd/go/internal/imports 0.191s
ok cmd/go/internal/load 0.440s
ok cmd/go/internal/lockedfile 0.264s
ok cmd/go/internal/lockedfile/internal/filelock 0.199s
ok cmd/go/internal/modconv 0.253s
ok cmd/go/internal/modfetch 0.358s
ok cmd/go/internal/modfetch/codehost 0.318s
ok cmd/go/internal/modfetch/zip_sum_test 0.602s
ok cmd/go/internal/modload 0.279s
ok cmd/go/internal/mvs 0.179s
ok cmd/go/internal/par 0.264s
ok cmd/go/internal/search 0.185s
ok cmd/go/internal/str 0.209s
ok cmd/go/internal/test 0.338s
ok cmd/go/internal/vcs 0.255s
ok cmd/go/internal/web 0.327s
ok cmd/go/internal/work 0.369s
ok cmd/gofmt 0.387s
ok cmd/internal/archive 2.357s
ok cmd/internal/buildid 0.714s
ok cmd/internal/dwarf 0.329s
ok cmd/internal/edit 0.171s
ok cmd/internal/goobj 0.157s
ok cmd/internal/moddeps 7.709s
ok cmd/internal/obj 0.175s
ok cmd/internal/obj/ppc64 0.412s
ok cmd/internal/obj/riscv 0.243s
ok cmd/internal/obj/s390x 0.377s
ok cmd/internal/obj/x86 13.578s
ok cmd/internal/objabi 0.185s
ok cmd/internal/pkgpath 0.270s
ok cmd/internal/quoted 0.378s
ok cmd/internal/src 0.230s
ok cmd/internal/sys 0.235s
ok cmd/internal/test2json 0.483s
ok cmd/link 52.501s
ok cmd/link/internal/benchmark 0.250s
ok cmd/link/internal/ld 32.176s
ok cmd/link/internal/loader 0.219s
ok cmd/nm 12.094s
ok cmd/objdump 10.416s
ok cmd/pack 6.588s
ok cmd/pprof 8.455s
ok cmd/trace 0.930s
ok cmd/vet 18.385s
FAIL
go tool dist: Failed: exit status 1
2022/01/28 12:50:40 release -target="darwin-amd64": waiting a bit and trying again
2022/01/28 12:51:10 release -target="linux-amd64": build succeeded (after 0 retries)
2022/01/28 12:51:10 DONE BUILDRELEASE go1.18beta2 linux-amd64
2022/01/28 12:51:13 release -target="freebsd-386": build succeeded (after 0 retries)
2022/01/28 12:51:13 DONE BUILDRELEASE go1.18beta2 freebsd-386
2022/01/28 12:51:40 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:52:22 release -target="linux-386": build succeeded (after 0 retries)
2022/01/28 12:52:22 DONE BUILDRELEASE go1.18beta2 linux-386
2022/01/28 12:57:21 release -target="linux-amd64-longtest": build succeeded (after 0 retries)
2022/01/28 12:57:21 DONE BUILDRELEASE go1.18beta2 linux-amd64-longtest
CC @dmitshur @toothrot
I think it's worth noting that the hangs are so long that this is unlikely to be a test deadlocking, because it should have timed out by now.
@golang/release
We may want to consider either adding an hour long timeout here, or waiting until these tasks are ported to relui.
Should be obsoleted by the watchdog checks in relui?