go icon indicating copy to clipboard operation
go copied to clipboard

x/build/cmd/release: buildlets inexplicably got stuck

Open mknyszek opened this issue 3 years ago • 5 comments

The below image basically summarizes it:

image

Running releasebot -mode=prepare goX.YbetaN resulted in some builders getting stuck like this. Nothing in the output appears to indicate a problem (except for a darwin/amd64 issue, filed separately):

~ $ releasebot -mode=prepare go1.18beta2
2022/01/28 12:29:24 Loading data from log *maintner.netMutSource ...
HTTP: 12:29:24.983 + https://maintner.golang.org/logs
HTTP: 12:29:25.322 - /logs 200 OK (0.339s)
2022/01/28 12:29:25 Downloading 2519 bytes of https://maintner.golang.org/logs/128 ...
HTTP: 12:29:25.349  + https://maintner.golang.org/logs/128
HTTP: 12:29:25.401  - /128 206 Partial Content (0.052s)
2022/01/28 12:29:25 wrote /Users/mknyszek/Library/Caches/golang-maintner/0128.growing.mutlog
2022/01/28 12:29:57 Reloaded data from log *maintner.netMutSource.
HTTP: 12:29:57.196   + https://oauth2.googleapis.com/token
HTTP: 12:29:57.352   - /token 200 OK (0.156s)
2022/01/28 12:29:57 starting
2022/01/28 12:29:57 working in /Users/mknyszek/go-releasebot-work/go1.18beta2
2022/01/28 12:29:57 $ git fetch origin master
2022/01/28 12:29:58 $ git fetch origin master
2022/01/28 12:30:09 $ git clone --reference /Users/mknyszek/go-releasebot-work/go1.18beta2/gitmirror -b master https://go.googlesource.com/go /Users/mknyszek/go-releasebot-work/go1.18beta2/gitwork
2022/01/28 12:30:13 $ git codereview change relwork
2022/01/28 12:30:14 $ git config gc.auto 0
2022/01/28 12:30:14 an associated security release branch "master-security" does not exist; assuming it has been merged and deleted, so proceeding as usual
2022/01/28 12:30:14 $ git rev-parse go1.18beta2
2022/01/28 12:30:14 Release status issue title: "all: Go 1.18beta2 release status"
HTTP: 12:30:14.925    + https://api.github.com/repos/golang/go/issues?per_page=100&state=all
HTTP: 12:30:16.042    - /issues 200 OK (1.117s)
HTTP: 12:30:16.154 + https://api.github.com/repos/golang/go/issues
HTTP: 12:30:16.571 - /issues 201 Created (0.418s)
2022/01/28 12:30:16 Release status issue: https://golang.org/issue/50886
2022/01/28 12:30:16 $ go clean -modcache
2022/01/28 12:30:16 $ go install golang.org/x/build/cmd/release@latest
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-386
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 src
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-arm64
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-amd64-longtest
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-amd64-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-armv6l
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-armv6l -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target src -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 darwin-amd64
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-386
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 freebsd-386
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target freebsd-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-arm64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 darwin-arm64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-s390x
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-s390x -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-ppc64le
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-ppc64le -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-amd64-longtest
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 freebsd-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target freebsd-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:52 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-amd64-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:52 BUILDRELEASE go1.18beta2 linux-386-longtest
2022/01/28 12:30:52 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-386-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:31:48 release -target="src": build succeeded (after 0 retries)
2022/01/28 12:31:48 DONE BUILDRELEASE go1.18beta2 src
2022/01/28 12:35:34 release -target="linux-ppc64le": build succeeded (after 0 retries)
2022/01/28 12:35:34 DONE BUILDRELEASE go1.18beta2 linux-ppc64le
2022/01/28 12:36:17 release -target="linux-s390x": build succeeded (after 0 retries)
2022/01/28 12:36:17 DONE BUILDRELEASE go1.18beta2 linux-s390x
2022/01/28 12:40:25 release -target="darwin-amd64" did not produce expected output files [go1.18beta2.darwin-amd64.tar.gz]:
error from cmd/release binary = <nil>
output from cmd/release binary:
2022/01/28 12:30:53 darwin-amd64: Start.
2022/01/28 12:30:53 darwin-amd64: Creating buildlet.
2022/01/28 12:30:55 darwin-amd64: Pushing source to buildlet.
2022/01/28 12:31:03 darwin-amd64: Writing VERSION file.
2022/01/28 12:31:03 darwin-amd64: Cleaning goroot (pre-build).
2022/01/28 12:31:03 darwin-amd64: Building (make.bash only).
2022/01/28 12:33:02 darwin-amd64: Building race detector.
2022/01/28 12:33:21 darwin-amd64: Cleaning goroot (post-build).
2022/01/28 12:33:21 darwin-amd64: Pushing and running releaselet.
2022/01/28 12:33:22 darwin-amd64: Cleaning workdir.
2022/01/28 12:33:24 darwin-amd64: Downloading tarball.
2022/01/28 12:33:45 darwin-amd64: Wrote "/Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821/go1.18beta2.darwin-amd64.tar.gz.untested".
2022/01/28 12:33:45 darwin-amd64: Building (all.bash to ensure tests pass).
2022/01/28 12:40:25 darwin-amd64: Error: Build failed: exit status 1
Output:
Building Go cmd/dist using /Users/gopher/goboot. (go1.17.3 darwin/amd64)
Building Go toolchain1 using /Users/gopher/goboot.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.

##### Testing packages.
ok  	archive/tar	0.300s
ok  	archive/zip	0.743s
ok  	bufio	0.303s
ok  	bytes	3.697s
ok  	compress/bzip2	0.335s
ok  	compress/flate	1.637s
ok  	compress/gzip	0.545s
ok  	compress/lzw	0.492s
ok  	compress/zlib	1.328s
ok  	constraints	0.298s
ok  	container/heap	0.173s
ok  	container/list	0.385s
ok  	container/ring	0.137s
ok  	context	0.184s
ok  	crypto	0.261s
ok  	crypto/aes	0.196s
ok  	crypto/cipher	0.340s
ok  	crypto/des	0.196s
ok  	crypto/dsa	0.482s
ok  	crypto/ecdsa	0.218s
ok  	crypto/ed25519	0.274s
ok  	crypto/ed25519/internal/edwards25519	8.250s
ok  	crypto/ed25519/internal/edwards25519/field	2.614s
ok  	crypto/elliptic	0.237s
ok  	crypto/elliptic/internal/fiat	0.181s [no tests to run]
ok  	crypto/elliptic/internal/nistec	1.356s
ok  	crypto/hmac	0.262s
ok  	crypto/internal/subtle	0.186s
ok  	crypto/md5	0.211s
ok  	crypto/rand	0.272s
ok  	crypto/rc4	0.241s
ok  	crypto/rsa	0.325s
ok  	crypto/sha1	0.246s
ok  	crypto/sha256	0.246s
ok  	crypto/sha512	0.185s
ok  	crypto/subtle	0.205s
ok  	crypto/tls	1.807s
ok  	crypto/x509	0.932s
ok  	database/sql	0.844s
ok  	database/sql/driver	0.195s
ok  	debug/buildinfo	0.160s
ok  	debug/dwarf	0.208s
ok  	debug/elf	0.192s
ok  	debug/gosym	0.194s
ok  	debug/macho	0.317s
ok  	debug/pe	0.383s
ok  	debug/plan9obj	0.568s
ok  	embed	0.291s [no tests to run]
ok  	embed/internal/embedtest	0.192s
ok  	encoding/ascii85	0.428s
ok  	encoding/asn1	0.250s
ok  	encoding/base32	0.367s
ok  	encoding/base64	0.394s
ok  	encoding/binary	0.170s
ok  	encoding/csv	0.179s
ok  	encoding/gob	0.272s
ok  	encoding/hex	0.235s
ok  	encoding/json	0.785s
ok  	encoding/pem	0.286s
ok  	encoding/xml	0.465s
ok  	errors	0.262s
ok  	expvar	0.304s
ok  	flag	0.266s
ok  	fmt	0.380s
ok  	go/ast	0.187s
ok  	go/build	3.269s
ok  	go/build/constraint	0.158s
ok  	go/constant	0.430s
ok  	go/doc	0.439s
ok  	go/format	0.348s
ok  	go/importer	0.350s
ok  	go/internal/gccgoimporter	0.185s
ok  	go/internal/gcimporter	11.260s
ok  	go/internal/srcimporter	15.459s
ok  	go/parser	0.290s
ok  	go/printer	0.408s
ok  	go/scanner	0.218s
ok  	go/token	0.182s
ok  	go/types	24.239s
ok  	hash	0.170s
ok  	hash/adler32	0.163s
ok  	hash/crc32	0.218s
ok  	hash/crc64	0.226s
ok  	hash/fnv	0.324s
ok  	hash/maphash	0.215s
ok  	html	0.208s
ok  	html/template	0.315s
ok  	image	0.224s
ok  	image/color	0.176s
ok  	image/draw	0.216s
ok  	image/gif	21.135s
ok  	image/jpeg	3.667s
ok  	image/png	0.336s
ok  	index/suffixarray	0.277s
ok  	internal/abi	0.233s
ok  	internal/buildcfg	0.148s
ok  	internal/cpu	0.219s
ok  	internal/execabs	0.148s
ok  	internal/fmtsort	0.191s
ok  	internal/fuzz	0.183s
ok  	internal/godebug	0.253s
ok  	internal/intern	0.672s
ok  	internal/itoa	0.207s
ok  	internal/poll	0.415s
ok  	internal/profile	0.144s
ok  	internal/reflectlite	0.412s
ok  	internal/singleflight	0.188s
ok  	internal/trace	0.216s
ok  	internal/unsafeheader	0.187s
ok  	internal/xcoff	0.270s
ok  	io	0.209s
ok  	io/fs	0.191s
ok  	io/ioutil	0.197s
ok  	log	0.199s
ok  	log/syslog	1.529s
ok  	math	0.182s
ok  	math/big	2.180s
ok  	math/bits	0.406s
ok  	math/cmplx	0.287s
ok  	math/rand	0.453s
ok  	mime	0.166s
ok  	mime/multipart	0.411s
ok  	mime/quotedprintable	0.148s
ok  	net	13.619s
ok  	net/http	9.803s
ok  	net/http/cgi	1.356s
ok  	net/http/cookiejar	0.229s
ok  	net/http/fcgi	0.495s
ok  	net/http/httptest	0.251s
ok  	net/http/httptrace	0.185s
ok  	net/http/httputil	0.828s
ok  	net/http/internal	0.158s
ok  	net/http/internal/ascii	0.156s
ok  	net/http/pprof	5.285s
ok  	net/internal/socktest	0.328s
ok  	net/mail	0.167s
ok  	net/netip	0.855s
ok  	net/rpc	0.220s
ok  	net/rpc/jsonrpc	0.171s
ok  	net/smtp	0.171s
ok  	net/textproto	0.160s
ok  	net/url	0.187s
ok  	os	2.097s
ok  	os/exec	0.959s
ok  	os/exec/internal/fdtest	0.218s
ok  	os/signal	2.536s
ok  	os/user	0.157s
ok  	path	0.184s
ok  	path/filepath	0.261s
ok  	plugin	0.341s
ok  	reflect	0.912s
ok  	regexp	0.323s
ok  	regexp/syntax	0.717s
ok  	runtime	47.848s
ok  	runtime/cgo	0.256s
ok  	runtime/debug	0.254s
ok  	runtime/internal/atomic	0.481s
ok  	runtime/internal/math	0.195s
ok  	runtime/internal/sys	0.507s
ok  	runtime/metrics	0.555s
ok  	runtime/pprof	11.277s
ok  	runtime/trace	1.926s
ok  	sort	0.260s
ok  	strconv	0.991s
ok  	strings	3.548s
ok  	sync	0.509s
ok  	sync/atomic	5.571s
ok  	syscall	1.570s
ok  	testing	1.638s
ok  	testing/fstest	0.154s
ok  	testing/iotest	0.174s
ok  	testing/quick	0.272s
ok  	text/scanner	0.160s
ok  	text/tabwriter	0.332s
ok  	text/template	0.406s
ok  	text/template/parse	0.171s
ok  	time	2.074s
ok  	unicode	0.197s
ok  	unicode/utf16	0.198s
ok  	unicode/utf8	0.186s
ok  	cmd/addr2line	2.633s
ok  	cmd/api	24.757s
ok  	cmd/asm/internal/asm	1.264s
ok  	cmd/asm/internal/lex	0.268s
ok  	cmd/compile/internal/amd64	11.869s
ok  	cmd/compile/internal/dwarfgen	0.838s
ok  	cmd/compile/internal/importer	6.028s
ok  	cmd/compile/internal/ir	0.195s
ok  	cmd/compile/internal/logopt	0.802s
ok  	cmd/compile/internal/noder	0.283s
ok  	cmd/compile/internal/ssa	47.879s
ok  	cmd/compile/internal/syntax	0.181s
ok  	cmd/compile/internal/test	42.507s
ok  	cmd/compile/internal/typecheck	1.149s
ok  	cmd/compile/internal/types	0.151s
ok  	cmd/compile/internal/types2	10.706s
ok  	cmd/cover	4.252s
ok  	cmd/dist	0.209s
ok  	cmd/doc	0.737s
ok  	cmd/fix	6.169s
go test proxy running at GOPROXY=http://127.0.0.1:51589/mod
--- FAIL: TestScript (0.03s)
    --- FAIL: TestScript/cgo_stale_precompiled (0.38s)
        script_test.go:257:
            # Regression test for https://go.dev/issue/47215 and https://go.dev/issue/50183:
            # A mismatched $GOROOT_FINAL or missing $CC caused the C dependencies of the net
            # package to appear stale, and it could not be rebuilt due to a missing $CC. (0.000s)
            # Control case: net must not already be stale. (0.374s)
            > ! stale net
            FAIL: testdata/script/cgo_stale_precompiled.txt:8: net is unexpectedly stale


FAIL
FAIL	cmd/go	124.028s
ok  	cmd/go/internal/auth	0.182s
ok  	cmd/go/internal/cache	0.849s
ok  	cmd/go/internal/fsys	0.321s
ok  	cmd/go/internal/generate	0.296s
ok  	cmd/go/internal/get	0.487s
ok  	cmd/go/internal/imports	0.466s
ok  	cmd/go/internal/load	0.224s
ok  	cmd/go/internal/lockedfile	0.257s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.208s
ok  	cmd/go/internal/modconv	0.463s
ok  	cmd/go/internal/modfetch	0.477s
ok  	cmd/go/internal/modfetch/codehost	0.457s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.741s
ok  	cmd/go/internal/modload	0.405s
ok  	cmd/go/internal/mvs	0.200s
ok  	cmd/go/internal/par	0.303s
ok  	cmd/go/internal/search	0.180s
ok  	cmd/go/internal/str	0.252s
ok  	cmd/go/internal/test	0.408s
ok  	cmd/go/internal/vcs	0.297s
ok  	cmd/go/internal/web	0.607s
ok  	cmd/go/internal/work	0.396s
ok  	cmd/gofmt	0.279s
ok  	cmd/internal/archive	2.720s
ok  	cmd/internal/buildid	0.828s
ok  	cmd/internal/dwarf	0.273s
ok  	cmd/internal/edit	0.193s
ok  	cmd/internal/goobj	0.209s
ok  	cmd/internal/moddeps	8.080s
ok  	cmd/internal/obj	0.207s
ok  	cmd/internal/obj/ppc64	0.476s
ok  	cmd/internal/obj/riscv	0.240s
ok  	cmd/internal/obj/s390x	0.292s
ok  	cmd/internal/obj/x86	14.339s
ok  	cmd/internal/objabi	0.182s
ok  	cmd/internal/pkgpath	0.318s
ok  	cmd/internal/quoted	0.461s
ok  	cmd/internal/src	0.446s
ok  	cmd/internal/sys	0.337s
ok  	cmd/internal/test2json	0.463s
ok  	cmd/link	60.705s
ok  	cmd/link/internal/benchmark	0.655s
ok  	cmd/link/internal/ld	34.839s
ok  	cmd/link/internal/loader	0.248s
ok  	cmd/nm	9.810s
ok  	cmd/objdump	12.877s
ok  	cmd/pack	9.122s
ok  	cmd/pprof	9.377s
ok  	cmd/trace	0.463s
ok  	cmd/vet	19.342s
FAIL
go tool dist: Failed: exit status 1
2022/01/28 12:40:25 release -target="darwin-amd64": waiting a bit and trying again
2022/01/28 12:41:15 release -target="darwin-arm64": build succeeded (after 0 retries)
2022/01/28 12:41:15 DONE BUILDRELEASE go1.18beta2 darwin-arm64
2022/01/28 12:41:25 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:43:00 release -target="linux-arm64": build succeeded (after 0 retries)
2022/01/28 12:43:00 DONE BUILDRELEASE go1.18beta2 linux-arm64
2022/01/28 12:45:12 release -target="linux-armv6l": build succeeded (after 0 retries)
2022/01/28 12:45:12 DONE BUILDRELEASE go1.18beta2 linux-armv6l
2022/01/28 12:46:20 release -target="freebsd-amd64": build succeeded (after 0 retries)
2022/01/28 12:46:20 DONE BUILDRELEASE go1.18beta2 freebsd-amd64
2022/01/28 12:49:11 release -target="windows-arm64": build succeeded (after 0 retries)
2022/01/28 12:49:11 DONE BUILDRELEASE go1.18beta2 windows-arm64
2022/01/28 12:50:40 release -target="darwin-amd64" did not produce expected output files [go1.18beta2.darwin-amd64.tar.gz]:
error from cmd/release binary = <nil>
output from cmd/release binary:
2022/01/28 12:41:25 darwin-amd64: Start.
2022/01/28 12:41:25 darwin-amd64: Creating buildlet.
2022/01/28 12:41:26 darwin-amd64: Pushing source to buildlet.
2022/01/28 12:41:32 darwin-amd64: Writing VERSION file.
2022/01/28 12:41:32 darwin-amd64: Cleaning goroot (pre-build).
2022/01/28 12:41:32 darwin-amd64: Building (make.bash only).
2022/01/28 12:43:28 darwin-amd64: Building race detector.
2022/01/28 12:43:47 darwin-amd64: Cleaning goroot (post-build).
2022/01/28 12:43:47 darwin-amd64: Pushing and running releaselet.
2022/01/28 12:43:48 darwin-amd64: Cleaning workdir.
2022/01/28 12:43:50 darwin-amd64: Downloading tarball.
2022/01/28 12:44:22 darwin-amd64: Wrote "/Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821/go1.18beta2.darwin-amd64.tar.gz.untested".
2022/01/28 12:44:22 darwin-amd64: Building (all.bash to ensure tests pass).
2022/01/28 12:50:40 darwin-amd64: Error: Build failed: exit status 1
Output:
Building Go cmd/dist using /Users/gopher/goboot. (go1.17.3 darwin/amd64)
Building Go toolchain1 using /Users/gopher/goboot.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.

##### Testing packages.
ok  	archive/tar	0.439s
ok  	archive/zip	0.764s
ok  	bufio	0.233s
ok  	bytes	3.091s
ok  	compress/bzip2	0.340s
ok  	compress/flate	1.985s
ok  	compress/gzip	0.616s
ok  	compress/lzw	0.241s
ok  	compress/zlib	1.280s
ok  	constraints	0.561s
ok  	container/heap	0.180s
ok  	container/list	0.180s
ok  	container/ring	0.339s
ok  	context	0.180s
ok  	crypto	0.431s
ok  	crypto/aes	0.184s
ok  	crypto/cipher	0.474s
ok  	crypto/des	0.189s
ok  	crypto/dsa	0.210s
ok  	crypto/ecdsa	0.240s
ok  	crypto/ed25519	0.213s
ok  	crypto/ed25519/internal/edwards25519	8.097s
ok  	crypto/ed25519/internal/edwards25519/field	2.873s
ok  	crypto/elliptic	0.166s
ok  	crypto/elliptic/internal/fiat	0.213s [no tests to run]
ok  	crypto/elliptic/internal/nistec	1.343s
ok  	crypto/hmac	0.352s
ok  	crypto/internal/subtle	0.308s
ok  	crypto/md5	0.216s
ok  	crypto/rand	0.295s
ok  	crypto/rc4	0.341s
ok  	crypto/rsa	0.413s
ok  	crypto/sha1	0.243s
ok  	crypto/sha256	0.175s
ok  	crypto/sha512	0.338s
ok  	crypto/subtle	0.181s
ok  	crypto/tls	1.756s
ok  	crypto/x509	0.936s
ok  	database/sql	0.809s
ok  	database/sql/driver	0.178s
ok  	debug/buildinfo	0.197s
ok  	debug/dwarf	0.180s
ok  	debug/elf	0.172s
ok  	debug/gosym	0.321s
ok  	debug/macho	0.233s
ok  	debug/pe	0.335s
ok  	debug/plan9obj	0.448s
ok  	embed	0.423s [no tests to run]
ok  	embed/internal/embedtest	0.225s
ok  	encoding/ascii85	0.236s
ok  	encoding/asn1	0.267s
ok  	encoding/base32	0.490s
ok  	encoding/base64	0.447s
ok  	encoding/binary	0.167s
ok  	encoding/csv	0.160s
ok  	encoding/gob	0.245s
ok  	encoding/hex	0.303s
ok  	encoding/json	0.723s
ok  	encoding/pem	0.208s
ok  	encoding/xml	0.309s
ok  	errors	0.147s
ok  	expvar	0.289s
ok  	flag	0.263s
ok  	fmt	0.340s
ok  	go/ast	0.263s
ok  	go/build	3.170s
ok  	go/build/constraint	0.189s
ok  	go/constant	0.392s
ok  	go/doc	0.460s
ok  	go/format	0.388s
ok  	go/importer	0.477s
ok  	go/internal/gccgoimporter	0.201s
ok  	go/internal/gcimporter	10.991s
ok  	go/internal/srcimporter	14.778s
ok  	go/parser	0.258s
ok  	go/printer	0.359s
ok  	go/scanner	0.168s
ok  	go/token	0.237s
ok  	go/types	23.112s
ok  	hash	0.145s
ok  	hash/adler32	0.247s
ok  	hash/crc32	0.220s
ok  	hash/crc64	0.183s
ok  	hash/fnv	0.171s
ok  	hash/maphash	0.231s
ok  	html	0.185s
ok  	html/template	0.257s
ok  	image	0.244s
ok  	image/color	0.225s
ok  	image/draw	0.250s
ok  	image/gif	21.088s
ok  	image/jpeg	3.566s
ok  	image/png	0.322s
ok  	index/suffixarray	0.280s
ok  	internal/abi	0.204s
ok  	internal/buildcfg	0.150s
ok  	internal/cpu	0.155s
ok  	internal/execabs	0.159s
ok  	internal/fmtsort	0.139s
ok  	internal/fuzz	0.172s
ok  	internal/godebug	0.151s
ok  	internal/intern	0.978s
ok  	internal/itoa	0.210s
ok  	internal/poll	0.289s
ok  	internal/profile	0.168s
ok  	internal/reflectlite	0.435s
ok  	internal/singleflight	0.181s
ok  	internal/trace	0.281s
ok  	internal/unsafeheader	0.162s
ok  	internal/xcoff	0.199s
ok  	io	0.170s
ok  	io/fs	0.156s
ok  	io/ioutil	0.253s
ok  	log	0.176s
ok  	log/syslog	1.397s
ok  	math	0.165s
ok  	math/big	2.063s
ok  	math/bits	0.183s
ok  	math/cmplx	0.201s
ok  	math/rand	0.444s
ok  	mime	0.175s
ok  	mime/multipart	0.415s
ok  	mime/quotedprintable	0.308s
ok  	net	11.038s
ok  	net/http	8.477s
ok  	net/http/cgi	1.274s
ok  	net/http/cookiejar	0.236s
ok  	net/http/fcgi	0.279s
ok  	net/http/httptest	0.332s
ok  	net/http/httptrace	0.293s
ok  	net/http/httputil	0.843s
ok  	net/http/internal	0.153s
ok  	net/http/internal/ascii	0.169s
ok  	net/http/pprof	5.306s
ok  	net/internal/socktest	0.167s
ok  	net/mail	0.162s
ok  	net/netip	0.913s
ok  	net/rpc	0.243s
ok  	net/rpc/jsonrpc	0.193s
ok  	net/smtp	0.277s
ok  	net/textproto	0.188s
ok  	net/url	0.191s
ok  	os	1.618s
ok  	os/exec	1.055s
ok  	os/exec/internal/fdtest	0.205s
ok  	os/signal	2.316s
ok  	os/user	0.151s
ok  	path	0.175s
ok  	path/filepath	0.244s
ok  	plugin	0.193s
ok  	reflect	0.597s
ok  	regexp	0.221s
ok  	regexp/syntax	0.745s
ok  	runtime	40.162s
ok  	runtime/cgo	0.312s
ok  	runtime/debug	0.236s
ok  	runtime/internal/atomic	0.269s
ok  	runtime/internal/math	0.287s
ok  	runtime/internal/sys	0.302s
ok  	runtime/metrics	0.168s
ok  	runtime/pprof	8.854s
ok  	runtime/trace	0.999s
ok  	sort	0.188s
ok  	strconv	0.767s
ok  	strings	2.575s
ok  	sync	0.463s
ok  	sync/atomic	3.157s
ok  	syscall	0.822s
ok  	testing	0.924s
ok  	testing/fstest	0.154s
ok  	testing/iotest	0.152s
ok  	testing/quick	0.177s
ok  	text/scanner	0.240s
ok  	text/tabwriter	0.180s
ok  	text/template	0.258s
ok  	text/template/parse	0.152s
ok  	time	2.054s
ok  	unicode	0.227s
ok  	unicode/utf16	0.154s
ok  	unicode/utf8	0.165s
ok  	cmd/addr2line	1.738s
ok  	cmd/api	22.207s
ok  	cmd/asm/internal/asm	0.946s
ok  	cmd/asm/internal/lex	0.145s
ok  	cmd/compile/internal/amd64	11.589s
ok  	cmd/compile/internal/dwarfgen	0.737s
ok  	cmd/compile/internal/importer	5.659s
ok  	cmd/compile/internal/ir	0.207s
ok  	cmd/compile/internal/logopt	0.595s
ok  	cmd/compile/internal/noder	0.210s
ok  	cmd/compile/internal/ssa	41.322s
ok  	cmd/compile/internal/syntax	0.214s
ok  	cmd/compile/internal/test	36.106s
ok  	cmd/compile/internal/typecheck	0.845s
ok  	cmd/compile/internal/types	0.158s
ok  	cmd/compile/internal/types2	8.383s
ok  	cmd/cover	4.041s
ok  	cmd/dist	0.210s
ok  	cmd/doc	0.559s
ok  	cmd/fix	5.303s
go test proxy running at GOPROXY=http://127.0.0.1:51540/mod
--- FAIL: TestScript (0.02s)
    --- FAIL: TestScript/cgo_stale_precompiled (0.66s)
        script_test.go:257:
            # Regression test for https://go.dev/issue/47215 and https://go.dev/issue/50183:
            # A mismatched $GOROOT_FINAL or missing $CC caused the C dependencies of the net
            # package to appear stale, and it could not be rebuilt due to a missing $CC. (0.000s)
            # Control case: net must not already be stale. (0.654s)
            > ! stale net
            FAIL: testdata/script/cgo_stale_precompiled.txt:8: net is unexpectedly stale


FAIL
FAIL	cmd/go	117.938s
ok  	cmd/go/internal/auth	0.219s
ok  	cmd/go/internal/cache	0.785s
ok  	cmd/go/internal/fsys	0.396s
ok  	cmd/go/internal/generate	0.301s
ok  	cmd/go/internal/get	0.386s
ok  	cmd/go/internal/imports	0.191s
ok  	cmd/go/internal/load	0.440s
ok  	cmd/go/internal/lockedfile	0.264s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.199s
ok  	cmd/go/internal/modconv	0.253s
ok  	cmd/go/internal/modfetch	0.358s
ok  	cmd/go/internal/modfetch/codehost	0.318s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.602s
ok  	cmd/go/internal/modload	0.279s
ok  	cmd/go/internal/mvs	0.179s
ok  	cmd/go/internal/par	0.264s
ok  	cmd/go/internal/search	0.185s
ok  	cmd/go/internal/str	0.209s
ok  	cmd/go/internal/test	0.338s
ok  	cmd/go/internal/vcs	0.255s
ok  	cmd/go/internal/web	0.327s
ok  	cmd/go/internal/work	0.369s
ok  	cmd/gofmt	0.387s
ok  	cmd/internal/archive	2.357s
ok  	cmd/internal/buildid	0.714s
ok  	cmd/internal/dwarf	0.329s
ok  	cmd/internal/edit	0.171s
ok  	cmd/internal/goobj	0.157s
ok  	cmd/internal/moddeps	7.709s
ok  	cmd/internal/obj	0.175s
ok  	cmd/internal/obj/ppc64	0.412s
ok  	cmd/internal/obj/riscv	0.243s
ok  	cmd/internal/obj/s390x	0.377s
ok  	cmd/internal/obj/x86	13.578s
ok  	cmd/internal/objabi	0.185s
ok  	cmd/internal/pkgpath	0.270s
ok  	cmd/internal/quoted	0.378s
ok  	cmd/internal/src	0.230s
ok  	cmd/internal/sys	0.235s
ok  	cmd/internal/test2json	0.483s
ok  	cmd/link	52.501s
ok  	cmd/link/internal/benchmark	0.250s
ok  	cmd/link/internal/ld	32.176s
ok  	cmd/link/internal/loader	0.219s
ok  	cmd/nm	12.094s
ok  	cmd/objdump	10.416s
ok  	cmd/pack	6.588s
ok  	cmd/pprof	8.455s
ok  	cmd/trace	0.930s
ok  	cmd/vet	18.385s
FAIL
go tool dist: Failed: exit status 1
2022/01/28 12:50:40 release -target="darwin-amd64": waiting a bit and trying again
2022/01/28 12:51:10 release -target="linux-amd64": build succeeded (after 0 retries)
2022/01/28 12:51:10 DONE BUILDRELEASE go1.18beta2 linux-amd64
2022/01/28 12:51:13 release -target="freebsd-386": build succeeded (after 0 retries)
2022/01/28 12:51:13 DONE BUILDRELEASE go1.18beta2 freebsd-386
2022/01/28 12:51:40 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:52:22 release -target="linux-386": build succeeded (after 0 retries)
2022/01/28 12:52:22 DONE BUILDRELEASE go1.18beta2 linux-386
2022/01/28 12:57:21 release -target="linux-amd64-longtest": build succeeded (after 0 retries)
2022/01/28 12:57:21 DONE BUILDRELEASE go1.18beta2 linux-amd64-longtest

mknyszek avatar Jan 28 '22 19:01 mknyszek

CC @dmitshur @toothrot

mknyszek avatar Jan 28 '22 19:01 mknyszek

I think it's worth noting that the hangs are so long that this is unlikely to be a test deadlocking, because it should have timed out by now.

mknyszek avatar Jan 28 '22 19:01 mknyszek

@golang/release

cagedmantis avatar Jan 31 '22 17:01 cagedmantis

We may want to consider either adding an hour long timeout here, or waiting until these tasks are ported to relui.

toothrot avatar Feb 02 '22 16:02 toothrot

Should be obsoleted by the watchdog checks in relui?

heschi avatar Aug 08 '22 21:08 heschi