storage: TestIntegration_ObjectReadGRPC failed
This test failed!
To configure my behavior, see the Flaky Bot documentation.
If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.
commit: dfda4d1e429951d309280a5e0ace24161c79d765 buildURL: Build Status, Sponge status: failed
Test output
/usr/local/go/src/testing/testing.go:1618 +0x11f /usr/local/go/src/time/sleep.go:167 +0x52 /usr/local/go/src/testing/testing.go:1169 +0x5f4 /usr/local/go/src/testing/testing.go:1439 +0xa7 /usr/local/go/src/testing/testing.go:1123 +0x203 /usr/local/go/src/testing/testing.go:1437 +0x613 /usr/local/go/src/testing/testing.go:1345 +0x3b4 /tmpfs/src/google-cloud-go/storage/integration_test.go:92 +0x5f _testmain.go:301 +0x272 /go/pkg/mod/[email protected]/stats/view/worker.go:276 +0x1e6 /go/pkg/mod/[email protected]/stats/view/worker.go:34 +0xb2 /usr/local/go/src/testing/testing.go:1005 +0x3c6 /tmpfs/src/google-cloud-go/storage/bucket_test.go:32 +0x56 /usr/local/go/src/testing/testing.go:1123 +0x203 /usr/local/go/src/testing/testing.go:1168 +0x5bc /usr/local/go/src/testing/testing.go:1005 +0x3c6 /tmpfs/src/google-cloud-go/storage/bucket_test.go:317 +0x56 /usr/local/go/src/testing/testing.go:1123 +0x203 /usr/local/go/src/testing/testing.go:1168 +0x5bc /usr/local/go/src/testing/testing.go:1005 +0x3c6 /tmpfs/src/google-cloud-go/storage/copy_test.go:25 +0x56 /usr/local/go/src/testing/testing.go:1123 +0x203 /usr/local/go/src/testing/testing.go:1168 +0x5bc /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x216 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x2df /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x133 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1b73 /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0x1ab /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x405 /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0x1ab /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x405 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x216 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x2df /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x133 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1b73 /usr/local/go/src/runtime/netpoll.go:222 +0x55 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe6 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 /usr/local/go/src/internal/poll/fd_unix.go:159 +0x23f /usr/local/go/src/net/fd_posix.go:55 +0x66 /usr/local/go/src/net/net.go:182 +0xec /usr/local/go/src/crypto/tls/conn.go:779 +0xa7 /usr/local/go/src/bytes/buffer.go:204 +0x159 /usr/local/go/src/crypto/tls/conn.go:801 +0x215 /usr/local/go/src/crypto/tls/conn.go:608 +0x42b /usr/local/go/src/crypto/tls/conn.go:576 /usr/local/go/src/crypto/tls/conn.go:1252 +0x225 /usr/local/go/src/bufio/bufio.go:227 +0x7c7 /usr/local/go/src/io/io.go:314 +0x9d /usr/local/go/src/io/io.go:333 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0xb1 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0xff /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x2a5 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x14f7 /usr/local/go/src/runtime/netpoll.go:222 +0x55 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe6 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 /usr/local/go/src/internal/poll/fd_unix.go:159 +0x23f /usr/local/go/src/net/fd_posix.go:55 +0x66 /usr/local/go/src/net/net.go:182 +0xec /usr/local/go/src/crypto/tls/conn.go:779 +0xa7 /usr/local/go/src/bytes/buffer.go:204 +0x159 /usr/local/go/src/crypto/tls/conn.go:801 +0x215 /usr/local/go/src/crypto/tls/conn.go:608 +0x42b /usr/local/go/src/crypto/tls/conn.go:576 /usr/local/go/src/crypto/tls/conn.go:1252 +0x225 /usr/local/go/src/bufio/bufio.go:227 +0x7c7 /usr/local/go/src/io/io.go:314 +0x9d /usr/local/go/src/io/io.go:333 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0xb1 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0xff /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x2a5 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x14f7 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:323 +0x166 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:338 /go/pkg/mod/google.golang.org/[email protected]/stream.go:963 +0x1047 /go/pkg/mod/google.golang.org/[email protected]/stream.go:828 +0x65 /go/pkg/mod/google.golang.org/[email protected]/stream.go:686 +0xe7 /go/pkg/mod/google.golang.org/[email protected]/stream.go:827 +0x165 /go/pkg/mod/google.golang.org/[email protected]/googleapis/storage/v2/storage.pb.go:9729 +0x85 /tmpfs/src/google-cloud-go/storage/reader.go:492 +0x230 /tmpfs/src/google-cloud-go/storage/invoke.go:55 +0x4b /go/pkg/mod/cloud.google.com/[email protected]/internal/retry.go:40 +0x6c /go/pkg/mod/cloud.google.com/[email protected]/internal/retry.go:33 /tmpfs/src/google-cloud-go/storage/invoke.go:54 +0x214 /tmpfs/src/google-cloud-go/storage/reader.go:486 +0x2e6 /tmpfs/src/google-cloud-go/storage/reader.go:506 +0x533 /tmpfs/src/google-cloud-go/storage/reader.go:99 +0x1ab /tmpfs/src/google-cloud-go/storage/reader.go:81 /tmpfs/src/google-cloud-go/storage/integration_test.go:814 +0x5ef /usr/local/go/src/testing/testing.go:1123 +0x203 /usr/local/go/src/testing/testing.go:1168 +0x5bc /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x216 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x2df /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x133 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1b73 /usr/local/go/src/runtime/netpoll.go:222 +0x55 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe6 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 /usr/local/go/src/internal/poll/fd_unix.go:159 +0x23f /usr/local/go/src/net/fd_posix.go:55 +0x66 /usr/local/go/src/net/net.go:182 +0xec /usr/local/go/src/crypto/tls/conn.go:779 +0xa7 /usr/local/go/src/bytes/buffer.go:204 +0x159 /usr/local/go/src/crypto/tls/conn.go:801 +0x215 /usr/local/go/src/crypto/tls/conn.go:608 +0x42b /usr/local/go/src/crypto/tls/conn.go:576 /usr/local/go/src/crypto/tls/conn.go:1252 +0x225 /usr/local/go/src/bufio/bufio.go:227 +0x7c7 /usr/local/go/src/io/io.go:314 +0x9d /usr/local/go/src/io/io.go:333 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0xb1 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0xff /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x2a5 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x14f7 /usr/local/go/src/runtime/netpoll.go:222 +0x55 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe6 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 /usr/local/go/src/internal/poll/fd_unix.go:159 +0x23f /usr/local/go/src/net/fd_posix.go:55 +0x66 /usr/local/go/src/net/net.go:182 +0xec /usr/local/go/src/crypto/tls/conn.go:779 +0xa7 /usr/local/go/src/bytes/buffer.go:204 +0x159 /usr/local/go/src/crypto/tls/conn.go:801 +0x215 /usr/local/go/src/crypto/tls/conn.go:608 +0x42b /usr/local/go/src/crypto/tls/conn.go:576 /usr/local/go/src/crypto/tls/conn.go:1252 +0x225 /usr/local/go/src/bufio/bufio.go:227 +0x7c7 /usr/local/go/src/io/io.go:314 +0x9d /usr/local/go/src/io/io.go:333 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0xb1 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0xff /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x2a5 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x14f7 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x216 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x2df /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x133 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1b73 /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0x1ab /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x405 /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0x1ab /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x405 /go/pkg/mod/google.golang.org/[email protected]/stream.go:335 +0x126 /go/pkg/mod/google.golang.org/[email protected]/stream.go:334 +0xdc7
commit: b01c03783d84cb7a3eba4f69d49d3fb7be1b6353 buildURL: Build Status, Sponge status: failed
Test output
/usr/local/go/src/testing/testing.go:1788 +0xbb /usr/local/go/src/time/sleep.go:180 +0x4a /usr/local/go/src/testing/testing.go:1307 +0x752 /usr/local/go/src/testing/testing.go:1598 +0x9a /usr/local/go/src/testing/testing.go:1259 +0x230 /usr/local/go/src/testing/testing.go:1596 +0x7cb /usr/local/go/src/testing/testing.go:1504 +0x9d2 /tmpfs/src/google-cloud-go/storage/integration_test.go:92 +0x56 _testmain.go:301 +0x265 /go/pkg/mod/[email protected]/stats/view/worker.go:276 +0x185 /go/pkg/mod/[email protected]/stats/view/worker.go:34 +0xf9 /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0x15c /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x4d9 /usr/local/go/src/testing/testing.go:1104 +0x48d /tmpfs/src/google-cloud-go/storage/bucket_test.go:32 +0x4b /usr/local/go/src/testing/testing.go:1259 +0x230 /usr/local/go/src/testing/testing.go:1306 +0x727 /usr/local/go/src/testing/testing.go:1104 +0x48d /tmpfs/src/google-cloud-go/storage/bucket_test.go:317 +0x4b /usr/local/go/src/testing/testing.go:1259 +0x230 /usr/local/go/src/testing/testing.go:1306 +0x727 /usr/local/go/src/testing/testing.go:1104 +0x48d /tmpfs/src/google-cloud-go/storage/copy_test.go:25 +0x4b /usr/local/go/src/testing/testing.go:1259 +0x230 /usr/local/go/src/testing/testing.go:1306 +0x727 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:323 +0x114 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:338 /go/pkg/mod/google.golang.org/[email protected]/stream.go:963 +0x13e /go/pkg/mod/google.golang.org/[email protected]/stream.go:828 +0x57 /go/pkg/mod/google.golang.org/[email protected]/stream.go:686 +0xc7 /go/pkg/mod/google.golang.org/[email protected]/stream.go:827 +0x186 /go/pkg/mod/google.golang.org/[email protected]/googleapis/storage/v2/storage.pb.go:9729 +0x68 /tmpfs/src/google-cloud-go/storage/reader.go:492 +0x1cb /tmpfs/src/google-cloud-go/storage/invoke.go:55 +0x3a /go/pkg/mod/cloud.google.com/[email protected]/internal/retry.go:40 +0x8c /go/pkg/mod/cloud.google.com/[email protected]/internal/retry.go:33 /tmpfs/src/google-cloud-go/storage/invoke.go:54 +0x293 /tmpfs/src/google-cloud-go/storage/reader.go:486 +0x27f /tmpfs/src/google-cloud-go/storage/reader.go:506 +0x589 /tmpfs/src/google-cloud-go/storage/reader.go:99 +0x1ab /tmpfs/src/google-cloud-go/storage/reader.go:81 /tmpfs/src/google-cloud-go/storage/integration_test.go:814 +0xaaa /usr/local/go/src/testing/testing.go:1259 +0x230 /usr/local/go/src/testing/testing.go:1306 +0x727 /usr/local/go/src/runtime/netpoll.go:234 +0x89 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 /usr/local/go/src/internal/poll/fd_unix.go:167 +0x419 /usr/local/go/src/net/fd_posix.go:56 +0x51 /usr/local/go/src/net/net.go:183 +0xb1 /usr/local/go/src/crypto/tls/conn.go:777 +0x86 /usr/local/go/src/bytes/buffer.go:204 +0x113 /usr/local/go/src/crypto/tls/conn.go:799 +0x1df /usr/local/go/src/crypto/tls/conn.go:606 +0x3fe /usr/local/go/src/crypto/tls/conn.go:574 /usr/local/go/src/crypto/tls/conn.go:1277 +0x29c /usr/local/go/src/bufio/bufio.go:227 +0x4db /usr/local/go/src/io/io.go:328 +0xde /usr/local/go/src/io/io.go:347 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x55d /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x269c /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1b3 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x125 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x106 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x2fd4 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1b3 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x125 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x106 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x2fd4 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1b3 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x125 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x106 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x2fd4 /usr/local/go/src/runtime/netpoll.go:234 +0x89 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 /usr/local/go/src/internal/poll/fd_unix.go:167 +0x419 /usr/local/go/src/net/fd_posix.go:56 +0x51 /usr/local/go/src/net/net.go:183 +0xb1 /usr/local/go/src/crypto/tls/conn.go:777 +0x86 /usr/local/go/src/bytes/buffer.go:204 +0x113 /usr/local/go/src/crypto/tls/conn.go:799 +0x1df /usr/local/go/src/crypto/tls/conn.go:606 +0x3fe /usr/local/go/src/crypto/tls/conn.go:574 /usr/local/go/src/crypto/tls/conn.go:1277 +0x29c /usr/local/go/src/bufio/bufio.go:227 +0x4db /usr/local/go/src/io/io.go:328 +0xde /usr/local/go/src/io/io.go:347 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x55d /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x269c /go/pkg/mod/google.golang.org/[email protected]/stream.go:335 +0xf2 /go/pkg/mod/google.golang.org/[email protected]/stream.go:334 +0x15fb /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0x15c /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x4d9 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1b3 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x125 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:414 +0x106 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x2fd4 /usr/local/go/src/runtime/netpoll.go:234 +0x89 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 /usr/local/go/src/internal/poll/fd_unix.go:167 +0x419 /usr/local/go/src/net/fd_posix.go:56 +0x51 /usr/local/go/src/net/net.go:183 +0xb1 /usr/local/go/src/crypto/tls/conn.go:777 +0x86 /usr/local/go/src/bytes/buffer.go:204 +0x113 /usr/local/go/src/crypto/tls/conn.go:799 +0x1df /usr/local/go/src/crypto/tls/conn.go:606 +0x3fe /usr/local/go/src/crypto/tls/conn.go:574 /usr/local/go/src/crypto/tls/conn.go:1277 +0x29c /usr/local/go/src/bufio/bufio.go:227 +0x4db /usr/local/go/src/io/io.go:328 +0xde /usr/local/go/src/io/io.go:347 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x55d /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x269c /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0x15c /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x4d9 /usr/local/go/src/runtime/netpoll.go:234 +0x89 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 /usr/local/go/src/internal/poll/fd_unix.go:167 +0x419 /usr/local/go/src/net/fd_posix.go:56 +0x51 /usr/local/go/src/net/net.go:183 +0xb1 /usr/local/go/src/crypto/tls/conn.go:777 +0x86 /usr/local/go/src/bytes/buffer.go:204 +0x113 /usr/local/go/src/crypto/tls/conn.go:799 +0x1df /usr/local/go/src/crypto/tls/conn.go:606 +0x3fe /usr/local/go/src/crypto/tls/conn.go:574 /usr/local/go/src/crypto/tls/conn.go:1277 +0x29c /usr/local/go/src/bufio/bufio.go:227 +0x4db /usr/local/go/src/io/io.go:328 +0xde /usr/local/go/src/io/io.go:347 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1499 +0x55d /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:364 +0x269c /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:77 +0x15c /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:67 +0x4d9
Looks like this issue is flaky. :worried:
I'm going to leave this open and stop commenting.
A human should fix and close this.
When run at the same commit (b01c03783d84cb7a3eba4f69d49d3fb7be1b6353), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).
Changing to type Process for now -- this is not a production code path so there isn't huge urgency on investigating.
Closing this issue as the test has been removed in favour of other existing tests. This was likely a GCS flakiness issue that has been resolved since. If not resolved, it should re-appear at some point in other tests.