nydus-snapshotter icon indicating copy to clipboard operation
nydus-snapshotter copied to clipboard

conatinerd call snapshotter's Commit and still wait for result,snapshotter maybe crash

Open hopegi opened this issue 1 year ago • 1 comments

conatinerd call snapshotter's Commit and still wait for a result.


goroutine 2183941 [select, 1 minutes]:\n
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc001cc6b40)\n\t/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/internal/transport/transport.go:324 +0x7c\n
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)\n\t/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/internal/transport/transport.go:339\n
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0031d2000, {0x5636f032cca0?, 0xc002b62000}, 0xc003796600?)\n\t/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/stream.go:1011 +0xbe\n
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x0?)\n\t/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/stream.go:865 +0x25\n
google.golang.org/grpc.(*clientStream).withRetry(0xc001cc67e0, 0xc0037fbf90, 0xc0037fbf60)\n\t/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/stream.go:724 +0x144\n
google.golang.org/grpc.(*clientStream).RecvMsg(0xc001cc67e0, {0x5636f032cca0?, 0xc002b62000?})\n\t/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/stream.go:864 +0x12e\n
google.golang.org/grpc.invoke({0x5636f03ad638?, 0xc002752c00?}, {0x5636efb4acd1?, 0x8?}, {0x5636f02d9180, 0xc003796600}, {0x5636f032cca0, 0xc002b62000}, 0x0?, {0xc000322360, ...})\n\t/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/call.go:73 +0xd7\n
google.golang.org/grpc.(*ClientConn).Invoke(0x7f1a2cf595b8?, {0x5636f03ad638?, 0xc002752c00?}, {0x5636efb4acd1?, 0x0?}, {0x5636f02d9180?, 0xc003796600?}, {0x5636f032cca0?, 0xc002b62000?}, {0x0, ...})\n\t/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/call.go:37 +0x265\n
github.com/containerd/containerd/api/services/snapshots/v1.(*snapshotsClient).Commit(0xc00011a7c8, {0x5636f03ad638, 0xc002752c00}, 0xc000f8b230?, {0x0, 0x0, 0x0})\n\t/go/src/github.com/containerd/containerd/api/services/snapshots/v1/snapshots.pb.go:983 +0xc9\n
github.com/containerd/containerd/snapshots/proxy.(*proxySnapshotter).Commit(0xc000322560, {0x5636f03ad638, 0xc002752c00}, {0xc00145c180, 0x54}, {0xc0025bb420, 0x6b}, {0xc00303e028, 0x1, 0x1})\n\t/go/src/github.com/containerd/containerd/snapshots/proxy/proxy.go:139 +0x182\n
github.com/containerd/containerd/metadata.(*snapshotter).Commit.func1(0x0?)\n\t/go/src/github.com/containerd/containerd/metadata/snapshot.go:596 +0xb9b\n
go.etcd.io/bbolt.(*DB).Update(0x0?, 0xc0037960c0)\n\t/go/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:862 +0x82\n
github.com/containerd/containerd/metadata.(*DB).Update(0xc00025f960, 0x5636f0154da0?)\n\t/go/src/github.com/containerd/containerd/metadata/db.go:246 +0x9e\n
github.com/containerd/containerd/metadata.update({0x5636f03ad638?, 0xc002752c00?}, {0x5636f039b610, 0xc00025f960}, 0xc0037960c0)\n\t/go/src/github.com/containerd/containerd/metadata/bolt.go:56 +0xd4\n
github.com/containerd/containerd/metadata.(*snapshotter).Commit(0xc000043300, {0x5636f03ad638?, 0xc002752c00}, {0xc001a44fa0, 0x47}, {0xc0015193e0, 0x5e}, {0xc001482348, 0x1, 0x1})\n\t/go/src/github.com/containerd/containerd/metadata/snapshot.go:515 +0x36c\n
github.com/containerd/containerd/services/snapshots.(*snapshotter).Commit(0xc000322820, {0x5636f03ad638, 0xc002752c00}, {0xc001a44fa0, 0x47}, {0xc0015193e0, 0x5e}, {0xc001482348, 0x1, 0x1})\n\t/go/src/github.com/containerd/containerd/services/snapshots/snapshotters.go:87 +0x7a\n
github.com/containerd/containerd.(*unpacker).unpack.func1(0x7, {{0xc002861200, 0x31}, {0xc003207d60, 0x47}, 0x3eede0, {0x0, 0x0, 0x0}, 0xc003071740, ...})\n\t/go/src/github.com/containerd/containerd/unpacker.go:235 +0x146e\n
github.com/containerd/containerd.(*unpacker).unpack(0xc00202c1c0, {0x5636f03ad638, 0xc0022d5f80}, 0xc0023a3860, {0x5636f0399560, 0xc0026ffec0}, {{0xc002e958f0, 0x2e}, {0xc003207a40, 0x47}, ...}, ...)\n\t/go/src/github.com/containerd/containerd/unpacker.go:258 +0xd82\n
github.com/containerd/containerd.(*unpacker).handlerWrapper.func1.1.1()\n\t/go/src/github.com/containerd/containerd/unpacker.go:368 +0x85\n
golang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/src/github.com/containerd/containerd/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x67\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/go/src/github.com/containerd/containerd/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x8d\n\n

It maybe crash in snapshotter,but there is not any error log in the snapshotter 1720507416787_企业微信截图_17204409704424

containerd-nydus-grpc --version Version: v0.4.0 Revision: 1e18acbf9d39588d39d0276a423e33ebeeb3462b

hopegi avatar Jul 10 '24 03:07 hopegi

This may need to capture the goroutine stack of snapshotter for further debugging.

imeoer avatar Jul 11 '24 01:07 imeoer