neofs-node icon indicating copy to clipboard operation
neofs-node copied to clipboard

tree: Replication request fails due to gRPC/TCP closed connection when node under load

Open cthulhu-rider opened this issue 1 year ago • 0 comments

when NeoFS storage nodes are under high load, Tree service's replication sometimes fail:

Jul 13 14:47:32 neofs-node[551040]: 2023-07-13T14:47:32.550Z        warn        tree/replicator.go:85        failed to sent update to the node        {"last_error": "rpc error: code = Unavailable desc = error reading from server: read tcp [<ipv6>]:53262->[<ipv6>]:8080: use of closed network connection", "address": "/dns4/dns/tcp/8080](http://dns/tcp/8080)", "key": "<key>"}

seems like used gRPC conn bases on already closed TCP conn, but it's not obvious what closed it and why

  • being Tree service client, NeoFS S3 Gateway encounters similar problem during load https://github.com/nspcc-dev/neofs-s3-gw/issues/797

cthulhu-rider avatar Jul 13 '23 15:07 cthulhu-rider