Dragonfly2 icon indicating copy to clipboard operation
Dragonfly2 copied to clipboard

Published 20 hours ago • verified publisher icon dragonflyoss

Dragonfly peer download from recycled node

Open Leo-Song opened this issue 2 years ago • 1 comments

Bug report:

Node will auto scale which depend auto scale rules. After a node is recycled, scheduler won't delete this peer from database, other peers will still download task from this peer.

scheduler:

2022-09-04T13:04:05.871Z	INFO	service/service.go:220	task size scope is small and return single piece: &v1.SinglePiece{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8(nil), DstPid:"172.100.169.116-1-e594907f-ffa4-45a4-be09-2cf2279bf126", DstAddr:"172.100.169.116:65002", PieceInfo:(*v1.PieceInfo)(0xc000445730)} &v1.PieceInfo{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8(nil), PieceNum:0, RangeStart:0x0, RangeSize:0x361, PieceMd5:"", PieceOffset:0x0, PieceStyle:0, DownloadCost:0x0}	{"taskID": "9a621d0e2e272d3d1c0f7017b51df21067b137e6e970a22f1b9f03beafdcc665", "peerID": "172.100.76.151-1-54fc995e-b767-43e8-b8ab-04e68791f4bb"}
2022-09-04T13:04:05.871Z	INFO	zap/server_interceptors.go:39	finished unary call with code OK	{"grpc.start_time": "2022-09-04T13:04:05Z", "grpc.request.deadline": "2022-09-04T13:04:08Z", "system": "grpc", "span.kind": "server", "grpc.service": "scheduler.Scheduler", "grpc.method": "RegisterPeerTask", "grpc.code": "OK", "grpc.time_ms": 1.387}
2022-09-04T13:04:05.871Z	INFO	service/service.go:305	receive begin of piece: &v1.PieceResult{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(0xc0009bf520)}, sizeCache:134, unknownFields:[]uint8(nil), TaskId:"9a621d0e2e272d3d1c0f7017b51df21067b137e6e970a22f1b9f03beafdcc665", SrcPid:"172.100.76.151-1-54fc995e-b767-43e8-b8ab-04e68791f4bb", DstPid:"", PieceInfo:(*v1.PieceInfo)(0xc003e6e150), BeginTime:0x0, EndTime:0x0, Success:false, Code:0, HostLoad:(*v1.HostLoad)(nil), FinishedCount:0, ExtendAttribute:(*v1.ExtendAttribute)(nil)} &v1.PieceInfo{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:11, unknownFields:[]uint8(nil), PieceNum:-1, RangeStart:0x0, RangeSize:0x0, PieceMd5:"", PieceOffset:0x0, PieceStyle:0, DownloadCost:0x0}	{"taskID": "9a621d0e2e272d3d1c0f7017b51df21067b137e6e970a22f1b9f03beafdcc665", "peerID": "172.100.76.151-1-54fc995e-b767-43e8-b8ab-04e68791f4bb"}
2022-09-04T13:04:05.872Z	INFO	service/service.go:630	file type is small, peer has already returned to the parent when registering	{"taskID": "9a621d0e2e272d3d1c0f7017b51df21067b137e6e970a22f1b9f03beafdcc665", "peerID": "172.100.76.151-1-54fc995e-b767-43e8-b8ab-04e68791f4bb"}
2022-09-04T13:04:05.872Z	INFO	resource/peer.go:241	peer state is Running	{"taskID": "9a621d0e2e272d3d1c0f7017b51df21067b137e6e970a22f1b9f03beafdcc665", "peerID": "172.100.76.151-1-54fc995e-b767-43e8-b8ab-04e68791f4bb"}
2022-09-04T13:04:06.135Z	INFO	grpclog/grpclog.go:37	[core][Server #8] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 172.100.24.218:8002->172.100.82.214:55490: read: connection reset by peer"
2022-09-04T13:04:07.777Z	ERROR	service/service.go:350	receive failed piece: &v1.PieceResult{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(0xc0009bf520)}, sizeCache:204, unknownFields:[]uint8(nil), TaskId:"50076f5d38b0b2dfc5c5b219159b223f629ce84a840cbe3bb5dd4ad7d52fb153", SrcPid:"172.100.76.151-1-a095c14c-6d5d-44cd-937d-c83d0f20f718", DstPid:"172.100.26.95-1-db8178c0-6599-48d6-8439-ca67e02f484c", PieceInfo:(*v1.PieceInfo)(0xc0004396c0), BeginTime:0x1711a9fdfad3e27c, EndTime:0x1711a9fe721573c0, Success:false, Code:4007, HostLoad:(*v1.HostLoad)(nil), FinishedCount:0, ExtendAttribute:(*v1.ExtendAttribute)(nil)}	{"taskID": "50076f5d38b0b2dfc5c5b219159b223f629ce84a840cbe3bb5dd4ad7d52fb153", "peerID": "172.100.76.151-1-a095c14c-6d5d-44cd-937d-c83d0f20f718"}
d7y.io/dragonfly/v2/scheduler/service.(*Service).ReportPieceResult
	/go/src/d7y.io/dragonfly/v2/scheduler/service/service.go:350
d7y.io/dragonfly/v2/scheduler/rpcserver.(*Server).ReportPieceResult
	/go/src/d7y.io/dragonfly/v2/scheduler/rpcserver/rpcserver.go:78
d7y.io/api/pkg/apis/scheduler/v1._Scheduler_ReportPieceResult_Handler
	/go/pkg/mod/d7y.io/[email protected]/pkg/apis/scheduler/v1/scheduler.pb.go:1965
github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:51
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/validator/validator.go:72
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/logging/zap/server_interceptors.go:53
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.StreamServerInterceptor.func1
	/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:412
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
d7y.io/dragonfly/v2/pkg/rpc.ConvertErrorStreamServerInterceptor
	/go/src/d7y.io/dragonfly/v2/pkg/rpc/interceptor.go:101
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware/ratelimit.StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/ratelimit/ratelimit.go:34
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58
google.golang.org/grpc.(*Server).processStreamingRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1571
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1646
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:938
2022-09-04T13:04:07.777Z	INFO	service/service.go:714	schedule parent because of peer receive failed piece	{"taskID": "50076f5d38b0b2dfc5c5b219159b223f629ce84a840cbe3bb5dd4ad7d52fb153", "peerID": "172.100.76.151-1-a095c14c-6d5d-44cd-937d-c83d0f20f718"}
2022-09-04T13:04:07.778Z	INFO	scheduler/scheduler.go:315	candidate parents include []string{"172.100.161.151-1-be6de58f-7b19-4efb-8b3e-10f891b3a018", "172.100.169.116-1-c822e10e-a8b9-4c24-9be4-01ca3d60ec94", "172.100.237.243-1-51bf2b99-c476-46a0-b73f-f0a6bcca0f9f"}	{"taskID": "50076f5d38b0b2dfc5c5b219159b223f629ce84a840cbe3bb5dd4ad7d52fb153", "peerID": "172.100.76.151-1-a095c14c-6d5d-44cd-937d-c83d0f20f718"}
2022-09-04T13:04:07.779Z	INFO	scheduler/scheduler.go:207	schedule parent successful, replace parent to 172.100.169.116-1-c822e10e-a8b9-4c24-9be4-01ca3d60ec94 and candidate parents is [172.100.161.151-1-be6de58f-7b19-4efb-8b3e-10f891b3a018 172.100.237.243-1-51bf2b99-c476-46a0-b73f-f0a6bcca0f9f]	{"taskID": "50076f5d38b0b2dfc5c5b219159b223f629ce84a840cbe3bb5dd4ad7d52fb153", "peerID": "172.100.76.151-1-a095c14c-6d5d-44cd-937d-c83d0f20f718"}
2022-09-04T13:04:07.779Z	INFO	scheduler/scheduler.go:139	schedule parent 1 times successfully	{"taskID": "50076f5d38b0b2dfc5c5b219159b223f629ce84a840cbe3bb5dd4ad7d52fb153", "peerID": "172.100.76.151-1-a095c14c-6d5d-44cd-937d-c83d0f20f718"}
2022-09-04T13:04:07.872Z	ERROR	service/service.go:350	receive failed piece: &v1.PieceResult{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(0xc0009bf520)}, sizeCache:205, unknownFields:[]uint8(nil), TaskId:"9a621d0e2e272d3d1c0f7017b51df21067b137e6e970a22f1b9f03beafdcc665", SrcPid:"172.100.76.151-1-54fc995e-b767-43e8-b8ab-04e68791f4bb", DstPid:"172.100.169.116-1-e594907f-ffa4-45a4-be09-2cf2279bf126", PieceInfo:(*v1.PieceInfo)(0xc000445a40), BeginTime:0x1711a9fe00834467, EndTime:0x1711a9fe77be91ca, Success:false, Code:4007, HostLoad:(*v1.HostLoad)(nil), FinishedCount:0, ExtendAttribute:(*v1.ExtendAttribute)(nil)}	{"taskID": "9a621d0e2e272d3d1c0f7017b51df21067b137e6e970a22f1b9f03beafdcc665", "peerID": "172.100.76.151-1-54fc995e-b767-43e8-b8ab-04e68791f4bb"}
d7y.io/dragonfly/v2/scheduler/service.(*Service).ReportPieceResult
	/go/src/d7y.io/dragonfly/v2/scheduler/service/service.go:350
d7y.io/dragonfly/v2/scheduler/rpcserver.(*Server).ReportPieceResult
	/go/src/d7y.io/dragonfly/v2/scheduler/rpcserver/rpcserver.go:78
d7y.io/api/pkg/apis/scheduler/v1._Scheduler_ReportPieceResult_Handler
	/go/pkg/mod/d7y.io/[email protected]/pkg/apis/scheduler/v1/scheduler.pb.go:1965
github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:51
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/validator/validator.go:72
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/logging/zap/server_interceptors.go:53
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.StreamServerInterceptor.func1
	/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:412
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
d7y.io/dragonfly/v2/pkg/rpc.ConvertErrorStreamServerInterceptor
	/go/src/d7y.io/dragonfly/v2/pkg/rpc/interceptor.go:101
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware/ratelimit.StreamServerInterceptor.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/ratelimit/ratelimit.go:34
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1
	/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58
google.golang.org/grpc.(*Server).processStreamingRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1571
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1646
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:938

dfdaemon:

2022-09-04T13:04:06.163Z	DEBUG	peer/peertask_manager.go:186	peer task found: 26d226635f29c84080c663a9c476020f25f6d84192df5aebd59e5a946bd08963/172.100.76.151-1-5e4ab6be-7950-428a-9674-64f1f4a012b9
2022-09-04T13:04:07.771Z	ERROR	peer/piece_downloader.go:163	task id: 001619013754cad5b656a57b3c149c1f546b79e63cd30c12b4ceb6faa71f7bbe, piece num: 0, dst: 172.100.161.151:65002, download piece failed: Get "http://172.100.161.151:65002/download/001/001619013754cad5b656a57b3c149c1f546b79e63cd30c12b4ceb6faa71f7bbe?peerId=172.100.161.151-1-d826461d-7de8-4f42-bd9c-79c5e53ee205": dial tcp 172.100.161.151:65002: i/o timeout
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceDownloader).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_downloader.go:163
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_manager.go:188
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:873
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	ERROR	peer/piece_downloader.go:163	task id: f2690ceb4271534dfa9013dcd61c0a0b699d94af8831b49df1c76f996b85f2b8, piece num: 0, dst: 172.100.161.151:65002, download piece failed: Get "http://172.100.161.151:65002/download/f26/f2690ceb4271534dfa9013dcd61c0a0b699d94af8831b49df1c76f996b85f2b8?peerId=172.100.161.151-1-b1b0d632-07ef-489a-bc42-119ae8bc702d": dial tcp 172.100.161.151:65002: i/o timeout
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceDownloader).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_downloader.go:163
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_manager.go:188
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:873
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	ERROR	peer/piece_manager.go:192	download piece failed, piece num: 0, error: connect with http://172.100.161.151:65002/download/001/001619013754cad5b656a57b3c149c1f546b79e63cd30c12b4ceb6faa71f7bbe?peerId=172.100.161.151-1-d826461d-7de8-4f42-bd9c-79c5e53ee205 with error: Get "http://172.100.161.151:65002/download/001/001619013754cad5b656a57b3c149c1f546b79e63cd30c12b4ceb6faa71f7bbe?peerId=172.100.161.151-1-d826461d-7de8-4f42-bd9c-79c5e53ee205": dial tcp 172.100.161.151:65002: i/o timeout, from peer: 172.100.161.151-1-d826461d-7de8-4f42-bd9c-79c5e53ee205	{"peer": "172.100.76.151-1-1a982a2e-fddd-4c01-93b2-3bdf0f2ccb07", "task": "001619013754cad5b656a57b3c149c1f546b79e63cd30c12b4ceb6faa71f7bbe", "component": "PeerTask"}
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_manager.go:192
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:873
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	WARN	peer/peertask_conductor.go:886	single piece download failed, switch to download from other peers	{"peer": "172.100.76.151-1-1a982a2e-fddd-4c01-93b2-3bdf0f2ccb07", "task": "001619013754cad5b656a57b3c149c1f546b79e63cd30c12b4ceb6faa71f7bbe", "component": "PeerTask"}
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:886
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	ERROR	peer/piece_downloader.go:163	task id: 8eb6e91ee7628b8158cf64be1b07803cc25cf188aaa9327be79662f51ff67225, piece num: 0, dst: 172.100.217.186:65002, download piece failed: Get "http://172.100.217.186:65002/download/8eb/8eb6e91ee7628b8158cf64be1b07803cc25cf188aaa9327be79662f51ff67225?peerId=172.100.217.186-1-1010acaf-64f2-41ac-b1ff-1873d376e1cb": dial tcp 172.100.217.186:65002: i/o timeout
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceDownloader).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_downloader.go:163
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_manager.go:188
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:873
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	ERROR	peer/piece_manager.go:192	download piece failed, piece num: 0, error: connect with http://172.100.217.186:65002/download/8eb/8eb6e91ee7628b8158cf64be1b07803cc25cf188aaa9327be79662f51ff67225?peerId=172.100.217.186-1-1010acaf-64f2-41ac-b1ff-1873d376e1cb with error: Get "http://172.100.217.186:65002/download/8eb/8eb6e91ee7628b8158cf64be1b07803cc25cf188aaa9327be79662f51ff67225?peerId=172.100.217.186-1-1010acaf-64f2-41ac-b1ff-1873d376e1cb": dial tcp 172.100.217.186:65002: i/o timeout, from peer: 172.100.217.186-1-1010acaf-64f2-41ac-b1ff-1873d376e1cb	{"peer": "172.100.76.151-1-a4f3ed7e-d89d-406a-864d-83eab8c1efff", "task": "8eb6e91ee7628b8158cf64be1b07803cc25cf188aaa9327be79662f51ff67225", "component": "PeerTask"}
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_manager.go:192
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:873
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	ERROR	peer/piece_manager.go:192	download piece failed, piece num: 0, error: connect with http://172.100.161.151:65002/download/f26/f2690ceb4271534dfa9013dcd61c0a0b699d94af8831b49df1c76f996b85f2b8?peerId=172.100.161.151-1-b1b0d632-07ef-489a-bc42-119ae8bc702d with error: Get "http://172.100.161.151:65002/download/f26/f2690ceb4271534dfa9013dcd61c0a0b699d94af8831b49df1c76f996b85f2b8?peerId=172.100.161.151-1-b1b0d632-07ef-489a-bc42-119ae8bc702d": dial tcp 172.100.161.151:65002: i/o timeout, from peer: 172.100.161.151-1-b1b0d632-07ef-489a-bc42-119ae8bc702d	{"peer": "172.100.76.151-1-46a6b2f7-52b6-471a-9351-a5f15206c2a0", "task": "f2690ceb4271534dfa9013dcd61c0a0b699d94af8831b49df1c76f996b85f2b8", "component": "PeerTask"}
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_manager.go:192
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:873
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	WARN	peer/peertask_conductor.go:886	single piece download failed, switch to download from other peers	{"peer": "172.100.76.151-1-a4f3ed7e-d89d-406a-864d-83eab8c1efff", "task": "8eb6e91ee7628b8158cf64be1b07803cc25cf188aaa9327be79662f51ff67225", "component": "PeerTask"}
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:886
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	WARN	peer/peertask_conductor.go:886	single piece download failed, switch to download from other peers	{"peer": "172.100.76.151-1-46a6b2f7-52b6-471a-9351-a5f15206c2a0", "task": "f2690ceb4271534dfa9013dcd61c0a0b699d94af8831b49df1c76f996b85f2b8", "component": "PeerTask"}
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:886
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	ERROR	peer/piece_downloader.go:163	task id: 8e7ef59855a7e6d1c62e33dee95b0dbf1c61c2d47d1011ea32d4c65b982b551c, piece num: 0, dst: 172.100.26.95:65002, download piece failed: Get "http://172.100.26.95:65002/download/8e7/8e7ef59855a7e6d1c62e33dee95b0dbf1c61c2d47d1011ea32d4c65b982b551c?peerId=172.100.26.95-1-3250489d-9146-4ada-878e-3fbb4bbfb3fe": dial tcp 172.100.26.95:65002: i/o timeout
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceDownloader).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_downloader.go:163
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_manager.go:188
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:873
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	ERROR	peer/piece_manager.go:192	download piece failed, piece num: 0, error: connect with http://172.100.26.95:65002/download/8e7/8e7ef59855a7e6d1c62e33dee95b0dbf1c61c2d47d1011ea32d4c65b982b551c?peerId=172.100.26.95-1-3250489d-9146-4ada-878e-3fbb4bbfb3fe with error: Get "http://172.100.26.95:65002/download/8e7/8e7ef59855a7e6d1c62e33dee95b0dbf1c61c2d47d1011ea32d4c65b982b551c?peerId=172.100.26.95-1-3250489d-9146-4ada-878e-3fbb4bbfb3fe": dial tcp 172.100.26.95:65002: i/o timeout, from peer: 172.100.26.95-1-3250489d-9146-4ada-878e-3fbb4bbfb3fe	{"peer": "172.100.76.151-1-45f20d9c-d3d2-49de-8534-38a5ee98bca9", "task": "8e7ef59855a7e6d1c62e33dee95b0dbf1c61c2d47d1011ea32d4c65b982b551c", "component": "PeerTask"}
d7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadPiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/piece_manager.go:192
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:873
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.771Z	WARN	peer/peertask_conductor.go:886	single piece download failed, switch to download from other peers	{"peer": "172.100.76.151-1-45f20d9c-d3d2-49de-8534-38a5ee98bca9", "task": "8e7ef59855a7e6d1c62e33dee95b0dbf1c61c2d47d1011ea32d4c65b982b551c", "component": "PeerTask"}
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullSinglePiece
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:886
d7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPieces
	/go/src/d7y.io/dragonfly/v2/client/daemon/peer/peertask_conductor.go:556
2022-09-04T13:04:07.774Z	DEBUG	peer/peertask_conductor.go:700	receive peerPacket task_id:"8e7ef59855a7e6d1c62e33dee95b0dbf1c61c2d47d1011ea32d4c65b982b551c"  src_pid:"172.100.76.151-1-45f20d9c-d3d2-49de-8534-38a5ee98bca9"  parallel_count:3  main_peer:{ip:"172.100.169.116"  rpc_port:65000  peer_id:"172.100.169.116-1-c75d390b-2fd8-43b2-b967-e132397062fd"}  candidate_peers:{ip:"172.100.159.76"  rpc_port:65000  peer_id:"172.100.159.76-1-4f90ca0c-ab62-4c9e-905d-9696e564458f"}  candidate_peers:{ip:"172.100.237.243"  rpc_port:65000  peer_id:"172.100.237.243-1-0a667530-aaa6-498d-b438-7d39109d89bf"}  code:Success	{"peer": "172.100.76.151-1-45f20d9c-d3d2-49de-8534-38a5ee98bca9", "task": "8e7ef59855a7e6d1c62e33dee95b0dbf1c61c2d47d1011ea32d4c65b982b551c", "component": "PeerTask"}

Expected behavior:

How to reproduce it:

Environment:

  • Dragonfly version: 2.0.6
  • OS:
  • Kernel (e.g. uname -a):
  • Others: Container runtime: docker://20.10.7, Host network: true

Leo-Song avatar Sep 13 '22 17:09 Leo-Song

When the peer exits, it will actively notify the scheduler to release the task associated with the peer. If you are in an abnormal situation, it will block the abnormal node, and then do the scheduling again.

gaius-qi avatar Sep 15 '22 02:09 gaius-qi

The v2.0.8 version has fix this problem.

gaius-qi avatar Dec 09 '22 04:12 gaius-qi