Substrate drops Gossamer connection after `Network(Timeout)`

[EclesioMeloJunior]

Describe the bug

In a cross-client setup of 2 substrate nodes and 1 gossamer node (12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn), it is possible to see after some time substrate dropping gossamer connection after a request timeout, which I believe is related to sync block request/response

2022-08-09 16:21:52.005 DEBUG tokio-runtime-worker sync: Request to peer PeerId("12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn") failed: Network(Timeout).
2022-08-09 16:21:52.005 TRACE tokio-runtime-worker sub-libp2p: External API => Disconnect(12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn, SetId(0))
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker sub-libp2p: PSM <= Dropped(12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn, SetId(0))
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker peerset: Fleeting 12D3KooWNMfwyFoBiGKPDi6FK9fNA6rNciSzXTuNiSgd46U8U5KD: 8838 -> 8662
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker peerset: Fleeting 12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn: 8021 -> 7861
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker peerset: Dropping 12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn: -256 to 7605
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker sub-libp2p: External API <= Closed(12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn, SetId(0))
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker sub-libp2p: Handler(PeerId("12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn"), ConnectionId(2)) <= Close(SetId(0))
2022-08-09 16:21:52.006 DEBUG tokio-runtime-worker sync: 12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn disconnected
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker sync: 0 blocks ready for import
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker sub-libp2p: PSM => Connect(12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn, SetId(0)): No connection in proper state. Delaying.
2022-08-09 16:21:52.006 TRACE tokio-runtime-worker peerset: Report 12D3KooWAtaxAqM5KojWyUj6g79R1kzKE3XGnz89Efj8KStR4msn: -1024 to 6581. Reason: Request timeout

[kishansagathiya]

@EclesioMeloJunior Can you upload full logs of all three nodes?

[EclesioMeloJunior]

@kishansagathiya unfortunately not, sorry for don't provide the full logs, I created this issue to start an investigation under the sync package

[EclesioMeloJunior]

Closing in favor of https://github.com/ChainSafe/gossamer/issues/2801