Crashing a node floods the terminal with warnings after creating channels

[AlianBenabdallah]

Summary of Bug

Many threads issue a warning when a node is unreachable. It results in many warning per second (and I suspect that the more channels you have, the more warnings you get) on the terminal which runs hermes start.

The major issue is that it buries useful logs under warnings.

Version

hermes 1.0.0-rc.0 gm v0.1.3

Steps to Reproduce

With gm and this gm.toml :

[global]
  add_to_hermes = true
  gaiad_binary = "~/go/bin/gaiad"
  ports_start_at = 27050

  [global.hermes]
    binary = "~/.cargo/bin/hermes"

[ibc-0]
  ports_start_at = 27020

[ibc-1]
  ports_start_at = 27030

Execute :

gm start
gm hermes config
gm hermes keys
gm hermes cc

Copy paste the output of gm hermes cc to create connections and channels

On another terminal : hermes start Wait a bit. Then gm stop ibc-0 Come back to the terminal running hermes start, you should see a lot of warnings.

2022-07-19T11:07:59.024656Z  WARN ThreadId(35) refresh{client=07-tendermint-0 src_chain=ibc-1 dst_chain=ibc-0}: task encountered ignorable error: error raised while trying to refresh client 07-tendermint-0: failed querying client state on dst chain: RPC error to endpoint http://localhost:27020/: HTTP error: error trying to connect: tcp connect error: Connection refused (os error 111)
2022-07-19T11:07:59.028355Z  WARN ThreadId(38) refresh{client=07-tendermint-0 src_chain=ibc-0 dst_chain=ibc-1}: task encountered ignorable error: error raised while trying to refresh client 07-tendermint-0: failed querying the application status of source chain: RPC error to endpoint http://localhost:27020/: HTTP error: error trying to connect: tcp connect error: Connection refused (os error 111)
2022-07-19T11:08:00.025282Z  WARN ThreadId(35) refresh{client=07-tendermint-0 src_chain=ibc-1 dst_chain=ibc-0}: task encountered ignorable error: error raised while trying to refresh client 07-tendermint-0: failed querying client state on dst chain: RPC error to endpoint http://localhost:27020/: HTTP error: error trying to connect: tcp connect error: Connection refused (os error 111)
2022-07-19T11:08:00.030022Z  WARN ThreadId(38) refresh{client=07-tendermint-0 src_chain=ibc-0 dst_chain=ibc-1}: task encountered ignorable error: error raised while trying to refresh client 07-tendermint-0: failed querying the application status of source chain: RPC error to endpoint http://localhost:27020/: HTTP error: error trying to connect: tcp connect error: Connection refused (os error 111)

Acceptance Criteria

Hermes outputs less warnings per second per issue.

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate milestone (priority) applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned