cache failed to update when etcd restarted

[wangxiaochuang]

Issue description

when I restart etcd, dashboard occure the error: 2022-05-31T10:27:32.003+0800 ERROR storage/etcd.go:245 channel canceled 2022-05-31T10:27:32.003+0800 WARN store/store.go:124 watch failed: channel canceled

then, cache data is no longer updated

Expected behavior

os: linux version: 2.11

How to Reproduce

just restart etcd

Screenshots

No response

Environment

• apisix version (cmd: apisix version):
• OS (cmd: uname -a):
• OpenResty / Nginx version (cmd: nginx -V or openresty -V):
• etcd version, if have (cmd: run etcd --version):
• apisix-dashboard version, if have:
• Browser version, if have:

Additional context

No response

[jwrookie]

Currently, there is no re-watch after watch fails, do you have any good ideas?

[starsz]

I think we can support re-connect the etcd.

[pixeldin]

Hello, @wangxiaochuang, It seems that this case did not reproduce on my machine(as I restarting the etcd durning the dashboard running and wait for canceled to notify outside channel). output.Error = fmt.Errorf("channel canceled") Could you show me more detail about your program environment, is your etcd service a single point or a cluster? What's the specific operation about 'cache failed to update'. Looking forward to your reply :)