There may be a bug exist when dubbo shutdown gracefully in dubbo 2.7.7. [[RpcException("No provider available in " + invokers);]]

[zhangjh]

• [x] I have searched the issues of this repository and believe that this is not a duplicate.

Environment

• Dubbo version: 2.7.7
• Operating System version: k8s+docker+linux
• Java version: 1.8

Steps to reproduce this issue

I'm not sure it is a bug already knew/reported or just a feature problem can not be solved.

When dubbo client shutdown, we all know dubbo will run the DubboShutdownHook class to shutdow gracefully. In that progress, there may some requests not finished, I occured a situation that the consumer instance throws exception as RpcException("No provider available in " + invokers);

1. In order to reproduce the problem, we should use multiple registies first:

// dubbo consumer config
dubbo.registries.a.address=zookeeper://xxx
dubbo.registries.b.address=zookeeper://xxx

2. then add the breakpoint at the dubbo consumer method call, such as: demoService.sayHello(name); and add the breakpoint at org.apache.dubbo.config.DubboShutdownHook#destroyAll yet (breakpoint must be thread level so that you switch thread whe debugging)

3. then run command kill or kill -15 to trigger dubbo instance shutdown,

ps -ef | grep java | grep "org.apache.dubbo.demo.consumer.Application" | awk '{print $2}' | xargs kill -15

4. when thread suspend on org.apache.dubbo.config.DubboShutdownHook#destroyAll, switch debug thread to main thread, step debuggind to org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker#invoke, there produce a invoker list. They are all available now.
5. then switch debug thread to dubboShutdownHook, step debug to "destroyAll" method, and let it run finish. When destroyProtocols run finish, the variable org.apache.dubbo.rpc.cluster.directory.AbstractDirectory#destroyed will be set true.
6. then switch debug thread to main again. The invokers get before now not available! Because the invoker.isAvailable() will check the variable org.apache.dubbo.rpc.cluster.directory.AbstractDirectory#destroyed.
7. Let the debug step over, it will enter in org.apache.dubbo.rpc.cluster.support.registry.ZoneAwareClusterInvoker#doInvoke. In this method, invokers will be checked available again. So a RpcException throws at last.

Expected Behavior

Do you think sleep shutdown timeout after destroy the registries, then run destroyProtocols will solve this problem? Give some timeout to let the requests remaining to run finish.