Sometimes got UNAVAILABLE status from rpc server?

[pigLoveRabbit520]

Problem description

here is error log:

Error: 14 UNAVAILABLE: Stream refused by server
    at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
    at processTicksAndRejections (internal/process/task_queues.js:75:11)

this error will occur 3 or more times a day

Environment

• OS name [Ubuntu 20.04 amd64]
• Node version [v14.16.1]
• Package name and version ["@grpc/grpc-js": "^1.3.0"]

[pigLoveRabbit520]

@murgatroid99 It seems this error is the bug of @grpc/grpc-js.

[murgatroid99]

Can you be more specific? What about this error do you think is caused by a bug?

[pigLoveRabbit520]

@murgatroid99 Because I can't reproduce the problem in my dev env, and it will occur one hundred times a day in production. Maybe plenty of concurrent requests are the cause.

[blastshielddown]

This may or may not be related but I have been running into a similar error when trying to make a request to an RPC endpoint on remote hosts using the @grpc/grpc-js library. Servers are located in both AWS and GCP, host:port reachable on both via telnet. Requests to endpoints on my local machine and my local network work perfectly fine.

Strangely, I am able to make the requests successfully to the remote hosts if I switch the package from @grpc/grpc-js to the deprecated grpc package.

I should add that the requests are over TLS.

Any thoughts on what the root cause could be?

node: 16.13.0 @grpc/grpc-js: 1.3.7 grpc: 1.24.11

Error: 14 UNAVAILABLE: No connection established
    at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:180:52)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:160:78

[murgatroid99]

@blastshielddown That is a completely different error from the one at the start of this issue. The original error "Stream refused by server" means that the client sent the request to the server, and the server refused to handle the request. Your error "No connection established" means that the client failed to ever establish a connection to the server in the first place. There are any number of reasons that a client might fail to connect to a server: the DNS name could be wrong, or the DNS entry could be wrong, or the server might not be running yet, or the network doesn't have a path to the server, etc.

If you run your code with the environment variables GRPC_TRACE=all and GRPC_VERBOSITY=DEBUG, the log output should have more information about why it is failing to connect.

[blastshielddown]

Thanks for clarifying @murgatroid99 and apologies for conflating the issues inaccurately. The logging is helping shed more light on the problem. I'll open a separate issue.

[murgatroid99]

Since @grpc/grpc-js version 1.8.0, "Stream refused by server" errors are transparently retried, and should rarely be surfaced.