[grpc-js] Error: 13 INTERNAL: Received RST_STREAM with code 0

[dnalborczyk]

Problem description

grpc-js client is throwing the following exception under high load:

ERROR Error: 13 INTERNAL: Received RST_STREAM with code 0
    at Object.callErrorFromStatus (/repro/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:342:141)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181)
    at Http2CallStream.outputStatus (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:117:74)
    at Http2CallStream.maybeOutputStatus (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:156:22)
    at Http2CallStream.endCall (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:142:18)
    at ClientHttp2Stream.<anonymous> (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:420:22)
    at ClientHttp2Stream.emit (events.js:314:20)
    at emitCloseNT (internal/streams/destroy.js:81:10) {
  code: 13,
  details: 'Received RST_STREAM with code 0',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Reproduction steps

I have a repro repository in the works and will update with the link.

Environment

• OS name, version and architecture: macOS Big Sur v11.0 [20A5343j]
• Node version: v14.7.0 (seems to be irrelevant, also reproduces with v10.22.0 and v12.18.3
• Node installation method: nvs
• If applicable, compiler version: n/a
• Package name and version: grpc-js v1.1.3

Additional context

[racerxdl]

Same issue here. With nodejs v10, and with no load.

[murgatroid99]

You say grpc-js throws this error. This is a standard request status error, which you should be handling. If your method has a response stream, the error will be emitted in the error event (handled with call.on('error', ...)).

Regarding the error itself, the message indicates that the server unexpectedly closed the stream before the client considered it to be complete. The code 0 indicates that the server did consider the stream to be complete. I have a guess about what that means, and an reproduction code you can provide should help verify that.

[dnalborczyk]

thank you @murgatroid99

I'll have a repro repo up in a bit. the exception is not being thrown when one switches out the client to use the native deprecated grpc module.

[civilizeddev]

I faced the same problem.

You need to make sure your call method path starts with '/' just like below.

/<package>.<service>/<method>

Having only <method> in the path might cause this problem.

[SpencerHehl]

any further progress on this? we have an issue with this error code as well.

[murgatroid99]

As I said before, reproduction code for this bug would help figure out what's happening. If you can share some that would be great.

[chroth7]

if it helps, we had these errors when running on GKE (google cloud's kubernetes engine), when under high load and resulting memory pressure. Essentially, our upper limit on the memory was too low, which caused the workloads to use more memory than its limit, which in turn put memory pressure on the nodes, which in turn made it fall apart...
In the specific case, we had a PubSub listener with surges at times.

[SpencerHehl]

@chroth7 you basically just described our setup as well. We are running on GKE, with relatively high load, and subscribing to a long standing stream. I'll go back and check resource monitoring, but I was considering running just on a high-spec GCE to see if it might be a resource thing. Thanks for the thoughts and I'll post back in here if that helps out for us as well.

[eli-stewart]

I am seeing this as well. I am running a Typescript React App with a node.js server. The server is throwing this error fairly regularly. Something like once an hour. I also get this similar error: @firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server but about 20% as frequently.

I had a crash on my site on Monday where I was not able to read or write the database and I am trying to figure out if its related.

These are the 3 types of errors I am receiving of this type: @firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 2

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 0

[castarco]

Does anybody know where to lookup for what each code means?

[dnalborczyk]

@castarco https://grpc.github.io/grpc/core/md_doc_statuscodes.html

[davebarkerKount]

error text Error: 14 UNAVAILABLE: Stream refused by server at Object.callErrorFromStatus (<local repo>/node_modules/@grpc/grpc-js/src/call.ts:81:24) at Object.onReceiveStatus (<local repo>/node_modules/@grpc/grpc-js/src/client.ts:343:36) at Object.onReceiveStatus (<local repo>/node_modules/@grpc/grpc-js/src/client-interceptors.ts:430:34) at Object.onReceiveStatus (<local repo>/node_modules/@grpc/grpc-js/src/client-interceptors.ts:392:48) at <local repo>/node_modules/@grpc/grpc-js/src/call-stream.ts:299:24 at processTicksAndRejections (node:internal/process/task_queues:78:11) { code: 14, details: 'Stream refused by server', metadata: Metadata { internalRepr: Map(0) {}, options: {} }

happens calling a gRPC/go micro-service intermittently after a wait. So use case is like this:

1. setup client.
2. make several calls to service to setup integration test.
3. hard wait 90 secs
4. make another call to server via client for test assert. At this step approx. one out of 10 times will get one of the errors described above or this one.

grpc-js: 1.4.2

[dc185308]

FWIW I have observed this specific error message when the server took longer than 60 seconds to respond. At 60s the client throws this error. I am using default settings for everything whatever they may be (e.g. no custom deadline specified).

[weimch]

If it will help, I'm developing grpc protocol using nghttp2-lib.(C++)

I found that if server sent ping to client, but client didn't response ping-ack to server, the server will send RST_STREAM with code 0 after sending the trailer.

[murgatroid99]

The server is always supposed to send an RST_STREAM with code 0 after sending trailers. That is expected and should not cause an error. This error occurs when the server sends an RST_STREAM with code 0 without sending trailers.

[weimch]

The server is always supposed to send an RST_STREAM with code 0 after sending trailers. That is expected and should not cause an error. This error occurs when the server sends an RST_STREAM with code 0 without sending trailers.

@murgatroid99 I enabled debug trace in gRPC framework. Below is the difference between grpc-client and my client(called as nghttp2-grpc-client for convenience). Sames RST_STREAM with code 0 will not send if ping is normally response from client.

[murgatroid99]

I checked the spec again and I was mistaken. An RST_STREAM is not needed/expected after the server sends trailers.

I notice in your first trace log that the line that corresponds to sending the RST_STREAM comes 6 seconds after the line that corresponds to sending trailers. And it seems to be more related to the BDP probe than to the individual stream. So I would guess that if the stream took longer to complete, the server would send the RST_STREAM before the trailers.

[fauzan-n]

For you guys how are using Kubernetes.

Please check your Istio Sidecar. In my case, the traffic to GRCP client was blocked by the Istio Sidecar.

After I disable the Istio Sidecar with annotation, it's working fine now.

[ziedHamdi]

I get the same error with firebase emulator.

Here are my firebase dependencies :

zied@zied-N752VX:~/Work/WS/linkedinCv-back$ pnpm view firebase dependencies
{
  '@firebase/app': '0.9.7',
  '@firebase/app-compat': '0.2.7',
  '@firebase/app-types': '0.9.0',
  '@firebase/auth': '0.22.0',
  '@firebase/auth-compat': '0.3.7',
  '@firebase/database': '0.14.4',
  '@firebase/database-compat': '0.3.4',
  '@firebase/firestore': '3.10.0',
  '@firebase/firestore-compat': '0.3.6',
  '@firebase/functions': '0.9.4',
  '@firebase/functions-compat': '0.3.4',
  '@firebase/installations': '0.6.4',
  '@firebase/installations-compat': '0.2.4',
  '@firebase/messaging': '0.12.4',
  '@firebase/messaging-compat': '0.2.4',
  '@firebase/storage': '0.11.2',
  '@firebase/storage-compat': '0.3.2',
  '@firebase/performance': '0.6.4',
  '@firebase/performance-compat': '0.2.4',
  '@firebase/remote-config': '0.4.4',
  '@firebase/remote-config-compat': '0.2.4',
  '@firebase/analytics': '0.9.5',
  '@firebase/analytics-compat': '0.2.5',
  '@firebase/app-check': '0.6.4',
  '@firebase/app-check-compat': '0.3.4',
  '@firebase/util': '1.9.3'
}

My initialization code is the following

import { initializeApp } from "firebase/app";
// import { getFirestore, connectFirestoreEmulator } from 'firebase-admin/firestore';

import { getFirestore, connectFirestoreEmulator } from "firebase/firestore";

// firebaseApps previously initialized using initializeApp()
// TODO: Add SDKs for Firebase products that you want to use
// https://firebase.google.com/docs/web/setup#available-libraries

// Your web app's Firebase configuration
// For Firebase JS SDK v7.20.0 and later, measurementId is optional
const firebaseConfig = {
	apiKey: "XXXXX",
	authDomain: "linkedincv-bdbec.firebaseapp.com",
	projectId: "linkedincv-bdbec",
	storageBucket: "linkedincv-bdbec.appspot.com",
	messagingSenderId: "472466845815",
	appId: "XXXXX",
	measurementId: "XXXXX",
};

// Initialize Firebase
const app = initializeApp(firebaseConfig);
// const db = getFirestore(app);

const db = getFirestore(app);
connectFirestoreEmulator(db, 'localhost', 4000);
// const analytics = getAnalytics(app);

export {db}

The code that triggers the error is the following:

import {db} from '../../../lib/middleware/firebaseInit'
import { doc, setDoc } from "firebase/firestore";

export async function POST({request}) {
	try {
		const payload = await request.json();
		const uniqueUrl = payload.uniqueUrl;

		// Store the 'uniqueUrl' field in Firestore
		const docRef = await setDoc(doc(db, "json_cv", "testId"), payload)
		// Return the document ID of the stored object
		return {
			body: JSON.stringify({ documentId: docRef.id }),
			headers: { 'Content-Type': 'application/json' },
			status: 200
		};
	} catch (error) {
		// Handle any errors that occur during processing
		console.error(error);
		return {
			body: JSON.stringify({ error: 'Failed to store unique URL' }),
			headers: { 'Content-Type': 'application/json' },
			status: 500
		};
	}
}

I saw some people updated firebase-functions to get it to work, but it seems this doesn't apply for me as I'm already at version 9.

Here's the error I receive: [2023-04-13T06:09:01.286Z] @firebase/firestore: Firestore (9.19.1): GrpcConnection RPC 'Write' stream 0x48b21b62 error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Protocol error

[lemndev]

had the same issue (Error: 13 INTERNAL: Received RST_STREAM with code 0) but it turns out that the service I was calling was not accepting calls from outside the VPC, updated it to accept all traffic and issue was sorted.