modifyAckDeadline fails with Request payload size exceeds the limit: 524288 bytes when it gets called with too many messages

[mkls]

Environment details

• OS: Kubernetes Engine
• Node.js version: 14.16.0
• npm version: -
• @google-cloud/pubsub version: 2.5.0

Steps to reproduce

1. Consume multiple stream from the same machine where the sum of { flowControl: { maxMessages: <count>} } is at least 3000
2. Subscribe to subscription error events with subscription.on('error', () => {})
3. Occasionally you will get errors like Failed to "modifyAckDeadline" for 2791 message(s). Reason: 3 INVALID_ARGUMENT: Request payload size exceeds the limit: 524288 bytes.

There was already an issue for this kind of error: https://github.com/googleapis/nodejs-pubsub/issues/62 Which also had a solution: https://github.com/googleapis/nodejs-pubsub/pull/65 The issue seems to be that the selected value for MAX_ACK_IDS_PER_REQUEST is probably too large (https://github.com/googleapis/nodejs-pubsub/pull/65/files#diff-22597dec0f27f27ae05243ae8a33dbd2b355f8146698989334bdc000877cd5a0R41)

From our logs it looks like it should be around 2500 instead of 3000.

[mkls]

I've realized that the lib changed since the referenced modification, and now this default 3000 batch size can be overridden with { batching: { maxMessages: } subscription options. Going to test if changing this value actually makes these errors disappear.

[mkls]

setting { batching: { maxMessages: 2500 } seems to make these errors go away. I would suggest changing the default to 2500 here: https://github.com/googleapis/nodejs-pubsub/blob/master/src/message-queues.ts#L185. Or since there is a workaround available I guess this ticket can be closed.

[joohncruz]

In my case I don't believe it is at the time of backing up the messages. How did you come to test this execution?

Error: 3 INVALID_ARGUMENT: Request payload size exceeds the limit: 524288 bytes. at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26) at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:180:52) at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141) at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181) at /usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:182:78 at processTicksAndRejections (node:internal/process/task_queues:76:11)

[feywind]

Given that stack trace, this seems like something that might need a grpc option if we want to support it.

@kamalaboulhosn did I bug you about this one already?

[kamalaboulhosn]

I don't think we need a grpc option here. I think we just need to make sure batches of modacks don't exceed the size limit.

[feywind]

This came up recently as something that's not consistent across language libraries for Pub/Sub, so I'm moving it to feature request.

[feywind]

We have moved this issue to our internal tracking system. However, the issue is considered low-priority at this time. If the issue is ultimately resolved, it will appear in the release notes for the client at that time. If this issue is of higher priority for you, please enter a support ticket for follow-up. Thanks.

(ref bug 267340848)