DaprServer: unable to receive messages >100KB through pubsub with HTTP protocol

[alecor191]

Expected Behavior

Ability to configure the max message when using DaprServer with http protocol. That way we can receive Dapr pubsub messages that are >100KB.

Actual Behavior

When receiving messages >100KB, the configured DaprServer.pubsub.subscribe() callback is not invoked. Looking at the Dapr sidecar logs, they show the following error:

[...] retriable error returned from app while processing pub/sub event 15533a01-f59c-4012-ac00-56bb1f73069b, topic: ..., body: {\"code\":413,\"message\":\"request entity too large\"}. status code returned: 413"

Steps to Reproduce the Problem

1. Create a simple app that uses DaprServer.pubsub.subscribe() and subscribes to a topic. Make sure to configure DaprServer to use http protocol.
2. Send a message >100KB to the subscribed topic
3. Callback is not invoked. Dapr sidecar shows error that the app didn't successfully process the message.

Additional information

• Note that the above scenario works fine when configuring DaprServer to use gRPC protocol (has higher default message limit)

• Debugging a bit through the code, it seems that:

  • The 413 error originates in module raw-body, file index.js, function readStream:

    if (limit !== null && length !== null && length > limit) {
      return done(createError(413, 'request entity too large', {
        expected: length,
        length: length,
        limit: limit,
        type: 'entity.too.large'
      }))
    }
    

  • raw-body fails because limit is set to its default value of 100kb

  • It's set to the default value because Dapr SDK's HTTPServer.js initializes body-parser module, json middleware without specifying a limit. See line 42 in HTTPServer.js -> only type option is set, but not limit:

    this.server.use(body_parser_1.default.json({
        type: [
            "application/json",
            // Support cloudevents 
            // https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md
            "application/cloudevents+json",
            "application/*+json",
        ]
    }));
    

Would it be possible to extend HTTPServer.js so that we could pass the max message size as parameter/option that HTTPServer would then forward to json middleware?

Or if I'm missing something, please let me know how I can configure the max message size. Thanks!

[DeepanshuA]

Thanks @alecor191 for raising this issue. This is really interesting to see, getting our APIs limits' checked. May be we need to start having more load tests checking APIs limits for multiple dimensions. As you seem to have already deep-dived and zeroed down the cause, would you like to go ahead, apply fix, test and raise PR for it?

[XavierGeerinck]

Linked to #330 and PR in progress #356

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[shubham1172]

👋🏻🤖

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[shubham1172]

👋🏻🤖