PubSub with large message does not trigger subscriber

[brunoshine]

Expected Behavior

Publishing a large message that, for instance, has a base64 image string should trigger the subscriber.

Publisher

import { DaprClient, CommunicationProtocolEnum } from '@dapr/dapr';
import * as image from './image.js'

const client = new DaprClient({
    communicationProtocol: CommunicationProtocolEnum.GRPC,
    maxBodySizeMb: 10,
});

console.log("Publishing...")
setInterval(async () => {
    await client.pubsub.publish("pubsub", "test", {
        payload: image.data.image, //base64 image string

    }, {
        metadata:{
            ttlInSeconds:"1"
        }
    });
    console.log("published")
}, 3000);

Subscriber:

import { DaprServer, CommunicationProtocolEnum, DaprPubSubStatusEnum } from '@dapr/dapr';
const server = new DaprServer({
    communicationProtocol: CommunicationProtocolEnum.GRPC,
    clientOptions: {
        communicationProtocol: CommunicationProtocolEnum.GRPC
    }
});

const configs = await server.client.configuration.get("configuration", ["key"])

server.pubsub.subscribe("pubsub", "test", async (data, headers) => {
    console.log("Subscriber received: " + JSON.stringify(data));
    return DaprPubSubStatusEnum.SUCCESS;
}
);

await server.start();

command:

"scripts": {
    "client": "dapr run --app-id checkout --dapr-http-port 3502   --app-protocol grpc --resources-path .dapr/components/ -- node client.js",
    "server": "dapr run --app-id order-processing --app-port 5002 --dapr-http-port 3501   --app-protocol grpc --resources-path .dapr/components/ -- node server.js"
  },

Redis PubSub Component Definition

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub
spec:
  type: pubsub.redis
  version: v1
  metadata:
  - name: redisHost
    value: localhost:6379
  - name: redisPassword
    value: ""

Actual Behavior

Although the message get published to Redis stream the subscriber does not get triggered. In this test we are using a base64 image that is around 150kb.

If I just set the payload to payload: "" the subscriber gets triggered.

Steps to Reproduce the Problem

• Convert an image to a base64 string and pass the string into the payload on the publisher.
• Execute both the client and the server npm scripts.
• Published messages should be on the redis stream named test.
• Subscriber is not triggered.
• Stop both client and server.
• Update the payload to be empty payload:""
• Execute both the client and the server npm scripts.
• Published messages should be on the redis stream named test.
• Subscriber is triggered.

[brunoshine]

It seems that this also happens with bindings. I have a MQTT Input binding where I have placed a 132kb image base64 string on the body of message topic, and with this the binding does not get triggered. If I remove the base64 string then the binding gets called.

Any thoughts?

thanks

[shubham1172]

I am able to reproduce this locally. Tried with both gRPC and HTTP. The message does gets published, and even comes to Dapr (as part of debug logs):

DEBU[0009] Processing Redis message 1709193127018-0      app_id=subscriber component="pubsub (pubsub.redis/v1)" instance=Shubhams-MacBook-Pro-3.local scope=dapr.contrib type=log ver=1.12.5

However, it never reaches the application. I am using a 1MB payload.

• Increasing the maxBodySizeMb does not work too.
• I also tried using a Go app like this, it can't receive the large payload either.
• Then I set to try this with vanilla JavaScript, and it works with the right bodyParser settings

import express from 'express';
import bodyParser from 'body-parser';

const APP_PORT = process.env.APP_PORT ?? '8080';

const app = express();
app.use(bodyParser.json({ type: 'application/*+json', limit: "50mb" }));
// app.use(bodyParser.urlencoded({ limit: "50mb" }));

app.get('/dapr/subscribe', (_req, res) => {
    res.json([
        {
            pubsubname: "pubsub",
            topic: "test",
            route: "/events"
        }
    ]);
});

// Dapr subscription routes orders topic to this route
app.post('/events', (req, res) => {
    console.log("Subscriber received:", req.body.data);
    res.sendStatus(200);
});

app.listen(APP_PORT);

We also have the same configuration in JS-SDK for bodyParser, so this needs further investigation why HTTP (and then also gRPC for some reason) is not working.

[brunoshine]

Hi @shubham1172 is there any temporary workaround? Thanks

[shubham1172]

@brunoshine I don't have a workaround for now, but I will further investigate this and update this issue. Thanks for your patience.

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.