Transactional outbox messages not being published in Outbox sample

[stephenchristian-aveva]

In what area(s)?

/area runtime

/area operator

/area placement

/area docs

/area test-and-release

What version of Dapr?

Sample is on 1.12.0, but I have also reproduced the behavior in 1.13.5

Expected Behavior

I was interested in trying out the Transactional Outbox feature which is currently in preview along with Azure Cosmos DB and Azure Service Bus. I downloaded the sample application for Transactional Outbox and made a couple modifications to use Cosmos and Service Bus instead, which I've detailed in the "Steps to Reproduce the Problem" section below. Since Cosmos DB supports Transactions, I was under the impression this should work out of the box.

The expected behavior is that, after the sample makes a state transaction, a message should be published to the configured outboxPublishPubsub and outboxPublishTopic given by the state component.

Actual Behavior

The actual behavior is that no messages are sent.

If I look in Azure Service Bus, I can see that defaultorder-processorordersoutbox, defaultorder-processorordersoutbox, and orders topics are created, but none of those topics have any messages published to them.

Steps to Reproduce the Problem

1. Clone the sample application for Transactional Outbox and replace the following resource files with the given snippets:

resources/pubsub.yaml:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: orderpubsub
spec:
  type: pubsub.azure.servicebus.topics
  version: v1
  metadata:
  - name: connectionString
    value: <CONNECTION-STRING>

resources/statestore.yaml:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore
spec:
  type: state.azure.cosmosdb
  version: v1
  metadata:
    - name: url
      value: <COSMOS-URL>
    - name: masterKey
      value: <COSMOS-MASTER-KEY>
    - name: database
      value: Orders
    - name: collection
      value: Items
    - name: outboxPublishPubsub
      value: orderpubsub
    - name: outboxPublishTopic
      value: orders

(Everything else is unmodified.)

2. Run the applications using dapr run -f . as indicated in the instructions.

3. Observe that the messages that are supposed to be published as a result of the state transaction are not actually published.

I have also tried replacing those with the SQLite storage component and the In-Memory pubsub component, and both of those appear to exhibit the same issue.

Release Note

RELEASE NOTE:

[yaron2]

Looks like there's something wrong with the sample itself, so I'll be moving this to the samples repository.

I was able to validate the outbox feature is working (1.13.5) with the following setup:

import express from 'express';
import bodyParser from 'body-parser';

const APP_PORT = process.env.APP_PORT ?? '3000';

const app = express();
app.use(bodyParser.json({ type: 'application/*+json' }));

app.get('/dapr/subscribe', (_req, res) => {
    res.json([
        {
            pubsubname: "orderpubsub",
            topic: "orders",
            route: "orders"
        }
    ]);
});

app.post('/orders', (req, res) => {
    console.log("order received:", req.body.data);
    res.sendStatus(200);
});

app.listen(APP_PORT);

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore
spec:
  type: state.sqlite
  version: v1
  metadata:
  - name: connectionString
    value: "data.db"
  - name: outboxPublishPubsub
    value: "orderpubsub"
  - name: outboxPublishTopic
    value: "orders"

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: orderpubsub
spec:
  type: pubsub.in-memory
  version: v1
  metadata: []

dapr run --app-id ordersapp --dapr-http-port 3500 --app-port 3000 --resources-path ./components -- node app.js

POST http://localhost:3500/v1.0/state/statestore/transaction

{
  "operations": [
    {
      "operation": "upsert",
      "request": {
        "key": "order1",
        "value": {
            "orderId": "7hf8374s",
            "type": "book",
            "name": "assasin's apprentice"
        }
    }
    }
  ]
}

== APP == order received: map[name:assasin's apprentice orderId:7hf8374s type:book] 

[stephenchristian-aveva]

Hi Yaron! I tried out your Node.js snippet, and it does appear to work on my environment - I'm getting the messages as expected. I also tried out the C# sample again, but this time instead of using the sample app to perform the state transaction, I made the POST http://localhost:3500/v1.0/state/statestore/transaction request you mentioned above, and that also seems to publish the message as expected.

As a result I think the issue lies somewhere on the order-processor side, either in the sample app itself or possibly in the Dapr.Client library for .NET. One thing I will note is that we did write our own C# app, separate from the sample, that did experience this same issue, but that also used the client.ExecuteStateTransactionAsync(...) method from the Dapr.Client library to make the state transaction.

[yaron2]

Hi Yaron! I tried out your Node.js snippet, and it does appear to work on my environment - I'm getting the messages as expected. I also tried out the C# sample again, but this time instead of using the sample app to perform the state transaction, I made the POST http://localhost:3500/v1.0/state/statestore/transaction request you mentioned above, and that also seems to publish the message as expected.

As a result I think the issue lies somewhere on the order-processor side, either in the sample app itself or possibly in the Dapr.Client library for .NET. One thing I will note is that we did write our own C# app, separate from the sample, that did experience this same issue, but that also used the client.ExecuteStateTransactionAsync(...) method from the Dapr.Client library to make the state transaction.

Interesting - I'll take a look at the C# sample and get to the bottom of this.