`sst start`: invocations stuck in `Pending` when there are a lot of concurrent requests

[fwang]

Requests:

• Kevin https://discord.com/channels/983865673656705025/1000149490214375534

[kevinggrimm]

Hey @fwang,

I'm noticing this behavior when inserting individual items into a table. Oddly enough, Pending is only thrown for 3 of 5 SNS subscribers; the other two run as expected.

Any ideas? I'll try clearing the .sst folder to see if that does the trick.

Adding relevant code below:

// MyStack.js
const table = new sst.Table(this, "Table", {
  defaultFunctionProps: {
    timeout: 600,
  },
  fields: {
    PK: "string",
    SK: "string",
    GSI1PK: "string",
  },
  primaryIndex: { partitionKey: "PK", sortKey: "SK" },
  globalIndexes: {
    GSI1: {
      partitionKey: "GSI1PK",
      sortKey: "SK",
    },
  },
  stream: dynamodb.StreamViewType.NEW_IMAGE,
  consumers: {
    mainStream: {
      function: {
        handler: "src/consumers/mainStream.main",
      },
    },
  },
});

// Utilities to (1) pull the event source and (2) override to items starting w/ PK_Prefix
const mainStreamEventSource = getEventSourceMapping(table, "mainStream");
addEventSourceKeyOverride(mainStreamEventSource, "PK_Prefix");

const mainTopic = new sst.Topic(this, "MainTopic", {
  defaultFunctionProps: {
    environment: {
      TABLE_NAME: table.tableName,
    },
    permissions: [table],
    timeout: 600,
  },
  subscribers: [
    // Throws 'Pending' in SST Console
    {
      function: {
        handler: "src/subscribers/aggregations/pipeline_one.main",
        permissions: ["sns"],
      },
    },
    // Throws 'Pending' in SST Console
    {
      function: {
        handler: "src/subscribers/aggregations/pipeline_two.main",
        permissions: ["sns"],
      },
    },
    // Throws 'Pending' in SST Console
    {
      function: {
        handler: "src/subscribers/aggregations/pipeline_three.main",
        permissions: ["sns"],
      },
    },
    // Subscriber executes as expected
    {
      function: {
        environment: {
          FREE_TOPIC_ARN: freeTopic.topicArn,
        },
        handler: "src/subscribers/free/getFreeUsers.main",
        permissions: ["sns"],
      },
    },
    // Subscriber executes as expected
    {
      function: {
        environment: {
          PAID_TOPIC_ARN: paidTopic.topicArn,
        },
        handler: "src/subscribers/paid/getPaidUsers.main",
        permissions: ["sns"],
      },
    },
  ],
});


// mainStream.js
import handler from "../libs/handler-lib";
import sns from "../libs/sns-lib";

export const main = handler(async (event, context) => {
    // 1. Extract payload
    const { Records } = event;
    // 2. Publish to an SNS topic 
    try {
      const r = await sns.publish({
        TopicArn: process.env.MAIN_TOPIC_ARN,
        Message: JSON.stringify({Records})
      });
      console.log(r)
    } catch (err) {
      console.log(err)
    }
});

[kevinggrimm]

Dax and I had a couple messages back and forth in the discord; this appears to be a dependency issue with one of my imports. Not yet sure which dependency is throwing the error, though I am going to refactor to isolate the methods I need for each subscriber and expect that will resolve.

ReferenceError: __dirname is not defined in ES module scope This file is being treated as an ES module
 because it has a '.js' file extension and '/my_project/.sst/artifacts/admin-my-project-MyStack-PaidAlertTopic-Subscriber_PaidAlertTopic_emailPaidUsers/package.json' 
contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.

Also worth noting that the Pending messages disappeared after a migration from V0 to V1 constructs - I think that is the main takeaway here. The Error responses were only emitted after updating. May not even be worth digging into since we should all be migrating to V1 as is.

Feel free to close out!

[thdxr]

Fixed in v2