powertools-lambda-typescript icon indicating copy to clipboard operation
powertools-lambda-typescript copied to clipboard

Docs: improve documention regarding the change to async when migrating from `BatchProcessorSync` to `BatchProcessor`

Open Tohaker opened this issue 1 month ago • 6 comments

Expected Behavior

Migrating from BatchProcessorSync and processPartialResponseSync to BatchProcessor and processPartialResponse with the processInParallel flag set to false should make no change to behaviour.

Current Behavior

Using BatchProcessor and processPartialResponse with the processInParallel flag set to false processes records in parallel, which means we lose data where before we didn't.

Code snippet

const processor = new BatchProcessor(EventType.SQS);

const recordHandler =
  (qtyMap: Record<string, number>) => (record: SQSRecord) => {
    const { product } = extractDataFromEnvelope(
      record,
      'powertools_json(body)',
    );

    const productCode = product.code;
    const items = product.items;

    for (const item of items) {
      const id = item.itemId;
      const qty = Math.ceil(item.quantity);

      if (!qtyMap[id]) {
        qtyMap[id]= [
          qty,
        ];
      } else {
        qtyMap[id]?.push(
          qty,
        );
      }
    }
  };

const handler = (event: SQSEvent) => processPartialResponse(
    event,
    recordHandler(elementQuantityMap),
    processor,
    { processInParallel: false }
  );

Steps to Reproduce

With some records that have a list of item IDs and quantities, you can run the above and see that records get overwritten because they're being process in parallel still, even when specified not to.

e.g.

{
  "product": { "id": "1" "items": [{
  itemId: 'sku1',
  quantity: 1.00000000, 
},
{
  itemId: 'sku2',
  quantity: 1.08300000, 
},
{
  itemId: 'sku3',
  quantity: 5.00000000, 
},
{
  itemId: 'sku4',
  quantity: 4.00000000, 
},] }

{
  "product": { "id": "2" "items": [{
  itemId: 'sku1',
  quantity: 1.00000000, 
},
{
  itemId: 'sku2',
  quantity: 1.08300000, 
},
{
  itemId: 'sku3',
  quantity: 5.00000000, 
},
{
  itemId: 'sku4',
  quantity: 4.00000000, 
},] }

I'd expect a map like;

{
  "sku1": [1, 1],
"sku2: [2, 2]
... etc
}

But instead I just get something like;

{
  "sku1" : [1],
"sku2: [2]
... etc, some keys even misisng
}

Possible Solution

No response

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

22.x

Packaging format used

Lambda Layers

Execution logs


Tohaker avatar Nov 17 '25 11:11 Tohaker

Thanks for opening your first issue here! We'll come back to you as soon as we can. In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

boring-cyborg[bot] avatar Nov 17 '25 11:11 boring-cyborg[bot]

Hi @Tohaker, thanks for opening this issue.

Please allow a couple hours for me to look into the report and formulate an answer.

dreamorosi avatar Nov 17 '25 11:11 dreamorosi

Hi @Tohaker, I tried to reproduce the scenario in https://github.com/sdangol/repro-4760/tree/main. Let me know if I'm doing this correctly.

I've created a simple lambda function which has the SQS Queue as the source. When I send some message to the queue using

QUEUE_URL=https://your-queue-url node src/sendMessage.js

If there are multiple records in a single batch, I am getting the map as expected in my logs.

{
    "level": "INFO",
    "message": "Processed Product 3",
    "timestamp": "2025-11-18T16:13:56.543Z",
    "service": "service_undefined",
    "sampling_rate": 0,
    "xray_trace_id": "1-691c9b44-fc0afb9c4316ca4fcd44a9fd",
    "qtyMap": {
        "sku1": [
            1,
            1,
            1
        ],
        "sku2": [
            2,
            2,
            2
        ],
        "sku3": [
            5,
            5,
            5
        ],
        "sku4": [
            4,
            4,
            4
        ]
    }
}

Is it that you're looking to persist these across multiple Lambda invocations?

sdangol avatar Nov 18 '25 16:11 sdangol

Your reproduction looks the same as mine... Not sure where the differences between ours is. At the very least, our unit tests don't behave the same between BatchProcessor and BatchProcessorSync. I'll dig a bit deeper to find why you can't replicate it.

Tohaker avatar Nov 19 '25 08:11 Tohaker

Hi @Tohaker, thanks - let us know if you find it.

We're also more than happy to jump on a call with you / your team to dive deeper into it, if you'd prefer.

dreamorosi avatar Nov 19 '25 09:11 dreamorosi

Thanks @Tohaker for meeting with us today. We were able to debug the issue and found out that it was missing the await operator in processPartialResponse. Thanks for the feedback about the documentation. I'll mark this as documentation improvement, and we will try to clarify about the asynchronous nature of BatchProcessor

sdangol avatar Dec 03 '25 10:12 sdangol