serverless-plugins icon indicating copy to clipboard operation
serverless-plugins copied to clipboard

DynamoDBStreamReadable prematurely emits `end`

Open asprouse opened this issue 5 years ago • 2 comments

DynamoDBStreamReadable prematurely emits end which causes the serverless-offline-dynamodb-streams plugin to close it's writable. This means that when the plugin is used with TRIM_HORIZON it will only fire handlers on sls offline start.

Looks like this PR to fix LATEST starting point might have broken this: https://github.com/CoorpAcademy/serverless-plugins/pull/37

asprouse avatar May 09 '19 14:05 asprouse

@asprouse Could you give me a way to reproduce ?

godu avatar May 15 '19 14:05 godu

This happens in serverless-offline-dynamodb-streams on line 165 you can observe end being emitted by simply inserting an on call to the readable:

const readable = DynamoDBReadable(
        dynamodbStreamsClient,
        streamARN,
        assign(this.config, {
          shardId,
          limit: tableEvent.batchSize,
          iterator: tableEvent.startingPosition || 'TRIM_HORIZON'
        })
      );

      readable.on('end', () => console.log('Readable End!');

      readable.pipe(
        new Writable({
          objectMode: true,
          write: (chunk, encoding, cb) => {
            this.eventHandler(streamARN, functionName, shardId, chunk, cb);
          }
        })
      );
    }, shards);

Since you are implementing your own writable this isn't an issue here but write streams are supposed to close when end is emitted.

asprouse avatar May 15 '19 21:05 asprouse