S3Store: Random NoSuchUpload errors

[Acconut]

A user of tus-node-server contacted me and described an issue they are having with tus-node-server in production: Some tus uploads randomly fail with a NoSuchUpload error when being resumed. Some more details:

• The way to reproduce this is to start uploading multiple files concurrently to tus-node-server (I believe they were using Uppy, but I am not 100% sure). Then pause all uploads. Then resume all unfinished uploads. For some uploads, the HEAD request will fail due to a NoSuchUpload error from S3.
• This approach did not always produce the error. Sometimes it worked, sometimes it didn't. We couldn't make a pattern on which the error depends.
• They were using a single tus-node-server without any load balancing. So all requests went to the same tus-node-server instance.
• I do not have the exact stack trace, but it was not very informative. It only contained references to the AWS SDK and not to the internals of tus-node-server. To give you a better feeling, the trace looked similar to this (adapted from https://github.com/tus/tus-node-server/issues/133):

{ NoSuchUpload: null
    at Request.extractError (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/services/s3.js:560:35)
    at Request.callListeners (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/sequential_executor.js:109:20)
    at Request.emit (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/sequential_executor.js:81:10)
    at Request.emit (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/fbs/Documents/example/express-tus-minio-server/node_modules/aws-sdk/lib/sequential_executor.js:119:18)
  message: null,
  code: 'NoSuchUpload',
  region: null,
  time: 2022-07-16T07:06:23.049Z,
  requestId: '155CCAA517A55A98',
  extendedRequestId: undefined,
  cfId: undefined,
  statusCode: 404,
  retryable: false,
  retryDelay: 44.82171261258596 }

• They were also using a custom namingFunction. I am not sure if this is related, but the logic behind it did not seem problematic.

All in all, I know that this is not a very conclusive bug report. However, it could point to bigger problems of running tus-node-server in production. It might be an indicator of a race condition in tus-node-server (similar to https://ninio.ninarski.com/2019/01/18/aws-s3-and-its-informative-errors-404-nosuchupload/).