nexus icon indicating copy to clipboard operation
nexus copied to clipboard

Delegate pushing the file to S3

Open imsdu opened this issue 2 months ago • 2 comments

Motivation Some files may be large and can not be uploaded directly to S3 through the upload endpoint and has to be delegated to the client for performance and reliability reasons.

One example are jobs in the parallel cluster which will rely on S3-DRA to push the file to S3

Acceptance criteria

  1. The user can provide optional metadata
  2. If the user has no write access to this storage (permission/deprecated project) if the metadata/id is invalid, an error is returned
  3. If the provided input is valid, a temporary he necessary information to push the file (bucket / object key / id) is returned to the user
  4. This payload is signed to make sure it is not modified by the client
  5. Once the file is pushed to the bucket, the client calls Delta again the signed payload, if the payload can be verified and has not reached an expiration date, the file is created in Nexus
  6. The default storage must be a S3 storage

Implementation notes https://connect2id.com/products/nimbus-jose-jwt/examples/jws-json-multiple-signatures

Not in scope

  1. The client having write access to the location in the given bucket
  2. Using presigned urls for other cases
  3. Providing a storage id which is not the default one
  4. Providing a file id
  5. Updating the file

imsdu avatar Apr 22 '24 17:04 imsdu