[FEATURE]: Upload Result files of Training Requests to S3

[dwu359]

Feature Name

Upload Result files of Training Requests to S3

Your Name

Daniel Wu

Description

Part 1 (backend): Once a training request is finished, it should upload its result fields to S3 in dlp-executions-bucket with the location {id}/{filename}. For tabular training specifically, there is dl_results.csv, model.pt, and my_deep_learning_model.onnx. Details on how to generate these files can be found in backend/dl_trainer.py (deprecated) and maybe other files.

Part 2 (full-stack): Add download buttons to download the result files from S3 in the trainspace id page, which is generated by frontend/src/pages/train/[trainspace_id].tsx. Put redux, constants, types, helper components, etc in frontend/src/Train/features/....

A separate SST endpoint may also be needed to generate the presigned url needed to download the files. You may find looking at serverless/packages/functions/src/datasets/user/upload_url.ts useful as an example, as well as instructions from #917.

[github-actions[bot]]

Hello @dwu359! Thank you for submitting the Feature Request Form. We appreciate your contribution. :wave:

We will look into it and provide a response as soon as possible.

To work on this feature request, you can follow these branch setup instructions:

1. Checkout the main branch:

```
 git checkout nextjs
```

2. Pull the latest changes from the remote main branch:

```
 git pull origin nextjs
```

3. Create a new branch specific to this feature request using the issue number:

```
 git checkout -b feature-921
```

Feel free to make the necessary changes in this branch and submit a pull request when you're ready.

Best regards, Deep Learning Playground (DLP) Team

[karkir0003]

For part 2 @dwu359, is this supposed to be doable from the dashboard?

[dwu359]

@karkir0003 Ig that can be a separate PR, but it probably can't be done anyways until GET /trainspace from #914 is complete.

[karkir0003]

i see. so what tasks are independent based on the issues u posted yesterday? @dwu359

[dwu359]

@karkir0003 most of them are independent of each other, it should say on there if another task is needed to be complete beforehand.

[karkir0003]

ok. got it

[abhay-ramesh]

Hello there,

I'd like to recommend trying out the next-s3-uploader package, which I created specifically for Next.js app directories. It simplifies the file upload process seamlessly.

You can explore the documentation for comprehensive details. Should you encounter any challenges or have questions, don't hesitate to get in touch with me or open a GitHub issue here. I'm here to provide assistance and support.

[karkir0003]

@abhay-ramesh link to documentaiton doesnt work

[abhay-ramesh]

@abhay-ramesh link to documentaiton doesnt work

https://next-s3-uploader.abhayramesh.com here you go

[karkir0003]

Link still doesn't work. Also, how is this library different from using AWS JS SDK to do S3 upload? I'm seeing in the docs that only public S3 buckets are supported. However, private S3 support doesn't seem to be present?

[abhay-ramesh]

Link still doesn't work. Also, how is this library different from using AWS JS SDK to do S3 upload? I'm seeing in the docs that only public S3 buckets are supported. However, private S3 support doesn't seem to be present?

Docs hasnt been completed yet, and private bucket works, ryt now this library aims to simplify file uploads have better dev experience

And to use private bucket In generatePresignedUrls function, set privateBucket to true.

generatePresignedUrls(s3Client, keys, bucket, prefix?)

Generates pre-signed URLs for uploading objects to S3.

• s3Client: An instance of the S3 client.
• keys: An array of object keys to upload.
• bucket: The S3 bucket name.
• prefix: (Optional) Prefix for object keys.
• privateBucket: (Optional) Whether the bucket is private. Default is false.

for now the missing docs are in github readme https://github.com/abhay-ramesh/next-s3-uploader

Sorry for the issues caused, will be updating things soon

[abhay-ramesh]

Link still doesn't work. Also, how is this library different from using AWS JS SDK to do S3 upload? I'm seeing in the docs that only public S3 buckets are supported. However, private S3 support doesn't seem to be present?

Docs hasnt been completed yet, and private bucket works, ryt now this library aims to simplify file uploads have better dev experience

And to use private bucket In generatePresignedUrls function, set privateBucket to true.

generatePresignedUrls(s3Client, keys, bucket, prefix?)

Generates pre-signed URLs for uploading objects to S3.

• s3Client: An instance of the S3 client.
• keys: An array of object keys to upload.
• bucket: The S3 bucket name.
• prefix: (Optional) Prefix for object keys.
• privateBucket: (Optional) Whether the bucket is private. Default is false.

for now the missing docs are in github readme https://github.com/abhay-ramesh/next-s3-uploader

Sorry for the issues caused, will be updating things soon

Have updated the docs