[Suggestions] Features for storage optimization

[Didayolo]

Hard limits (bundle upload max size, submissions max size, duration of computation on default queue, etc.) should be configurable in the .env file in order to be adapted to each instances.

My Datasets

• [x] When a user delete a dataset from My Datasets page, erase it from storage (garbage collection, with retro-action)
• [x] Warn for unused datasets (e.g. display in red)
• [ ] Button to delete all unused datasets
• [ ] Max upload size for bundles and datasets (e.g. 10 GB)

Redundancy

• [ ] Delete redundant files and add sym links (pointers) Users upload the same files many times: submissions, bundles, etc.
• [ ] Can we erase ZIP archives after they got unzipped? (uploads directory)
• [ ] Problem with files of size 0 #3004
• [ ] Remove uploads (zip files uploaded) after x months. These are the files that take up the most storage and have the least need to remain permanent.

Submissions

• [ ] Button to erase all failed submissions
• [ ] Button to erase all the submissions while keeping the latest 3 submissions per participant in each phase
• [ ] Automatic purge of failed submissions?
• [ ] Submissions should count in the storage quota, and users should have an interface to manage their submissions from all competitions and delete them.
• [x] Submission size limitation (edit menu): the default value is 0, which means "no limitation". We may want to put a reasonable value (e.g. 30 MB) to limit submissions by default. #3052
• [ ] Cool down before being able to re-submit
• [ ] Max submission size (e.g. 2 GB to avoid 50 GB submissions...)

Others

• [ ] Quota : impose a storage limitation to users (also for security reasons)
• [ ] Monitoring tools
• [ ] Glacial Storage? #2836
• [ ] Notification to admin when storage is low #2291
• [ ] Limitation on the number of competitions allowed by user

Compute workers storage

• [ ] Make compute workers prune docker images when needed
• [x] #2996

[dtuantran]

I'd like to add this issue https://github.com/codalab/codalab-competitions/issues/3004