Make S3 part size configurable for backups to prevent 504 errors

[pascal-zarrad]

trafficstars

Current Behavior

Currently there is a predefined part size for the multi-part S3 upload defined.

Definition of part size: app/Http/Controllers/Api/Remote/Backups/BackupRemoteUploadController.php#L18

Splitting: app/Http/Controllers/Api/Remote/Backups/BackupRemoteUploadController.php#L94

5GB is a quite large part and can cause issues on some S3 providers due to the time amount necessary to upload a part (depending on network speed). This sometimes causes timeouts (504 Gateway timeout) due to the time effort necessary to upload a single 5GB part. A configuration for the part size is missing.

Expected Behavior

Some environment variable like "S3_MAX_PART_SIZE" allows to configure some option in bytes (Default: 5GB). This allows to define a reduced chunk size for S3 providers that run into a timeout with larger parts.

In the end, the solution should work like BACKUP_PRESIGNED_URL_LIFESPAN, where there is a default value and an environment variable that allows specifiying this value as needed.

This would allow for example to lower the size of a single part to 1GB instead of 5GB.

Steps to Reproduce

• Setup the panel
• Setup wings
• Create a server that requires a larger amount of disk space (e. g. 20GB)
  • Something like dd can be used to just dump some random data for testing purposes
• Setup S3 as backup provider
  • you must use a service with a short timeout or mediocre network speed
  • alternativly you have to manually limit the speed of the upload
• Create a backup of the large server
• Depending on current conditions, there is a mediocre chance that the upload of a part times out
• -> Backup failed

Panel Version

1.10.1

Wings Version

1.7.0

Games and/or Eggs Affected

No response

Docker Image

No response

Error Logs

No response

Is there an existing issue for this?

• [X] I have searched the existing issues before opening this issue.
• [X] I have provided all relevant details, including the specific game and Docker images I am using if this issue is related to running a server.
• [X] I have checked in the Discord server and believe this is a bug with the software, and not a configuration issue with my specific system.

[pascal-zarrad]

Note that making the value configurable shouldn't be too hard. Tests on my instance where I reduced the part size to 1GB worked well. Backup files are working as expected.

Therefore if the team approves the issue I'd offer to implement the fix / config option. Also documentation has to be updated with BACKUP_PRESIGNED_URL_LIFESPAN (which exists but is missing) and the new part size variable - when added.