Feature/rclone export support s3 compatible

[jameshod5]

Closes #8512

Change Description

Added env variables to the lakefs_export tool for S3 provider and S3 endpoint, with the same added to the rclone.conf.temp.

Changed the names of the env variables to improve readability.

Background

Raised the original issue in the Slack (thought it was a bug at first), but quickly noticed it was because our S3 storage was not an AWS S3 storage, and the config only allowed AWS providers.

New Feature

Previously, the export tool was hardcoded to only export to AWS S3 storages. This change adds the functionaility of changing the provider of the S3 storage, along with the endpoint, such that the export can work with non-AWS storages.

Testing Details

I built the image locally for the exporter with my changes.

Using the following command, I was able to export my LakeFS repo successfully to our S3 storage:

docker run \
    --network=host \
    -e S3_ENDPOINT_URL=XXX\
    -e LAKEFS_ACCESS_KEY_ID=XXX \
    -e LAKEFS_SECRET_ACCESS_KEY=XXX \
    -e LAKEFS_ENDPOINT=http://127.0.0.1:8000 \
    -e S3_PROVIDER=Other \
    -e S3_ACCESS_KEY_ID=XXX -e S3_SECRET_ACCESS_KEY=XXX \
    localhost/export:latest \
    example-repo \
    s3:mast/test/lakefs/export \
    --branch="main"
2025/01/29 08:59:57 NOTICE: S3 bucket mast path test/lakefs/export: 0 differences found
2025/01/29 08:59:57 NOTICE: S3 bucket mast path test/lakefs/export: 673 matching files

Checking the endpoint storage:

$ rclone lsf "s3:mast/test/lakefs/export"
30420.zarr/
EXPORT_main_29-01-2025_08:59:23_SUCCESS

Additional info

I do not have an AWS storage to test this funciton with, but the current changes should(!) default to AWS and no specified endpoint if none are supplied. I would appreciate if this could be tested.

This is my first PR for an open-source project like this, I have followed the contribution guide but do let me know if I have missed anything. I will try and make changes, if requested, ASAP.

Contact Details

[email protected]

[jameshod5]

The "Testing Details" section is the extent to which I tested this feature. I assume the CI has more detailed tests?