Improve documentation and s3 config for self hosting

[maxirozay]

Improve documentation and s3 config

Link

Self-hosting doc

Describe the problem

The doc says to setup env variable that doesn't seem to be used.

In the self-hosted documentation for s3 it says to setup

STORAGE_BACKEND=s3
GLOBAL_S3_BUCKET=name-of-your-s3-bucket
REGION=region-of-your-s3-bucket

but nothing about the access key, the secret key and the endpoint.

Then it point to a link to an .env file with the following value

# S3 Protocol
#######################################
S3_PROTOCOL_ACCESS_KEY_ID=b585f311d839730f8a980a3457be2787
S3_PROTOCOL_ACCESS_KEY_SECRET=67d161a7a8a46a24a17a75b26e7724f11d56b8d49a119227c66b13b6595601fb

#######################################
# Storage Backend Driver
# values: s3 | file
#######################################
STORAGE_BACKEND=s3

#######################################
# S3 Backend
#######################################
STORAGE_S3_BUCKET=supa-storage-bucket
STORAGE_S3_MAX_SOCKETS=200
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_FORCE_PATH_STYLE=true
STORAGE_S3_REGION=us-east-1

AWS_ACCESS_KEY_ID=supa-storage
AWS_SECRET_ACCESS_KEY=secret1234

But those properties are not used in docker-composed.yml and in this file it says "# To use S3 backed storage: docker compose -f docker-compose.yml -f docker-compose.s3.yml up" but docker-compose.s3.yml seems to use minio so and is not mentioned in the doc.

The cloud based doc talk about a ~/.aws/credentials file which would be nice to use to config the self-hosted but even better if we can put everything in the .env.

So basically you can find many differents ways to setup s3 but none seems to work for self-hosting unless you gather all the info and edit the docker-compose.yml file by trying all variables and hoping for the best.

Describe the improvement

The docs should says to configure the env variables which would be in the .env file already (they could be commented). Those env should be used in docker-compose.yml. AWS_ACCESS_KEY_ID should be only S3_PROTOCOL_ACCESS_KEY_ID, unless it is needed to have both.

.env

STORAGE_BACKEND=s3
STORAGE_S3_BUCKET=supa-storage-bucket
STORAGE_S3_MAX_SOCKETS=200
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_FORCE_PATH_STYLE=true
STORAGE_S3_REGION=us-east-1
S3_PROTOCOL_ACCESS_KEY_ID=b585f311d839730f8a980a3457be2787
S3_PROTOCOL_ACCESS_KEY_SECRET=67d161a7a8a46a24a17a75b26e7724f11d56b8d49a119227c66b13b6595601fb

docker-compose.yml

STORAGE_BACKEND: ${STORAGE_BACKEND}
STORAGE_S3_REGION: ${STORAGE_S3_REGION}
STORAGE_S3_ENDPOINT: ${STORAGE_S3_ENDPOINT}
STORAGE_S3_BUCKET: ${STORAGE_S3_BUCKET}
STORAGE_S3_ACCESS_KEY_ID: ${STORAGE_S3_ACCESS_KEY_ID}
STORAGE_S3_SECRET_ACCESS_KEY: ${STORAGE_S3_SECRET_ACCESS_KEY}

Thank you for your great work. I can do the PR but I want to make sure I understand things correctly and I that I can setup my own s3 (with infomaniak/openstack) before.

[maxirozay]

In case somebody needs the correct parameters for docker-compose it's:

      STORAGE_BACKEND: s3
      REGION: us-east-1
      GLOBAL_S3_BUCKET: <bucketName>
      STORAGE_S3_ENDPOINT: https://s3.pub2.infomaniak.cloud
      STORAGE_S3_FORCE_PATH_STYLE: "true"
      AWS_ACCESS_KEY_ID: <accessKey>
      AWS_SECRET_ACCESS_KEY: <secretKey>

To support resummable upload add:

kong environment:
      KONG_PORT_MAPS: 443:8000
...
storage environment:
      TUS_ALLOW_S3_TAGS: "false"
      REQUEST_ALLOW_X_FORWARDED_PATH: "true"
      NODE_ENV: production

[jkwasniak]

The documentation around external S3 storage integration into a self-hosted Supabase is quite incomplete and confusing.

My main question is: Is it possible to integrate an external S3-compatible service (e.g., Amazon S3, Hetzner, etc.) into a self-hosted Supabase setup without using the docker-compose.storage.yml file (i.e., without using MinIO)?

If this is supported, could someone please provide a working example of a docker-compose.yml and .env configuration for this setup?

Thanks in advance!

[maxirozay]

Look at my previous post. You don't need to touch the .env just change or add the variables from the first block in the storage environment of the default docker-compose.yml, with your own bucket, endpoint, access and secret. The second block is for resummable upload you can copy and paste those env in their block.