litestream icon indicating copy to clipboard operation
litestream copied to clipboard

Clarify the Reference on How to Configure an S3 Compatible Replica

Open ky-bd opened this issue 9 months ago • 0 comments

Quote from the reference doc:

The easiest way to configure an S3 replica is to use the url field:

dbs:
  - path: /var/lib/db
    replicas:
      - url: s3://mybkt.litestream.io/db

However, you can break this out into separate fields as well:

dbs:
  - path: /var/lib/db
    replicas:
      - type:   s3
        bucket: mybkt.litestream.io
        path:   db

From my understanding, this indicates that setting the s3://... URL is equivalent to specifying each field or vice versa. However it is not true, since the URL is only parsed for limited providers, and fallback to AWS: https://github.com/benbjohnson/litestream/blob/749bc0d95a5019dfa814ca85809168fb77d15657/s3/replica_client.go#L696-L738

In case the user misconfigured the URL (e.g. putting a Cloudflare R2 endpoint in it), litestream will not only failed to replicate the database, but also wronly send the access_key_id to the AWS. The reference document should clarify that the URL is only available to some listed providers, and specifying each field has a better compatibility.

Related issues:

  • https://github.com/benbjohnson/litestream/issues/591
  • https://github.com/benbjohnson/litestream/issues/491 (The forcePathStyle is not overwritten actually, but the author did find the compatibility issue of the URL)
  • https://github.com/benbjohnson/litestream/issues/372 (Where I found how to configure Cloudflare R2 with litestream properly)

ky-bd avatar Mar 15 '25 06:03 ky-bd