litestream icon indicating copy to clipboard operation
litestream copied to clipboard
verified publisher icon benbjohnson

Please provide cli options to specify s3 config LogLevel

Open johnspurlock opened this issue 3 years ago • 7 comments

I'm testing litestream with an upcoming service with an S3-compatible API.

Found myself missing some sort of s3 verbose output mode to help debug signing and payload/encoding issues.

I ended up sticking a hardcoded config.LogLevel inside the config function in s3/replica_client.go and recompiling, but it would be nice to be able to set these flags from nice options.

Also perhaps unrelated, I notice restore describes a -v options in the cli help, but it does not currently appear to be a valid flag at runtime.

johnspurlock avatar Apr 30 '22 17:04 johnspurlock

Thanks, @johnspurlock. Are you running v0.3.8? The logging has been updated on main to be more verbose in general although adding a CLI flag for the S3 log is a good idea too. I created a PR to fix the verbose flag doc: https://github.com/benbjohnson/litestream/pull/367

benbjohnson avatar May 03 '22 13:05 benbjohnson

I don't like brew, so I was running from source (main) -> go install.

The existing logging wasn't quite specific enough, so I added config.LogLevel = aws.LogLevel(aws.LogDebugWithSigning | aws.LogDebugWithHTTPBody) to identify issues I was having with this new S3-compatible service.

This new service also currently only supports the new ListObjectsV2 call, and it appears litestream still uses the older ListObjects v1 (thanks to the older v1 go aws sdk). Not a problem in your tool per se, but a bummer that this blocker exists, as I'm sure many folks will be looking to use litestream with this service once it's publicly available.

johnspurlock avatar May 03 '22 15:05 johnspurlock

I originally used ListObjectsV1 because Google Cloud Storage had an S3-compatible API that couldn't use the V2 version. However, that API actually doesn't work very well (which is why I added the gs replica client). I could switch the s3 client in Litestream to use the V2 version instead now.

benbjohnson avatar May 04 '22 17:05 benbjohnson

Whelp, the new service just implemented ListObjects v1 and a basic litestream replication test seems to work!

Still would be nice to have those s3 debugging options

johnspurlock avatar May 10 '22 00:05 johnspurlock

Ok, the service is live to everyone now: https://blog.cloudflare.com/r2-open-beta/

I've been keeping notes about R2 which has a listing for Litestream

johnspurlock avatar May 11 '22 14:05 johnspurlock