S3 API Doesn't Respect linode Private Bucket Endpoint After the Upload

[mustafa519]

Version:

• listmonk: listmonk/listmonk:v3.0.0-amd64 docker image
• OS: Ubuntu 22

Description of the bug and steps to reproduce: Hello, I am trying to setup listmonk.

I set the linode s3 credentials like below screenshot. Linode object storage endpoint works well, however previewing it is broken. It tries to get the media from AWS s3 endpoint rather than the linode endpoint. It's happening on the campaign add attachment step.

E.g: https://mybucket.s3.amazonaws.com/folder/thumb_388291.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=RN7IENFKQAG4IU64E7TW%2F20240313%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20240313T210501Z&X-Amz-Expires=601200&X-Amz-SignedHeaders=host&X-Amz-Signature=dca06ed307f4beacf5da282ab3234d98c746165756fc1084435417bbbcb4e9f7

I am not familiar with the go language, so I couldn't find the broken part of the code.

I hope those information helps!

Edit: Just figured it out, on this issue #669 mentioned the same bug.

Screenshots:

[knadh]

Hi @mustafa519. #669 was closed as the issue couldn't be replicated.

1. Perhaps the bucket type should be public?
2. Tried setting the custom public URL?

[mustafa519]

Hello @knadh

I currently ended up using the public bucket type. Public bucket or providing custom public url works well. However, when I set a private bucket, it uses bucketName.s3.amazonaws.com endpoint, doesn't respect my custom endpoint.

Then I used aws bucket instead linode, it still didn't work.

In Summary:

• Uploading works in any case.
• Public bucket type works normally.
• When I set different endpoint(linode) other than aws, it tries to fetch from aws.
• Private buckets don't work with neither aws nor different object storage services.

Those all I can provide as the details.

Thank you for the investigation.

[knadh]

That's very helpful. Sounds like a URL generation issue when type=private. Will investigate this.

[osmantuna]

Same setup here. I am using on prem cloudian s3 service. I get exact same behaviour when i set up private bucket. Uploads just fine but generates "amazonaws.com" url.

https://mybucket.s3.amazonaws.com/xxx.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=00e7763fa067f25c71d0%2F20240326%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240326T074639Z&X-Amz-Expires=601200&X-Amz-SignedHeaders=host&X-Amz-Signature=928a740c6ea92e7d6ffb4914ca57cfa8d1c9481a7fc4beb4584044eeab94226f

[knadh]

hm, when the bucket type is empty AND there is no public URL set (the "Custom Public URL" field in settings), then a pre-signed URL is generated.

https://github.com/knadh/listmonk/blob/1bf7e362bf6bee23e5e2e15f8c7cf12e23860df6/internal/media/providers/s3/s3.go#L91

Assuming that the S3-compatible private endpoints also support generating signed URLs, the https://github.com/rhnvrm/simples3 lib should ideally support custom URLs.

@rhnvrm could you confirm whether setting the Endpoint field to the root of the custom backend will work as intended?

[rhnvrm]

Yeah .Endpoint should work.

https://github.com/rhnvrm/simples3/blob/master/presigned.go#L58-L60

There is also a helper:

https://github.com/rhnvrm/simples3/blob/master/simples3.go#L244-L260

[nemunaire]

The simples3 library seems to want the "Endpoint" to be given as argument: https://github.com/rhnvrm/simples3/blob/master/presigned.go#L59

But as simples3 appends the bucket name in front of the hostname (which is definitely not the endpoint), this will probably not doing anything good.

I think simples3 code needs to be reworked to extract the expected hostname from the real Endpoint if given in the s3 struct, and fallback to the current code if no Endpoint is given.

[nemunaire]

I made a PR in simples3 to fix the presign function, after it is merged, there is nothing to change on listmonk side, the current code is sufficient.