mastodon
Document that S3 ACLs must be enabled
When using S3 compatible object stores to save attachments, object-level ACLs must be enabled. If they are not enabled, attaching pictures to toots will fail. For some object stores, the error message does not indicate the root cause of the problem.
Unfortunately it is likely that people will accidentally disable ACLs when first setting up Mastodon. This is because both AWS S3 and Google cloud storage disable object-level ACLs by default. The GUIs for creating buckets recommend somewhat strongly that they should not be enabled.
For example, the AWS docs say
we recommend that you disable ACLs except in unusual circumstances
And the creation UI looks like this:
It appears I'm not the only person who has had trouble with this:
- mastodon/mastodon#17435
- mastodon/mastodon#17978
I'm not sure there's any good reason for Mastodon to require Object-level ACLs, maybe https://github.com/kreeti/kt-paperclip/pull/92 would be a better approach instead?
One place that Mastodon explicitly sets ACLs is when suspending an account. All media from the suspended account is made private using ACLs. There is a PR (https://github.com/mastodon/mastodon/pull/17979) that adds support disabling the use of ACLs in Mastodon. However a comment from @Gargron suggests that hiding media from suspended accounts is an important feature.
Besides the places where Mastodon explicitly sets ACLs, I needed to enable ACLs to successfully attach a picture to a toot. I tested this using Mastodon v3.5.3 against AWS S3 and GCS.
That's a great point, I wasn't aware of that feature.
On Sat, Jul 2, 2022 at 8:17 PM Austin Wise @.***> wrote:
One place that Mastodon explicitly sets ACLs is when suspending an account. All media from the suspended account is made private using ACLs https://github.com/mastodon/mastodon/blob/a233a9bfb5f384e89bdaef6e519fa20db2a99ae5/app/services/suspend_account_service.rb#L80. There is a PR (mastodon/mastodon#17979 https://github.com/mastodon/mastodon/pull/17979) that adds support disabling the use of ACLs in Mastodon. However a comment from @Gargron https://github.com/mastodon/mastodon/pull/17979#issuecomment-1091421987 suggests that hiding media from suspended accounts is an important feature.
Besides the places where Mastodon explicitly sets ACLs, I needed to enable ACLs to successfully attach a picture to a toot. I tested this using Mastodon v3.5.3 against AWS S3 and GCS.
— Reply to this email directly, view it on GitHub https://github.com/mastodon/documentation/pull/952#issuecomment-1172988288, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABZCV6A2MCICW6XGORTRZ3VSDSZDANCNFSM52PQNS5Q . You are receiving this because you commented.Message ID: @.***>
This actually took down my instance. I enabled S3 uploads but used the recommended configuration from Amazon - i.e. no ACLs. This meant that all the uploads fail because Amazon does not have a "transparently ignore public ACLs but let the uploads succeed" option.
I didn't notice that jobs were failing at the time for two reasons:
- I was migrating old media from a local filesystem.
- I set up Amazon Cloudfront to get around the fact that no ACLs = no public files.
This resulted in a massive queue of Sidekiq jobs piling up and Sidekiq constantly using more RAM than I had. Previously I had added swap space specifically for occasional memory excursions, but constant usage burns through EBS's burst balance really quick. And once you run out of EBS burst the instance locks up and dies.
I rebased the commit to resolve merge conflicts and update the description of this PR with more details.
@nightpool Can you take another look at this PR. I updated the description with a more detailed description on why people are likely to incorrectly setup their S3-compatible storage. And there have been a couple of commenters mentioning that they hit this issue.
Although note that the underlying bug here was fixed in 4.1.0 hopefully so it's still possible to use non-ACL services, it's just highly recommended to have ACLs enabled
On Sat, Jul 2, 2022 at 3:44 PM Austin Wise @.***> wrote:
When using the web GUI to create a bucket, both AWS S3 and Google Cloud Storage have object-level ACLs disabled by default.
It appears I'm not the only person who has had trouble with this:
- mastodon/mastodon#17435 https://github.com/mastodon/mastodon/issues/17435
- mastodon/mastodon#17978 https://github.com/mastodon/mastodon/issues/17978
You can view, comment on, or merge this pull request online at:
https://github.com/mastodon/documentation/pull/952 Commit Summary
- 14e0057 https://github.com/mastodon/documentation/pull/952/commits/14e0057a485cf66746fb0faf4007162b7f8903d3 Document that S3 ACLs must be enabled
File Changes
(1 file https://github.com/mastodon/documentation/pull/952/files)
- M content/en/admin/config.md https://github.com/mastodon/documentation/pull/952/files#diff-38e8ddf43a9a32293552d94137f678df4008dd396d3859b9908c21eb0557faf1 (2)
Patch Links:
- https://github.com/mastodon/documentation/pull/952.patch
- https://github.com/mastodon/documentation/pull/952.diff
— Reply to this email directly, view it on GitHub https://github.com/mastodon/documentation/pull/952, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABZCV7JPUQ6O7EE66CCAZTVSCL2HANCNFSM52PQNS5Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>