Add option to store images up to 5K resolution

[mhkhung]

This PR is a result of the discussion from Issue https://github.com/mastodon/mastodon/issues/20255

Specifically, this address the issue that uploaded images are forced to scaled down too much (under 1920x1080) in the client side javascript.

How this is addressed:

1. Increase client side upload limit to 5K (5120x2880). 5K is chosen because it is just under the mastodon limit of 4096x4096 MAX_MATRIX_LIMIT = 16_777_216 # 4096x4096px or approx. 16MB
2. Make it an admin option that is off by default. The option is added to Administration->Server Settings->Content Retention
3. If an uploaded image is over 2MB (png or otherwise), it will be forced to convert to jpg to save storage
4. Server will based on the admin setting, scale the image as needed to 5120x2880 or 1920x1080 limit.

How this is tested:

1. Set store-5k, upload 1.6MB 5393x2766 PNG using python client => Image converted to 5362x2750
2. Unset store-5k, upload 1.6MB 5393x2766 PNG using python client => Image converted to 2010x1031
3. Set store-5k, upload 10MB 3984x1538 PNG using python client => Image converted to 3984x1538 JPEG
4. Unset store-5k, upload 10MB 3984x1538 PNG using python client => Image converted to 2318x895 JPEG
5. Set store-5k, upload 8MB 4542x2555 JPEG using python client => Image converted to 3.1MB 4542x2555 JPEG
6. Unset store-5k, upload 8MB 4542x2555 JPEG using python client => Image converted to 600k 1920x1080 JPEG
7. Unset store-5k, upload 8MB 10000x5252 PNG using web client => Image converted to 800k 1987x1044 JPEG
8. Set store-5k, upload 8MB 10000x5252 PNG using web client => Image converted to 3.4MB 5299x2783 JPEG

[floogulinc]

I think the term "image" would be preferred over "photo". Also do you know what quality setting the jpeg encoding will use?

[mhkhung]

I think the term "image" would be preferred over "photo". Also do you know what quality setting the jpeg encoding will use?

Changed.. to store_5k_images

ImageMagick default setting here

  GLOBAL_CONVERT_OPTIONS = {
    all: '-quality 90 +profile "!icc,*" +set modify-date +set create-date',
  }.freeze

[enn-nafnlaus]

I tested out converting a 15MP image using the above settings; it ended up at 1,5M. So a 15MP image would save at about 3x larger than Mastodon is currently saving 2MP images ;)

It still would be really nice to give server administrators the ability to set a target file size and adjust the quality settings in order to meet it. But this is very important start. And I suspect that even in its current form, the overall impact to storage size won't be that great (because of the PNG/JPEG conversions saving space and the fact that such large images are a relatively small fraction of the total)

Anyway, I'll check it out as it stands this weekend (need to get my test server back up first - briefly ran out of disk space on Sunday and now it's not happy with me)!

[floogulinc]

We need to consider how PNGs with transparency are handled. They shouldn't be converted to jpeg ever. This probably requires actually inspecting the image to see if it truly has transparent pixels or not since often images without transparency are still encoded with the alpha channel. Twitter's handling of this may be a good reference: https://twittercommunity.com/t/upcoming-changes-to-png-image-support/118695

[mhkhung]

We need to consider how PNGs with transparency are handled. They shouldn't be converted to jpeg ever. This probably requires actually inspecting the image to see if it truly has transparent pixels or not since often images without transparency are still encoded with the alpha channel. Twitter's handling of this may be a good reference: https://twittercommunity.com/t/upcoming-changes-to-png-image-support/118695

Thanks.. I just read how Twitter did it. Basically for jpeg, we are similar.. Mastodon uses 90% in ImageMagick.. I have no idea if that 90% matches 85% in twitter or 9 in Photoshop etc.. We always send the file thru ImageMagick to strip icc profiles etc. We have a 10MB limit for images.

For PNGs, twitter would check the type of PNG format and may convert to jpeg and compare the before/after sizes, and they would decide to use the converted jpeg or keep PNG. In this PR, we keep it simple for now to convert PNG files bigger than 2MB. Note that ImageMagick's png handling seems to be pretty bad.. a scaled-down PNG can be bigger than original PNG, that's why forcing to jpeg is important for big PNG files.

I want to keep this simple for now. In theory all images are run thru identify so the code might be able to access the file attributes and do smarter processing.

[PinballsWizard]

As an artist who operates a single user instance for myself, I really appreciate seeing this PR and the thought that's gone into it.

I agree with above with regards to PNG files with transparency, and I've a few questions here in regards to this PR:

• If the limit imposed by server processing of images/federation is 16777216 (4096x4096), why was 14745600 (5120x2880) chosen here? Essentially, if the change is to provide instance administrators the option of greater file uploads, why limit to something below the theoretical maximum? (Additionally, why limit to below what Twitter permits, in this instance an admin has chosen to enable this?)
• All testing examples given are landscape, does portrait work just as well?
• Why are PNG files knocked down to JPEG when above 2MB, but the file limit is 10MB? For example, if I upload an image that fits the pixel limit, is a PNG, and is below 10MB I would not expect my image to be compressed to a JPEG. At most I would expect lossless compression.
• Why was the 5k terminology selected in the admin UI, I've had to deal with a number of sites with confusing terminology around limits, in this case the most precise would be to list megapixels and file size, not resolution?
• Could the pixel limit be a drop down? Allowing instances to choose from the default all the way up to the theoretical limit?

Really do appreciate seeing this PR, I understand this is the sort of thing a larger instances may never touch, but for me paying my own hosting for my own art, I'd really prefer just post it and have it be as I intended it.

[mhkhung]

Thanks for the questions. In general, the principles of how I made this PR are:

• As little change as possible - This is my first PR to merge into mastodon, I am quite new to the mastodon world so I don't want to make risky changes that potentially break the fediverse..
• Smaller PR make it easier to get reviewed/approved/merged..

* If the limit imposed by server processing of images/federation is 16777216 (4096x4096), why was 14745600 (5120x2880) chosen here?  Essentially, if the change is to provide instance administrators the option of greater file uploads, why limit to something below the theoretical maximum?  (Additionally, why limit to below what Twitter permits, in this instance an admin has chosen to enable this?)

4096x4096 is assumed to be a federation limit right now, so we don't want to change it. I picked 5120x2880 because:

1. It is just a little smaller than that 4096x4096 limit, I don't want to risk going above this number or changing this number.
2. It is 2.67x both width and height of the original 1920x1080 (which is HD 1080p resolution). 5120x2880 is 5K standard.
3. Thus, we are keeping the code dealing with with aspect ratios.

* All testing examples given are landscape, does portrait work just as well?

Yes, the code automatically deal with portrait vs landscape.. I just tested with https://commons.wikimedia.org/wiki/File:Charlotte_of_France_by_Jean_Clouet.jpg 2345x3109 image -> Saved at same resolution Upscaled to 4526x6000 in Photoshop -> Scaled down to 3335x4421

* Why are PNG files knocked down to JPEG when above 2MB, but the file limit is 10MB?  For example, if I upload an image that fits the pixel limit, is a PNG, and is below 10MB I would not expect my image to be compressed to a JPEG.  At most I would expect lossless compression.

2MB is a magic number I picked to balance current file storage need and the increased need when we start to allow bigger images, I found that even big JPEG files at nearly max resolution supported will be less than 2MB after imageMagick at 90%. And I found that many bigger-than-2MB PNGs are really photos that should be JPEGs. Non-photo PNGs.. mostly illustrations or vector-based graphics, can be encoded smaller than 2MB by the artist - I also choose not to add extra logic to do lossy PNG (pngnq, pngquant) to keep logic simple in this PR.

* Why was the 5k terminology selected in the admin UI, I've had to deal with a number of sites with confusing terminology around limits, in this case the most precise would be to list megapixels and file size, not resolution?

Since 5K is a standard - And the settings logic simply takes the name of the setting.. And also the exact limit is the product of width x height, not by any one dimension, I don't know how better to name/label this. And this is for admin only so I just keep it simple.

* Could the pixel limit be a drop down?  Allowing instances to choose from the default all the way up to the theoretical limit?

Yes.. again I choose to keep it simple.. another choice would make the code even more complicated..

Really do appreciate seeing this PR, I understand this is the sort of thing a larger instances may never touch, but for me paying my own hosting for my own art, I'd really prefer just post it and have it be as I intended it.

You are welcome. May be you can add some of your art into the issue https://github.com/mastodon/mastodon/issues/20255 that we can test with? As you see I have been testing with wiki common images and illustrations.

[github-actions[bot]]

This pull request has merge conflicts that must be resolved before it can be merged.

[mjankowski]

Closing due to merge conflicts and because these limits have been increased (not in same manner as the PR, but increased nonetheless) in time since the PR.

If you'd still like to propose new/different limits, can you rebase and reopen and comment along those lines?