Allow more preview formats

[rullzer]

Right now we serve previews in png or jpg. Which has the advantage of being support on nearly every platform since the late 90s. But of course it is no longer the best solution out there.

I would propose to allow different image formats. The top candidate in my opinion is webp. This allows for smaller images at comparable quality. Of course the backend would need to distinquis what to server but I'm sure we can solve that with some js magic to set the accept headers ;)

This doesn't mean we fully get rid of the jpg an png as older firefox and IE versions do not support it. But I bet this could be performance boost especially when using a lot of images.

[lapineige]

:+1: for WebP, it is now supported by the major web browsers, and is more interesting than JPEG, PNG and GIF if we look at the quality/file size.

I think that would be a nice addition, for instance for people who wants to save some storage space on their servers thanks to WebP, for a galery of pictures, and so on. And images in this format would speed up page loading (smaller file size).

[MorrisJobke]

ref #9857

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity and it seems to be missing some essential informations. It will be closed if no further activity occurs. Thank you for your contributions.

[J0WI]

There is a tiny app for WebP support written by @Flow86: https://github.com/Flow86/preview_webp

Would you agree to merge this functionality upstream?

[Flow86]

yes of course you can merge the functionality to upstream

[stefaang]

Here's an interesting demo with blurred webp preloads: https://jmperezperez.com/webp-placeholder-images/

[SimplyCorbett]

Can we get a status update on this?

[mrAceT]

There is a tiny app for WebP support written by @Flow86: https://github.com/Flow86/preview_webp Would you agree to merge this functionality upstream?

This has been written in 2019.. when I see the code one would think this could be added to the main server software?

Just a thought, when one takes a look at 'oc_filecache' you see that the preview images are generated in the same format as the original (JPG / PNG). Webp is now (version 24.0.3) not really supported. When I upload a webp image, the preview generated is in PNG! Could it be an idea to add the functionality for webp and add the following option to the config:

'preview_format' => 'webp',

This because webp is smaller then PNG and JPG, so when one would want previews in webp, oen would want it for all previews..

[J0WI]

same for AVIF btw.

[utack]

We even regressed on this ticket, heif previews are now disabled

[whitephoenix117]

Hope this is still active, adding AVIF support would outstanding, its supported in imageMagick.

I did some testing with some of my large (60MP) pictures and its able to reduce ~10MB photos (jpg) to 0.5M-2MB (avif)

[aentwist]

Clearly the frontrunners here are WEBP and AVIF. AVIF for "best" format (with caveats) and WEBP for the (still current, yet quickly shrinking) advantage of browser support it has over AVIF. JPG and PNG are ancient.

https://jakearchibald.com/2020/avif-has-landed/ https://caniuse.com/avif

I am interested in helping with this if it gets more direction.

[aentwist]

I haven't actually used Nextcloud, but from what I can tell after reading the code, previews are generated by simply resizing original images and preserving the format (whether it be JPEG, PNG, WebP, ETC). Some compression is added, at least for JPEGs.

So we want to allow all previews to be of a specific format.

Note that Nextcloud seems to have WebP support [ https://github.com/nextcloud/server/commit/6d4afca7ace0b62f4256b18a0cab60f180ebbc07 ]. It does not seem to have AVIF support.

Let's break this down -

this issue

• [x] Allow previews to be saved in specific formats*
• [x] Add a setting for preview format
• [ ] Add documentation for the setting
• [ ] Update/add tests

at this point WebP previews are available

maybe this issue, maybe other issues

• [x] Add AVIF support - can use #30118.. not sure how involved this is
• [x] Add setting(s) for format parameters to mirror the existing setting for JPEG quality (for now leave off or choose balanced defaults)

I am going to look into the main part first - saving in specific formats. The rest is chores.

[aentwist]

I turned on AVIF quality 1 and now my Nextcloud is screaming fast. Also, if you wear glasses but forget to put them on, there is no disadvantage. This is the way.

On a more serious note go grill my initial pass at this. I look forward to it. Still needs polished. Next steps for me, put up a draft PR for the documentation.

[aentwist]

From #30118

At this time using AVIF is *not* trivial due to system dependencies. TLDR - compile a current version of libaom for use on Debian bullseye. [docker-php-extension-installer](https://github.com/mlocati/docker-php-extension-installer) does this for you. The package manager's version should work in bookworm.

https://github.com/juliushaertl/nextcloud-docker-dev/issues/182

I can provide my doctored Docker image (which uses docker-php-extension-installer) for the time being. In ~ 6 months - 1 year it probably won't be an issue whatsoever.

[darkBuddha]

AVIF is essential

[JanisPlayer]

I once conducted a test and added AVIF to the preview generator in the following files:

• lib/private/Preview/AVIF.php
• lib/private/PreviewManager.php
• apps/theming/lib/ImageManager.php
• lib/private/Preview/Imaginary.php
• lib/private/legacy/OC_Image.php
• lib/private/Metadata/Provider/ExifProvider.php
• resources/config/mimetypemapping.dist.json
• lib/composer/composer/autoload_static.php
• lib/composer/composer/autoload_classmap.php
• lib/private/Collaboration/Reference/LinkReferenceProvider.php

To identify all the places where I needed to make changes, I simply searched for WebP. Here's the link to my search: GitHub WebP Search.

In the end, I only modified the most important parts. Now, AVIF images are also being converted into preview thumbnails. If there's still interest and someone would like to assist me, I would create a pull request.

The only issue I'm facing is that the images are not being accessed within the APP file on the web and observed with the image viewer, although the thumbnail generation is working. However, based on my test, it's feasible and even relatively straightforward to implement, as you essentially need to copy everything from WebP and adapt it for AVIF.

[lapineige]

I can't help with the implementation part, but there is an interest, yes !

[JanisPlayer]

I've gone ahead and created a PR based on my personal experimentation, incorporating what has worked relatively well so far. However, there are still a few issues that need to be addressed, so the work is not yet complete. Nonetheless, it's a starting point that demonstrates the feasibility of the concept. I'll see later how I can add contributors to the patch, so that you might contribute the parts I couldn't address quickly.

[aentwist]

See also my PR, which never gained any traction for some reason despite being fully working afaict. Maybe if you swirl them together you'll get something good. But you need to work with someone who will review properly and merge, and I am not sure who that would be.

[JanisPlayer]

PR

Ah, I see you've even taken it a step further and wanted to fully integrate AVIF as an alternative to JPEG. I think that was the issue. If I focus on AVIF support for decoding and conversion, they might accept it. It's not like it's a rare, poorly supported format anymore, as JPEG XL unfortunately turned out to be, where you now need WebAssembly to display images in the browser and even GIMP doesn't fully support it.

I mean, AVIF has really gained excellent support today, and owning an AVIF image is no longer a rarity. So, from a pure support perspective, I don't foresee any problems. I think the concern was that your idea didn't go over well due to the high overhead that the encoder requires. A developer from a Nextcloud image application actually demonstrated to me that anything above 1080p takes an extremely long time and can lead to disruptions. Therefore, the best approach might be to decouple something like this from Nextcloud. That's what I'm also doing with my WebP images, which causes no inconvenience whatsoever. However, WebP images also have significantly lower overhead, so with AVIF, it's inevitable to impose restrictions on resolution and quality. This limitation, though, would suffice for preview images without significant zoom, as is common with Nextcloud.

Thank you; I'll take some inspiration from your approach, and perhaps I can resolve a few issues this way.

[darkBuddha]

I converted all my documents and images to AVIF, and all my videos to AV1. To have first-class support for AV1 + AVIF is an absolute necessity. It will be everywhere very soon.

[JanisPlayer]

@darkBuddha Yeah, the browser support for AVIF has greatly improved. Only Edge, Opera Mini, QQ Browser, KaiOS Browser, and of course IE don't support it anymore. It's questionable why Edge doesn't have support yet, considering it's based on Chromium. Microsoft seems to only make amazing modern browsers that can do everything. ;D

But there are enough user-friendly and widely-used options to display and create this format. I've been quite surprised that Nextcloud still doesn't have support for previews.

In the meantime, I've also looked at the PR from @aentwist. I think it was well-structured and had interesting ideas included. However, the issue is that the code needs to be updated to the current state. I'm currently facing a few challenges to implement the previews correctly.

For example, deciding whether to use a PNG or WebP with an alpha channel. And ensuring that the images are displayed correctly within the Nextcloud Apps interface. I just can't seem to find the point in the interface where the app decides whether to use the viewer or to download the file.

Edit: It has turned out that adding to the Viewer app is also possible. There was also a PR for WebP and HEIF back then. However, doing this today is apparently easier than it was at the time of the PR. Nowadays, you only need to add an image.avif.cy.ts file to cypress/e2e/images/, and in /src/models/images.js, you'll need to decide what to add AVIF for, as many browsers support it, but not all of them. That's a good question.

As of now, the following is possible:

• A thumbnail is displayed next to the name in the file browser.
• The image is converted to a PNG preview, and it can be accessed via the URL.

So, anyone who has solutions to these problems could contribute them. I can't say for sure whether AVIF will actually be added to Nextcloud, because @aentwist's PR intended to do so, but somehow it wasn't accepted.

There seemed to be a lack of review and interest, despite there being a milestone. There was hardly any feedback afterwards, regarding improvements or functionalities.

@aentwist, do you know what your PR still needs in order to be merged?

[JanisPlayer]

I've added AVIF support experimentally with an extra option to DG. However, I've designed it in a way that makes it easy to remove, and by default, it will use JPEG without this option for AVIF images.

One could consider converting AVIF images to lower-resolution AVIF images in the standard settings, but I believe JPEG is sufficient for this purpose. As mentioned earlier, due to browser support, this might not be a feasible option.

If you set 'preview_format' => 'avif' in the configuration, all images, except GIFs, will be converted to AVIF by DG Preview Generator. This seems to work acceptably up to a resolution of 1080p, even though it heavily taxes the CPU. So, the question is, would AVIF support as a preview format be useful to you, even though it might only yield reasonable performance at lower resolutions on fast systems?

Additionally, some browsers, as mentioned above, won't display the images anymore because of the lack of native support. When you open the image, the original is loaded, and the thumbnail is missing because all images, regardless of resolution, are converted to AVIF with this option.

I didn't want to add extra code to handle different resolutions for small thumbnails in JPEG and large previews in AVIF.

I'm considering fully supporting AVIF through Imaginary, but this project needs some adjustments. The significant advantage is that it's separate from Nextcloud and can be better optimized. Although there are forks of Imaginary, they are not entirely finished, and you can't create PRs for changes in Nextcloud. So, I would need to ask a developer there first.

But first, I have a question for you all:

• Is a resolution of 1080p sufficient for you?
• Would a CPU maxing out during generation be an issue?
• Do you have a system with 4 cores, at least 2 GHz, and 4-8 GB RAM as a server?
• Does it bother you that the above-mentioned browsers won't display previews anymore?

If all of these conditions are met, I can try adding AVIF experimentally with this setting as a preview format. I've also created a benchmark to give you a sense of what it's like when generating 8 images compared to JPEG and WebP.

Since my goal is to make as few changes as possible and the highest priority is to convert AVIF to a preview initially, this is just an experiment. I would even suggest that someone with more experience completely rewrite the Preview Generator, and we first add AVIF in a way that can be easily removed. I've already adjusted most of the Nextcloud apps for AVIF support. If I missed any, please let me know, or feel free to make the adjustments yourself as a PR.

[DesertCookie]

To answer your survey:

• 1080p is sufficient for previews, though I use 2000p quality 65 JPEGs on my server.
• I see no issue in my CPU maxing out. I can pin threads in unRAID; since image generation is done in small chunks, I'm not worried.
• My server runs a 6-core 11400 with 32GB of RAM; I've done AV1 encoding on weaker machines.
• No previews are my main concern. Would it load the original on not-supported browsers (though that only seems to be Edge and IE - who'd have thought)?

In my experience too, AV1 is mostly efficient at higher resolutions. 1080p is about the lowest I'd ever go with it (at lower resolutions it's usually beaten by well-tuned VP9 and HEVC). I'm mainly looking towards native AVIF support for originals; those currently don't even produce a preview image. However, working with 6-8k images that AVIF can reduce to 25% of their JPEG 80% size while also using 10bit just makes it very hard to not want to move all my images to it.

[Glutamat42]

This is only relavant for my personal small instance of Nextcloud as this is the only one where the photos app is used.

• 1080p would be acceptable although i would prefer a little bit higher (maybe 1440p)
• CPU maxing: no, not at all. Ideally give it a low priority and i don't care
• HW requirements: no, 2 cores 8GB RAM. But this is totally sufficient. Tested with the benchmark script, absolutely working fine, no problems in sight.
• supported browsers: I would not care, but this should also be easily avoidable. The picture html tag allows defining different sources (filetypes) and also an <img> fallback. Assuming these browsers support the picture tag the browsers could choose whatever they support. Disadvantage would be that the server has to generate thumbnails for different filetypes. Letting an admin choose either multiple filetypes (eg i want jpeg and webp, or jpeg and avif) or "generate fallback jpeg thumbnails" would allow everybody to choose the ideal config for his environment.

For me the relevant factor is bandwidth. Nevertheless, support by/for thumbnail pregeneration would be a nice feature to reduce the impact of the longer preview generation time.

[vid-bin]

• I don't mind if it takes longer to generate higher resolution previews but 1080p is absolutely a no-go for me. Needs to be a higher resolution. Some people have high resolution monitors and 1080p looks like a tiny little thing on them.
• I do not mind the CPU maxing.
• I have a high core system with high memory.
• No it doesn't bother me that some browsers don't support the image format

[JanisPlayer]

Thank you for the feedback. I won't impose a limit on this experimental feature, but I'll include a warning in the documentation that resolutions above 1080p may lead to increased resource usage.

Regarding browser support, it does seem a bit challenging. It's not straightforward to set an alternative source since that would require rewriting every client app. The client would need to inform the server of the user agent it's using. The server would then have to send the appropriate preview. To make this work, I'd also need to change how images are stored in the database and how client apps request previews. So, while it's technically possible, whether the core developers would appreciate such a significant overhaul just for optimal AVIF preview support for an experimental feature is questionable.

I checked, and most browsers will soon support AVIF, including Edge, which can already handle it if activated. However, whether you trust this for a third-party user is another matter. Since some seem to want to control the encoder efficiency, it might be possible to make it configurable via a config command. However, this will likely slow down your system, although the images would be smaller. I'm not entirely sure if this approach will be well-received, as it was hastily implemented with the hope that another developer might revamp the preview generator, as there is already a pull request for that.

I'll soon test the code on the latest version of Nextcloud and provide a patch for you to test by replacing the files.

If AVIF as a preview format doesn't come to fruition, you might consider building an app as a temporary solution. What does work well, in any case, is converting AVIF to JPEG previews. I've done the same with WebP, using it as a preview format instead of PNG since it doesn't rely on ImageMagick, which is currently used for HEIC, for example.

I'll need to consult the developers about the priority of the preview generator for each format. There's GD from PHP and ImageMagick. My assumption was that it doesn't matter much and is more about which formats are supported and practical.

I also inquired about using Imaginary (an external tool), but Nextcloud is reluctant to create a genuine fork or use another existing one, which would be necessary to support AVIF. Unfortunately, this option is off the table. We'll have to wait for the developer to add AVIF support to Imaginary, which would have made things much simpler.

Regarding CPU priority, it's also a bit tricky, especially since I'm currently using GD. Changing the script's priority to run with a different one would likely require too much code for an experimental feature. I added AVIF support as a preview format in a way that it can be easily removed if it's not accepted, as, ideally, it would require a complete rewrite of the generator for proper support.

[vid-bin]

@JanisPlayer While this may be outside of the norm it would be nice if AVIF previews included HDR support for images that have HDR data. My understanding is the format does support HDR content.

[JanisPlayer]

@vid-bin Yes, AVIF can support 12-bit color depth, but I'm not sure if PHP's GD library can create such images. Whether it's practical for previews depends on the user's preferences.

I have tested whether GD can handle 12-bit AVIF images in the sense of opening them, but I don't know if it can actually generate them.

I use the Photo App Memories, which allows loading original images when zooming, and it has saved me a lot of storage.

I've just built a patch for version 27.1.2 of Nextcloud. You can test it by finding the patch under the Pull Request. To use it, simply replace the files, and your Nextcloud will be patched. Please use it only for version 27.1.2 to avoid potential errors.

I'm currently using this version, but I can't rule out the possibility of missing errors during testing. If you encounter any issues, discover new formats or settings causing problems, feel free to report them or even post code suggestions.

Regarding the Viewer App, I had difficulties building it, so I made adjustments directly in the viewer-main.js file. If you encounter any issues, please let me know. Note that replacing this file in a version newer than 2.2.0 might lead to problems. Also, keep in mind that this is not a final solution and is intended for testing purposes only. This patch will be removed automatically with the next update of your Nextcloud or apps.