photoprism
photoprism copied to clipboard
photoprism
JPEG: Embed ICC Color Profiles in Thumbnails
Embedding ICC color profiles into our JPEG thumbnails seems to be possible with libvips. Libvips is a very fast C image processing library for which a Go integration library also exists:
- https://www.libvips.org/
- https://github.com/libvips/libvips
- https://github.com/davidbyttow/govips
Any help with that would be much appreciated! 💎
Additional Research Tasks and Considerations:
- [x] How to get ICC Profiles in newly created JPEGs with Go (also needed for JPEG XL when available)? It seems the library we use for READING doesn't cover this. Using a Perl script (Exiftool) or an external C-library isn't a great solution. It may please your eyes, but it's more difficult to maintain, adds complexity, and comes with a performance overhead.
- [ ] Do all social media apps and other common sites support v4 profiles? This is where users may want to upload resampled images without original metadata ("thumbnails") to protect their privacy. Also most sites typically compress and downscale images anyway.
- [ ] While the latest browser versions may support ICC Profiles, there may be users with older OS and Browser versions. How many of those exist in total and relative numbers?
- [ ] Assuming Display P3 still looks ok even without software support, are there other profiles that look completely broken and should be avoided? Please test and provide a list.
- [x] Is it possible to embed custom profiles or additional parameters? Seems like a great way to track images without raising attention 🕵️
- [ ] The following color profiles were found in an actual photo library with ~100k files. Do we know how to handle each of these? 🧐
+---------------------------------+
| file_color_profile |
+---------------------------------+
| Adobe RGB (1998) |
| AdobeRGB |
| Apple RGB |
| Camera RGB Profile |
| Canon EOS 5D |
| Color LCD Calibrated |
| DELL U2515H |
| Display |
| Display P3 |
| Dot Gain 15% |
| Dot Gain 20% |
| EPSON Standard RGB - Gamma 1.8 |
| GIMP built-in sRGB |
| Generic RGB Profile |
| Google Skia |
| Gray Gamma 2.2 |
| HD 709-A |
| PHL 271E1 |
| ProPhoto RGB |
| S2431W |
| c2 |
| iMac |
| sRGB |
| sRGB IEC61966-2-1 black scaled |
| sRGB IEC61966-2.1 |
| sRGB Transfer with DCI-P3 Gamut |
| sRGB built-in |
+---------------------------------+
Steps to reproduce:
- Obtain image with an ICC profile applied (e.g. AdobeRGB)
- Import into Photoprism
- Navigate to photo in Photoprism
- Visually compare to original
Expected result: Lower resolution, identical colours.
Actual result: Lower resolution, very noticeable reduction in saturation.
Downloading the thumbnail image and comparing to the original with exiftool shows that the thumbbnail lacks the ICC profile of the original. Copying the ICC profile from the original to the new resolves the problem:
exiftool -icc_profile -b -w icc file.jpg
exiftool "-icc_profile<=file.icc" fit_2048.jpg
As for a solution, it looks like somebody had a similar problem in the past and ended up using libvips via golang bindings. The most mature of these bindings at the moment seems to be govips.
Great, one more dependency 😂 can take a look at this when we're done with facial recognition and batch edit...
Hmm, another option might be ImageFlow Server. Point it at a bucket of images and you can have it process them with URL query parameters.
That would limit PhotoPrism's role to generating the original previews. It could also potentially write the previews to some cloud storage which ImageFlow could read from, going some way towards facilitating cloud storage, something I've seen asked about a few times.
related issue with videos: see https://github.com/photoprism/photoprism/discussions/1765
This change converts colors to standard sRGB if an Apple "Display P3" color profile is detected:
- https://github.com/photoprism/photoprism/commit/5be456a09f7a846d233649c8de21dbb7f4e1a45a
Not sure if color profile conversion also works when originals are HEIF images, needs to be tested. Videos and file formats other than JPEG haven't been touched yet.
Adding support for additional color profiles should be easy (we can leave that to external contributors).
A small challenge may be profile name comparison: For example "Adobe RGB (1998)" vs just "Adobe RGB". There may be differences or there could be different names for the same profile... research needed.
I'll start a new Development Preview build for testing:
- https://docs.photoprism.org/release-notes/#development-preview
I think it'd be preferable to preserve the colour profile. If you change it to sRGB, information is lost. Displays, devices and software these days support P3 or AdobeRGB more often than not so if you convert, you end up with a situation where a user sees one set of colours on their device, then a different set of colours in Photoprism.
Why did I even think you would be happy about the progress? :rofl:
As I've said many times: We're focusing on multi-user support right now, in addition to handling tons of support emails and other questions - so this unexpected improvement is definitely something to be excited about! :warning:
The sRGB color space is widely used for files, displays, printers, and other devices. Almost all displays can reproduce sRGB or something close to it. If you have time, please provide us with an overview which apps, devices & printers have native support for ICC Profiles and Apple Display P3 in particular (or more importantly, which apps and devices do NOT) :+1:
Keep in mind: When there is just one device that doesn't support Display P3, you end up maintaining two sets of thumbnails. It's the same as with AVC: Some browsers support more recent codecs and they may have slightly better details, but storing (and transcoding to) multiple formats is a huge overhead and adds complexity to no end.
According to my information, colors should be converted anyway before downscaling since the algorithms are not designed for P3 colors, which may cause artifacts. Update: Found the links again to this excellent article and this.Turns out, Display P3 and sRGB both use a gamma of 2.2 so for these two profiles there might be no major difference, only the professional DCI-P3 uses 2.6. Still, simply adding metadata to thumbs won't get you the best possible quality and may cause issues with older computers (yes, old people and computers still exist). There are tradeoffs everywhere. Let's spend more time with research!
Don't get me wrong, it's definitely an improvement! I just think that for me, it's a move in the wrong direction, since the problem (seeing different colours in Photoprism to the ones I see on my phone) is still there, it's just different now.
If you have time, please provide us with an overview which apps, devices & printers have native support for ICC Profiles
ICC profiles are software dependent, not device or printer dependent. It's up to the software displaying the image to handle it in the way the "display" (including printers here) expects.
They're supported on all major browsers, and on Android and iOS (where they're part of the OS, so should be supported in ~all apps).
One exception is ICC v4, which is needed for the normal Display P3 profile. This has fairly wide support but isn't supported on Firefox on Windows (works on Mac and Linux though).
There is this project which provides an ICC v2 substitute for Display P3, however.
So my suggestion would be:
- If the image has an ICC v2 profile, leave it. Support for ICC v2 is near-universal.
- If the image has an ICC v4 profile and you're worried about Firefox support on Windows, convert to sRGB but maybe flag it somewhere like in image metadata details so the user can discover the problem organically.
- One day, maybe think about converting to ICC v2 equivalents of ICC v4 profiles, or storing a v2 or sRGB fallback thumbnail for images with v4 profiles.
Keep in mind: When there is just one device that doesn't support Display P3, you end up maintaining two sets of thumbnails. It's the same as with AVC: Some browsers support more recent codecs and they may have slightly better details, but storing (and transcoding to) multiple formats is a huge overhead and adds complexity to no end.
Does that make sense in the scope of Photoprism though? It's currently 100% browser-based. Does it matter if you can put a Photoprism thumbnail on an SD card and stick it in a digital photo frame or printer?
According to my research, colors should be converted anyway before downscaling since the algorithms are not designed for P3 colors, which causes artifacts.
I mostly switched to Piwigo after this issue and haven't noticed any problems. It might be a problem with super tiny images or if you're really pixel peeping but practically I don't notice anything wrong.
ICC profiles are software dependent, not device or printer dependent. It's up to the software displaying the image to handle it in the way the "display" (including printers here) expects.
Yes, so it's out of our control. If it doesn't work, users blame us first just like you do - not their software.
One exception is ICC v4, which is needed for the normal Display P3 profile.
Display P3 is the single biggest issue as Apple seems to use it now by default, so all Apple users are affected. That's why I've spent extra time providing a "good enough" solution while we give more sponsors time to join the party. Quid pro quo.
There is this project which provides an ICC v2 substitute for Display P3, however.
We have too much work already. No hacks please.
Does that make sense in the scope of Photoprism though? It's currently 100% browser-based. Does it matter if you can put a Photoprism thumbnail on an SD card and stick it in a digital photo frame or printer?
Yes, it's a huge business and people do this. Ever heard of photo books? 😂
I mostly switched to Piwigo after this issue and haven't noticed any problems.
So you already have a good solution. That takes some pressure off us! We will come back to ICC Profiles when other high priority items on our roadmap are done :)
Yes, so it's out of our control. If it doesn't work, users blame us first just like you do - not their software.
There are multiple ways it can "not work":
- The client software might not support the ICC profile. In this case the image might just look a bit off (99% of cases) or it might look totally crazy (for very exotic ICC profiles).
- PhotoPrism removes or converts the image colours. In this case the image will, 100% of the time, look wrong.
Clearly, you have no control over the rare and minor first case but you have total control over the second.
Yes, it's a huge business and people do this. Ever heard of photo books? 😂
Sure but why on earth would you make a photobook from a thumbnail?
Take it easy! We will get to it, just not today unless you get us more developers.
To be honest, I can't tell ANY difference on my mid-range 25" Dell screen and my Huawei P30 phone. Could be because I'm too old and my hardware is still too cheap, granted.
I use high quality resampled images for almost everything so that the size fits the format. For my needs, that has the highest impact. Most services and social media sites do a terrible job with downsampling. Speaking of it, do all social media apps and sites support v4 profiles? This kind of information would really help, not your ranting.
I generally don't want my metadata spread all over the Internet. A side effect is that ICC Profiles are currently also missing. When we've developed this, Display P4 wasn't used by default - at least not that I can remember.
In addition, we need somebody to do research on how to get the ICC Profiles in the thumbs. It seems the library we use for READING doesn't cover this:
- https://github.com/mandykoh/prism
Using a Perl script (as suggested in the issue description) or an external C-library isn't a great solution. It may please your luxury eyes, but it's more difficult to maintain, comes with a performance overhead, and also gives me pimples.
BTW: Is it possible to embed custom profiles or additional parameters? Seems like a great way to track images without raising attention 🕵️
Now that we have this informative thread, I'll create a new issue for the current improvements and save this for later 👍
Take it easy! We will get to it, just not today unless you get us more developers. This kind of information would really help, not your ranting.
I think you might be reading frustration/aggression/impatience where there is none. I'm not upset with you or anything (I'd have absolutely no right to be if that were the case), I'm just trying to elaborate where it looks like you might not fully understand something.
As for the "ranting" point, I've been trying to act in good faith, I've done all the research you asked me to do and I think my suggestions have been reasonable given the information available to me at the time so it's really disappointing that you seem to see me as "ranting".
To be honest, I can't tell ANY difference on my mid-range 25" Dell screen and my Huawei P30 phone. Could be because I'm too old and my hardware is still too cheap, granted.
Have a look at the webkit wide gamut test on a device with a wide-gamut display (e.g. a recent iPhone). The "Iceland" and "Italy" examples have some pretty prominent differences in P3 vs. sRGB.
Speaking of it, do all social media apps and sites support v4 profiles?
"All" is a very rigid word but Instagram for example has supported Display P3 since 2017, same as 500px. Flickr supported it since at least 2007, Google Photos seems to have added it in 2019 (though it's unclear whether that counts backups).
Not sure about Facebook or whatever else the kids are using these days but I expect on sites that don't really focus on photos, the treatment is whatever's easiest at the time.
I generally don't want my metadata spread all over the Internet.
I agree and I'd consider most metadata to be a bug, even timestamps if they're sufficiently precise.
Using a Perl script (as suggested in the issue description) or an external C-library isn't a great solution. It may please your luxury eyes, but it's more difficult to maintain, comes with a performance overhead, and also gives me pimples.
To be clear, the exiftool example wasn't intended as a proposed solution, just a way to verify that the problem was a ICC profile and not something else.
As for libvips, I do see the problem from the perspective of preferring a self-contained Go binary. Aside from being a dependency itself, it has a number of other dependencies on libraries used for reading image formats and the like.
But what are the constraints you want to impose on dependencies? There's going to be some dependency no matter what, even if it's just another pure Go package. Have you documented your preferences anywhere (e.g. licensing)?
Aside from maybe FreeBSD, it seems the only supported deployment method for PhotoPrism is Docker and looking at the image, there are already a ton of C dependencies. So why not add another? (To be clear that I don't intend to be belligerent here, I mean this in the sense of "what is the very good reason I'm sure you have", not "you're just being silly and I'm poking you to make you admit it")
BTW: Is it possible to embed custom profiles or additional parameters? Seems like a great way to track images without raising attention 🕵️
Yes it is and yes, that's a potential risk. I'm not sure that it's worth defending against though. If image fingerprinting is the goal, better to put it in the image data itself à la Digimarc.
See #1798 for the rationale of today's change. We'd rather have a good enough solution now than maybe a perfect solution some time next year. Well possible that somebody publishes a pure Go library by then. It may save a ton of work, and our to-do list is full of other tasks anyway.
Remember, creating images is something PhotoPrism does very, very often - for example, every time a face crop is needed, for every single person in an image.
You'd expect some sort of performance overhead in this case, especially on less powerful hardware (which you don't have from what I remember). There are also safety considerations when working with C code. Going into more details is out of scope at this point. There's other work waiting.
FWIW, I put together a PoC for integrating libvips with Photoprism: https://github.com/photoprism/photoprism/compare/develop...bobobo1618:vips?expand=1
This is mainly intended for my personal use but I thought I'd mention it here in case it's useful to someone.
The main things it does for me are:
- Resolve this bug (note that the vips
RemoveMetadataremoves everything except for the ICC profile and rotation metadata) - Add support for HEIC/HEIF/AVIF (though there's other work needed to support those in Photoprism methinks)
- Improve performance substantially
I have no expectations that my changes will make it into upstream, though of course I'd be happy if there were interest.
Thanks, much appreciated! We'll take a look when we have a bit of time after the multi-user release 👍
Currently PhotoPrism is converting RAW image with wide color gamut to JPEG with web-safe color gamut.
I think PP should have an default option to preserve color profile from original image, or an option to normalize color profile to a target one (like DCI-P3) since a lot of devices have supported displaying and taking wide color gamut images (like all iPhones since 2017). If a photo has a wider color gamut than the targeted color profile, then normalize it, if not then preserve it.
Attempting to answer some of the questions from OP:
Do all social media apps and other common sites support v4 profiles? This is where users may want to upload resampled images without original metadata ("thumbnails") to protect their privacy. Also most sites typically compress and downscale images anyway.
This is hard to answer, as "all" is a strong qualifier. The safest thing to do is convert to sRGB before exporting to SOME sites, see end of this wall of text. :)
While the latest browser versions may support ICC Profiles, there may be users with older OS and Browser versions. How many of those exist in total and relative numbers?
Based on: https://www.stetic.com/market-share/browser/ and https://en.wikipedia.org/wiki/Color_management#Application_level
- Chrome based browsers support v2 since version 22 (2012) on all platforms. The oldest chrome version I see is 99. We can safely assume all chromium based browsers (including Edge) in use support at least ICC v2.
- Firefox supports v2 since 3.5 (2011) and v4 since v8 (2011). The oldest version I see in use is 115.
- Safari supports v2 and v4 since version 2.0 (2005). The oldes safari version I can see is 15.
- Opera supports v2 and v4 since 12.10 (2012). The oldest version I see is 109.
- Internet Explorer supports profiles (unclear what version) but converts to sRGB. The image is still displayed correctly but with gamut compression.
The above link lists about 93% of all desktop browsers seen, and they all support ICC profiles on images. And many of the older versions not listed still support at least ICC v2.
For mobile, from the wiki:
- Android supports profiles since 9, but converts to sRGB. Content is rendered correctly but with a smaller gamut. So about 5 year old phones. This is probably the biggest question mark. But 5 year old phones that haven't been updated are really pushing it at this point and won't be a problem for much longer (read: dying batteries or hacked and replaced)
- iOS supports it since 13.4.1 (2020). The oldes iphone that can be upgraded to ios 13 is the iPhone 6s (2015). So even 9 year old iPhones will work.
In summary. Just about any browser from the past 10 years on both mobile and desktop support at least ICC v2. With the asterisk on Android, where this only applies for phones that have had an OS upgrade in the last 5 years. We can safely assume that this will not be an issue.
Assuming Display P3 still looks ok even without software support, are there other profiles that look completely broken and should be avoided? Please test and provide a list.
AND
The following color profiles were found in an actual photo library with ~100k files. Do we know how to handle each of these? 🧐
These questions are ill posed and betray either my misunderstanding of the question or a lack of understanding of the asker.
To clarify, there are an infinity number of possible colour profiles and they are either embedded in the file metadata or referenced by name. Typically they're embedded, and only referenced if they're universally available profiles (read: sRGB, possibly AdobeRGB) or in special circumstances.
It's not the job of PhotoPrism to "handle" each and everyone of these profiles, and neither is that possible. For an application like PP the responsibilites align as follows:
- When rendering an image in HTML, do not strip any ICC profiles from the image. It's up to the browser to render it correctly, and they do (see above).
- When processing an image (resize, transcode etc) there's two options: Process the image data transparently and copy over the ICC profile from the source image OR convert the original to for example sRGB and then process it as sRGB.
Under linux, littlecms (lcms) is one of the most commonly used tools for conversion between colour spaces.
Final thoughts, I understand very much the phenomena that users will blame the application rather than their device. It's a whole thing. So I would propose the following: Add a setting "Convert to sRGB" which is enabled by default, and metion it in the getting started guide. This means that those who do not know what a colour space is will still see colours correctly even if they're watching it on their Babushka's Borst Jar™ from 1969, and those who care about colour reproduction will discover it either from the getting started guide or by rummaging throught the settings. Everyone is happy :)
@cmdremily These changes add a thumb.Vips() function that lets you generate thumbnails with libvips. However, it does not yet support all the options that our currently used thumbnail generator supports, e.g. some of the crops may still be wrong and the Thumbnail() method does not have a kernel parameter to choose a downscaling filter:
- https://pkg.go.dev/github.com/davidbyttow/govips/v2/vips#ImageRef.Thumbnail
Since I also have a ton of other issues and requests on my plate, it would be great if you or someone else could figure out these details so we can eventually use this to generate thumbnails (either by default and based on a config option that would then need to be added). Thank you very much!
Since nobody seemed interested to jump in, I completed the implementation and added a --thumb-library (PHOTOPRISM_THUMB_LIBRARY) config option for you.
The following config options will be available with our upcoming release (or when using the preview build):
| Environment | CLI Flag | Default | Description |
|---|---|---|---|
| PHOTOPRISM_THUMB_LIBRARY | --thumb-library | auto | image processing LIBRARY to be used for generating thumbnails (auto, imaging, vips) |
| PHOTOPRISM_THUMB_COLOR | --thumb-color | auto | default color PROFILE for thumbnails (auto, preserve, srgb, none) |
| PHOTOPRISM_THUMB_FILTER | --thumb-filter | auto | downscaling filter NAME (imaging best to worst: blackman, lanczos, cubic, linear, nearest) |
| PHOTOPRISM_THUMB_SIZE | --thumb-size | 1920 | maximum size of pre-generated thumbnails in PIXELS (720-7680) |
| PHOTOPRISM_THUMB_SIZE_UNCACHED | --thumb-size-uncached | 7680 | maximum size of thumbnails generated on demand in PIXELS (720-7680) |
| PHOTOPRISM_THUMB_UNCACHED | --thumb-uncached | generate missing thumbnails on demand (high memory and cpu usage) | |
| PHOTOPRISM_JPEG_QUALITY | --jpeg-quality | 83 | higher values increase the image QUALITY and file size (25-100) |
| PHOTOPRISM_JPEG_SIZE | --jpeg-size | 7680 | maximum size of generated JPEG images in PIXELS (720-30000) |
| PHOTOPRISM_PNG_SIZE | --png-size | 7680 | maximum size of generated PNG images in PIXELS (720-30000) |
Any help with testing this would also be much appreciated! :gem:
An updated development preview build is now available on Docker Hub so you can test these changes:
- https://docs.photoprism.app/getting-started/updates/#development-preview
- https://docs.photoprism.app/release-notes/#development-preview
Note that I've changed PHOTOPRISM_THUMB_GENERATOR to PHOTOPRISM_THUMB_LIBRARY as that seems clearer to me. A new preview build that includes this change will be available soon.
Based on tests performed with the "photoprism thumbs -f" command, libvips requires twice as much time than the native Go implementation we currently use. Changing the cache and concurrency settings didn't have much of an effect, so I guess we need to optimize it in the same way as the native rendering - by generating smaller sizes from existing thumbs rather than the original image files.
With these changes, thumbnail rendering performance is 42% better with libvips than with our native Go implementation. So I think we can release this, unless someone finds another issue that should be resolved?
Since thumbnail generation with libvips seems mature enough and we haven't noticed any particular issues or downsides, we've decided to make it the default in our upcoming release without waiting any further. We have updated our preview build so that you can test the latest changes and improvements.
If you build from source or use one of our binary installation packages, please note that the system on which you build and/or run PhotoPrism must have libvips >= 8.10 installed.
This is really cool! Thank you for this feature. Are video thumbnails also generated using libvips? Live pictures thumbnails look fine now but video thumbnails still look whitewashed.
@johnappletree we use FFmpeg to extract still images from video files:
https://github.com/photoprism/photoprism/blob/243fa3e83928e23a39148ea26490fa7245d5ccdc/internal/photoprism/convert_image_jpeg.go#L29-L33
Unfortunately, we didn't have time to test if FFmpeg preserves color profiles. It is possible that this requires an additional command flag or a newer version. Our stable release is based on Ubuntu 24.04 LTS, so we are currently using FFmpeg 6.1.1. However, we would like to upgrade to FFmpeg 7 as soon as possible.