LibRaw
Adds ability to enable Adobe DNG SDK
This makes changes to allow us to specify we want the Adobe SDK, it currently relies on a Cmake script I have in a fork of the DNG SDK (See here https://github.com/Autoenhance-ai/adobe-dng-sdk/blob/1.6-linux/dng_sdk/projects/linux/CMakeLists.txt), we could pull that script into this PR but would love your feedback first.
Thanks for the effort on this. I think from a maintenance point of view I'm not super happy including third-party build files in this repository.
Could I ask what your use case is? Quoting from https://github.com/LibRaw/LibRaw/blob/master/README.DNGSDK.txt:
Some rare flavours of DNG files (e.g. for 8-bit files) are not implemented in LibRaw. These files never come from camera or DNG files created by Adobe DNG Converter, but created by some processing software that uses DNG as intermediate format. To decode such files you need to compile LibRaw with Adobe DNG SDK.
Thanks for the effort on this. I think from a maintenance point of view I'm not super happy including third-party build files in this repository.
Could I ask what your use case is? Quoting from https://github.com/LibRaw/LibRaw/blob/master/README.DNGSDK.txt:
So for unrelated reasons, I tweaked it so the DNG build script is now part of this Cmakefile which should make it easier to maintain and had benefit of solving some linker errors.
So that's now resolved. There are some limitations with this script currently but hopefully we can resolve them (Its built for Linux and there are some issues regarding symbols which I have to tell linker to ignore)
The XMP SDK is significantly more complicated since it's a series of targets each with their own various Cmakelists but considering it's available on Github and has scripts for Linux maintained by Adobe, it should be possible to just download the code and point this script to it's location similar to Rawspeed.
I think doing this way is easier to maintain whilst reducing the learning curve but if this is still too much, perhaps I can instead allow user provided path to the library files emitted by the XMP toolkit when built. As well as provide instructions on how to compile it.
The Use Case
So our use-case is for DNGs which contain compressed JPEG data. Although the documentation claims that they are rare, which relative to the large amount of RAW files may be true, these kind of files can be exported by Adobe Lightroom by Photographers exporting an express DNG to reduce file sizes.
Once you get to handling a certain volume DNG files, these kinds of files actually become very common. Unfortunately the stage2 operations needed aren't implemented by libraw yet and can only be done via Adobe's DNG SDK.
The Problem
The problem is although the libraw documentation mentions that in order to enable the Adobe SDK all you need is to set the USE_DNG flag to true and pass it an instance of the DNG SDK.
It misses the steps on how to download and compile the SDK. Which is made more complicated that the SDK doesn't have any scripts for Linux. On top of this the cmake script and rawpy (PR coming soon) lacked the enabling of Adobe's DNG SDK support
So in practice switching on this feature had a steep curve, which this PR hopes to reduce to just enabling the DNG SDK in the cmake script and pointing it to where the downloaded source code is.
I think doing this way is easier to maintain whilst reducing the learning curve but if this is still too much, perhaps I can instead allow user provided path to the library files emitted by the XMP toolkit when built. As well as provide instructions on how to compile it.
Could you outline what this would look like? That would be my preferred option. It might be better to create a separate repo that allows to easily build the SDK, assuming this can't be upstreamed.
Could you outline what this would look like? That would be my preferred option. It might be better to create a separate repo that allows to easily build the SDK, assuming this can't be upstreamed.
Sp we would just need a location of the folder where the static libraries are compiled i.e XMP_LIB_FOLDER=/xmp/libs/ and we can just link that way.
Another option might just to provide the ability to provide a path to the compiled DNG SDK with XMP static libraries already linked i.e DNG_LIB=/path/to/file/dng.a, I already created a mirror copy of the DNG SDK modified to have the linux cmake script handle all of these details, it unfortunately can't be upstream as the original SDK isn't hosted on a git repo.
Another option rather than parsing the path could be to use pkg_config to detect an installed copy of the DNG SDK ?
@jcampbell05 How do you want to continue here? I think the best would be to keep this repo as small as possible to reduce ongoing maintenance effort.
I'll have to check if possible - but seems like best approach is just to have the lines to at least allow those who have gone through the effort of building the DNG and XMP SDK a way of telling the cmake project to compile with Adobe DNG support and where to find the libraries.
This feels like a happy medium between keeping maintenance low but allowing those who need it to plug-and-play without having to dig through the cmake file.
Hi, is there any chance of getting this merged (even just the minimal version that takes a path to the already-built DNG SDK)? This would enable support for JPEG-XL in libraw.
Not sure if this is the only sensible way to add DNG 1.7.x and JPEG XL support but if it is then +10.
I have already converted my whole RAW collection to DNG 1.7 lossy (with Adobe DNG converter) as it reduces space consumption by 50-60% (or even 80% if it is uncompressed DNG like in Galaxy S21fe). In couple of A/B tests I did with Adobe Lightroom I could not see a difference between lossless DNG and much smaller lossy DNG.
So now I have from each pic a lossless DNG (typically 10-12MB as it is 12mpx) and lossy DNG 1.7 (typically 4-5MB). I will delete all lossless versions and gain about 50GB of HD space but will have to wait until Darktable and couple of other programs have DNG 1.7 support.
If you are professional photographer with much larger collection you can probably save hundreds or even thousands of GB of space by converting "not so important" pics to DNG lossy! And I am already replying in advance for potential "storage is cheap comments", it is actually not that cheap if you have your stuff in cloud. And being able to send the files more than 50% faster is also very appealing.
There is also another very good feature with DNG 1.7, when you convert with Adobe DNG converter to DNG 1.7 lossy you can also reduce resolution. So if you have huge 50mpx or larger RAW pics you can reduce space consumption by up to 95% ! if you drop the resolution also to 10-12mpx which is probably good enough for 95% of applications.
For people interested in this PR and have the time to finish it (Unfortunately I don't), we just need to modify this Cmake file to find and link to a pre-compiled library for the DNG SDK
We can remove all the compilation steps and leave it as an exercise for the end-user of this library. I do not have the time to make these changes and check that they work