darktable
darktable copied to clipboard
darktable-org
Regression where CR2 Lens ID is no longer read from metadata in 4.4.1
Describe the bug
Lens ID metadata is no longer imported on .CR2 raw images from a Canon 550D, which prevents me from using the lens correction module automatically.
Steps to reproduce
Import raw image in Darktable 4.4.1. Metadata is missing.
Expected behavior
Metadata should not be missing (it was present in earlier versions).
Logfile | Screenshot | Screencast
Metadata of existing raw image in my library (originally imported with 4.3.x) ; lens is set to Tamron AF 17-50mm f/2.8 Di-II LD Aspherical:
Metadata of the same raw image imported with 4.4.1; lens is set to Unknown Lens (161):
Where did you install darktable from?
OS package repository.
darktable version
4.4.1
What OS are you using?
Linux
What is the version of your OS?
Archlinux
Describe your system?
X11, 16G RAM, on Intel GPU. Up-to-date at the time of writing.
Are you using OpenCL GPU in darktable?
Yes
If yes, what is the GPU card and driver?
Intel(R) HD Graphics 530
Both rawspeed and exiv2 report the lens model (with a shortened name, but this seems to be how it's stored in the metadata):
$ exiv2 -pe IMG_9755.CR2 | grep LensModel
Exif.Canon.LensModel Ascii 70 TAMRON 17-50mm f/2.8 Di II A16
$ raw-identify IMG_9755.CR2 -v | grep Lens:
Lens:
Lens: TAMRON 17-50mm f/2.8 Di II A16
Downgrading rawspeed does not yield different results. Still looking into where the 'Unknown Lens' comes from.
In src/common/exif.cc (on master b7af1b456f) around line 1325:
....snip....
/* Read lens name */
if((FIND_EXIF_TAG("Exif.CanonCs.LensType")
&& pos->toLong() != 61182 // prefer the other tag for RF lenses
&& pos->toLong() != 0
&& pos->toLong() != 65535)
|| FIND_EXIF_TAG("Exif.Canon.LensModel"))
{
dt_strlcpy_to_utf8(img->exif_lens, sizeof(img->exif_lens), pos, exifData);
}
....
The FIND_EXIF_TAG("Exif.CanonCs.LensType") call is hit first and yields Unknown Lens (161), but the actual tag is present in FIND_EXIF_TAG("Exif.Canon.LensModel").
Replacing the code with this makes Darktable detect the correct lens again:
if(FIND_EXIF_TAG("Exif.Canon.LensModel"))
{
dt_strlcpy_to_utf8(img->exif_lens, sizeof(img->exif_lens), pos, exifData);
}
else if((FIND_EXIF_TAG("Exif.CanonCs.LensType")
&& pos->toLong() != 61182 // prefer the other tag for RF lenses
&& pos->toLong() != 0
&& pos->toLong() != 65535))
{
dt_strlcpy_to_utf8(img->exif_lens, sizeof(img->exif_lens), pos, exifData);
}
Last commits touching this part are 27826707c1 and 18394dce055, but given that they already try to fix something related to lens detection I'm not sure how I should correct that without messing up support for another user...
It is worth noting that Arch has also switched to Exiv2 0.28 at about the same time dt 4.4 was released, which might include some changes re lens detection, especially w.r.t. Canon.
Thanks for the insightful comments and the workaround!
Arch has also switched to Exiv2 0.28 at about the same time dt 4.4
Can confirm that the same master commit works after downgrading Exiv2 to 0.27.5 then rebuilding (I couldn't test on the distro-provided binaries because downgrading breaks dynamic linking).
In the meantime, you can always add id 161 to your exiv2.ini
Can confirm that the workaround works, by adding this in ~/.exiv2:
[canon]
161=TAMRON 17-50mm f/2.8 Di II A16
@kmilos as you know a lot about exiv2.
Wouldn't be the Exif.Canon.LensModel tag the more "general" one to be used with higher priority if both tags are available? So change the order being checked would be right?
I'm not a Canon shooter, and there are many legacy corner cases for the reason the things are the way they are. AFAIK nobody cracked the Canon lens detection in general...
In any case, might be better to discuss this further with other Exiv2 devs who are more familiar with Canon lens detection, there are already numerous related issues/PRs over at https://github.com/Exiv2/exiv2/issues?q=sort%3Aupdated-desc+canon+lens
In the case of lens id 161, it looks like there is some aliasing to be resolved, exiftool also lists several possibilities: https://exiftool.org/TagNames/Canon.html#LensType
Wouldn't be the
Exif.Canon.LensModeltag the more "general" one to be used with higher priority if both tags are available?
Not really - both exiftool and exiv2 (through Exif.CanonCs.LensType) prefer to use internal, user contributed ID to "pretty" string translation tables for legacy Canon lenses (Exif.Canon.LensModel seems only reliable/clean and is prioritized since RF bodies).
These legacy "pretty" strings are also used by lensfun, so changing this in any major way is very risky...
Why this stopped working for this particular ID 161 in exiv2 0.28 (although the lens is in the internal table still) is a question for the exiv2 devs who have been bravely dealing with Canon lens detection insanity for years. @alexvanderberkel @hassec @postscript-dev
@kmilos you keep track on this as you know so much about this area?
@jenshannoschwalm I also don't own a Canon camera and only have a little experience with lens code.
However, I remember a change was made to the Exif.CanonCs.LensType translation code (in 0.28.0) to try and improve the accuracy. As you have found, Canon only stores an ID in the tag and it is left to others to translate it into a string. As IDs are not unique, an integer can resolve into a long list of potential lens. An Exiv2 PR has been merged that tries to filter the lens list by using the photo's aperture and F-number values. These are discovered by looking in other tags.
However, something is not working correctly and for your file, Exif.CanonCs.LensType outputs Unknown lens (161).
The Exiv2 Exif.CanonCs.LensType translation code can be found in https://github.com/Exiv2/exiv2/blob/e3f7509be1b3b383d7020bcdbe6f8889b9b4c10e/src/canonmn_int.cpp#L2633
Moving forward, I would find out the Exiv2 0.28.0 lens and apperture values for your test file.
I realize I let this one sleep for a while among the various summer activities & holidays... sorry for that!
To summarize, from what I understand:
- The name of the lens is somewhere in Exiv2. The key under which Exiv2 stores it has changed in some way from 0.27 to 0.28 (only for the C API, as the
exiv2command-line output seems equivalent) and darktable wasn't adapted. - The question now is whether Exiv2 should change it back/add something for compatibility, or whether darktable should adapt to how the new Exiv2 does things right?
Am I correct in my understanding?
I'm not sure how I should submit a bug to the Exiv2 developers given that information besides "it worked before in darktable and now it doesn't without a workaround".
Just in case, there is a complete metadata for one of my shots using exiv2 -pe:
was able to reproduce that with current master on windows.
exiv2 -pv --grep lens/i /c/Users/xxxxx/Bilder/raw/yyyyyy.CR2
0x0016 CanonCs LensType Short 1 254
0x0017 CanonCs Lens Short 3 100 100 1
0x0095 Canon LensModel Ascii 74 EF100mm f/2.8L Macro IS USM
0x0000 CanonLe LensSerialNumber SLong 1 -2046820352
0x0007 CanonAfC USMLensElectronicMF SLong 1 0
0x000b CanonAfC LensDriveWhenAFImpossible SLong 1 0
0xa432 Photo LensSpecification Rational 4 100/1 100/1 0/1 0/1
0xa434 Photo LensModel Ascii 28 EF100mm f/2.8L Macro IS USM
0xa435 Photo LensSerialNumber Ascii 11 0000008672
also reproducible with cr3 files
but not consistent:
Canon 7D/Canon EF 24-105mm f/4L IS : ok
0x0016 CanonCs LensType Short 1 237
0x0017 CanonCs Lens Short 3 105 24 1
0x0095 Canon LensModel Ascii 74 EF24-105mm f/4L IS USM
0x0000 CanonLe LensSerialNumber SLong 1 1980760064
0xa432 Photo LensSpecification Rational 4 24/1 105/1 0/1 0/1
0xa434 Photo LensModel Ascii 23 EF24-105mm f/4L IS USM
Canon 5DmIV/several EF lenses: nok Canon R6/several EF lenses: nok Canon R6/Canon RF 24-105mm F4 L IS : ok
0xa432 Photo LensSpecification Rational 4 24/1 105/1 0/1 0/1
0xa434 Photo LensModel Ascii 23 RF24-105mm F4 L IS USM
0xa435 Photo LensSerialNumber Ascii 11 7413007751
0x0016 CanonCs LensType Short 1 61182
0x0017 CanonCs Lens Short 3 105 24 1
0x003d CanonFi RFLensType SShort 1 258
0x0095 Canon LensModel Ascii 138 RF24-105mm F4 L IS USM
0x0000 CanonLe LensSerialNumber SLong 1 1996493684
0x0007 CanonAfC USMLensElectronicMF SLong 1 0
0x000b CanonAfC LensDriveWhenAFImpossible SLong 1 0
was able to reproduce that with current master on windows
FWIW, MSYS2 switched to exiv2 0.28 recently (around 2023-10-03).
Canon 5DmIV/several EF lenses: nok Canon R6/several EF lenses: nok
The code to try to figure out this ID aliasing that (non RF) lenses on Canons are notorious for has indeed changed for 0.28, as already mentioned. Someone just has to get involved and debug this upstream, "metoo" only goes so far...
i'll keep macports at exiv2 0.27
You might at least want to update to 0.27.7 to benefit from some minor fixes.
how to convince exiv2 team that there's something weird even they report valid results from exiv2 commandline tool?
how to convince exiv2 team that there's something weird even they report valid results from exiv2 commandline tool?
Maybe by first continuing discussion at e.g. already linked https://github.com/Exiv2/exiv2/issues/2746 (or another issue upstream) where someone from Exiv2 can actually see some activity?
And it is not strictly valid - please test w/ -pa or -pt - it should return a (correct) string (looked up from internal table or ~/.exiv2) in Exif.CanonCs.LensType instead of a number.
ok -pt shows the "unknown lens"
exiv2 -pt --grep lens/i /c/Users/xxxxxxxx/Downloads/20230625-IMG_0144.CR3
Exif.Photo.LensSpecification Rational 4 100mm
Exif.Photo.LensModel Ascii 28 EF100mm f/2.8L Macro IS USM
Exif.Photo.LensSerialNumber Ascii 11 0000008672
Exif.CanonCs.LensType Short 1 Unknown Lens (254)
Exif.CanonCs.Lens Short 3 100.0 mm
Exif.CanonFi.RFLensType SShort 1 n/v
Exif.Canon.LensModel Ascii 138 EF100mm f/2.8L Macro IS USM
Exif.CanonLe.LensSerialNumber SLong 1 -2046820352
Exif.CanonAfC.USMLensElectronicMF SLong 1 Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible SLong 1 Continue Focus Search
I wonder, if there is any update on this issue?
I would like to add a model, which is no longer recognized correctly
For the Canon EF-S 24mm f/2.8 STM the entry "Exif.CanonCs.LensType" is "Unknown Lens (4154)"
from exiv2.exe -pt:
Exif.Photo.LensModel Ascii 19 EF-S24mm f/2.8 STM
Exif.CanonCs.LensType Short 1 Unknown Lens (4154)
For another lens, the entry is correct:
Exif.Canon.LensModel Ascii 138 EF-S10-18mm f/4.5-5.6 IS STM
Exif.CanonCs.LensType Short 1 Canon EF-S 10-18mm f/4.5-5.6 IS STM
you need to follow https://github.com/Exiv2/exiv2/issues/2746 for updates ... until it's fixed you need a build made with exiv2 0.27.7
This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hello, This bug also affect myself with my Canon 760D and TAMRON 18-400mm F/3.5-6.3 Di II VC HLD B028. Darktable shows it as Unknown Lens (117).... Do you have any plan to fix this bug?
@ktatar Yep, I don't think darktable can do a lot of things until this is sorted in https://github.com/Exiv2/exiv2/issues/2746. In the meantime the workaround of adding a config file described earlier in the thread should work!
seems something may be wrong ONLY with exiv2 0.28.2 on MacOS On MacOS, search for LensType failed for Canon EF zoom lenses at the upper half of the focal range (exiv2 returned Unknown Lens (xxx) ) but reported correct lens for shots at the lower half of focal length range On a different OS, same test done by someone of the exiv2 team on the same test files returned correct lens identification for all files
Forget the entry above It's the locale, as long yas our locale is, eg, US or UK, all works fine both with exiv2 (0.28.2) and darktable (4.6.1) Locales like FR, PT or DE make both fail, most likely because of the decimal separator