darktable
darktable copied to clipboard
darktable-org
Leica M10 support (0.3% of my images can't be loaded)
:wave: Hi, Darktable already works perfectly fine with almost all of the Leica M10 DNG files I've imported so far, but about 0.3% of my images can't be loaded at all. It says:
darktable could not load 'L1007566.DNG'; switching to lighttable now.
I'm using:
- Darktable 3.8.1
- Ubuntu 20.04.4 LTS
- Leica M10 Firmware 2.7.5
I've uploaded two samples to https://raw.pixls.us:
- L1008407.DNG (brick wall): Can be loaded and edited successfully
- L1007566.DNG (birds & moon): This is one of the odd files that can't be loaded
Reimporting the problematic files does not help. They can be opened with other programs, so they are probably not corrupted.
Please let me know if you need more information / samples.
The problematic file is not in the database. Please share it otherwise.
From the log:
[rawspeed] (L1007566.DNG) void rawspeed::AbstractDngDecompressor::decompress() const, line 217: Too many errors encountered. Giving up. First Error:
rawspeed::Buffer::size_type rawspeed::ByteStream::check(rawspeed::Buffer::size_type) const, line 59: Out of bounds access in ByteStream
Next is to check the w/ debug build what's going on, whether the file is really corrupt or something else...
Loading fails because rawspeed can't find the JPEG EOI marker, and it can't find it because the DNG StripByteCounts is wrong (it is too small; the EOI marker does exist, but not within the file announced StripByteCounts).
So the file, is in fact, corrupt/invalid strictly speaking. If I edit the StripByteCounts to include the EOI marker, the file loads in dt just fine.
Other software works because it probably does no such strict checking, which in fact is not an advantage (i.e. can be seen as a security vulnerability).
So either a rotten card, a problem of the connection to you pc or a firmware issue for Leica to report there.
Would be interested how you import the images. Copy first to your hard disk and import or directly importing from your camera?
I'll try to reproduce the problem with other SD cards, maybe it really was a faulty card.
I use a card reader, but my process is finicky because my card reader doesn't work with Ubuntu. I have to copy the files from the SD card to a USB flash drive using Mac OS first, then switch to Ubuntu and use "copy & import" in darktable to import the files from the USB drive. Many possible sources of error here, I know.
At one time I tried copying a problematic file from the SD card for a second time (to find out if it had just been a one-off error during file transfer), but darktable was still unable to open it.
By the way: The problematic file can be converted successfully using Adobe DNG Converter, and the converted DNG can be opened in darktable. Not sure what that says about possible file corruption, though.
Would it be pragmatic for darktable to tolerate invalid StripByteCounts (maybe with a warning), or would that be too much of a security issue?
Loading fails because rawspeed can't find the JPEG EOI marker,
It is not that needed to decode. ffplay -i L1007566.DNG
[mjpeg @ 0000021ea1cc8b40] EOI missing, emulating sq= 0B f=0/0
Input #0, tiff_pipe, from 'L1007566.DNG':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: tiff, bayer_rggb16le, 5984x3992 [SAR 1:1 DAR 748:499], 25 fps, 25 tbr, 25 tbn
[mjpeg @ 0000021ea1cd1500] EOI missing, emulating sq= 0B f=0/0
IMHO, I think your time is better spent trying to figure out why these files are in this state in the first place, and if eventually something needs to be flagged to the camera vendor.
As mentioned, there is workaround available - they decode just fine once you correct the StripByteCounts on these 0.3%.
there is workaround available - they decode just fine once you correct the StripByteCounts on these 0.3%.
Can you please attach corrected in hex editor file? Just for posterity. I think it is a bug in firmware, of course. Is it maybe some off-by-one?
Sure. It is way off, 23442499 instead of 23564132 for this file, very strange...
Funny that (3 bytes) change does not decode in ffmpeg anymore, because before that it was skeeping (?) 2992x3992 and 1440x960 (1440x960 it could not even see before)
It fails with Pixel format 0x21111100 bits:8 is not implemented.
Are you sure you are not reading some kind of preview, jpeg/tiff thumbnail?
This is how jpeg xl sample looks like, BTW. Yep, jpeg xl is 2 times less space yet it is lossless. ascaq.zip
Funny that (3 bytes) change does not decode in ffmpeg anymore, because before that it was skeeping (?) 2992x3992 and 1440x960 (1440x960 it could not even see before)
It fails with Pixel format 0x21111100 bits:8 is not implemented.
Does it also fail the same way w/ the working Leica M10 sample?
Are you sure you are not reading some kind of preview, jpeg/tiff thumbnail?
Rawspeed doesn't read previews/thumbnails.
I am experiencing similar (the same?) issues with a Leica M240 with firmware version 2.0.5.0 on darktable 3.8.1 at roughly the same rate, across multiple SD cards, and SD card readers (confirmed that the checksums are identical between readers).
[rawspeed] (L1008551.DNG) void rawspeed::AbstractDngDecompressor::decompress() const, line 217: Too many errors encountered. Giving up. First Error:
const uint8_t* rawspeed::Buffer::getData(rawspeed::Buffer::size_type, rawspeed::Buffer::size_type) const, line 173: Buffer overflow: image file may be truncated
[dt_imageio_export_with_flags] mipmap allocation for `unused' failed
This is how jpeg xl sample looks like, BTW. Yep, jpeg xl is 2 times less space yet it is lossless.
Wow, so there was a big bug, in https://github.com/FFmpeg/FFmpeg/commit/bf64a75c5ae58ed575303f70b2ab9b2208ded339. The image is correct now: aswefqawe2.zip
I shot an e-mail to [email protected] and referenced this GitHub issue. Not sure how their engineers are in terms of responsiveness to issues like this, but it felt like the right thing to do.
Afterwards I also went digging into firmware versions for my own M240. I should probably upgrade it as 2.0.6.0 brought “General bug fixing” (got to love those specific changelogs we get these days…) 1. There is also 2.1.0.0, but it only introduced new lens profiles from what I can tell. Still, will grab 2.1.0.0 to see if I still encounter the issue in the coming weeks and months.
Shot 251 images yesterday with the 2.1.0.0 firmware and they could all be loaded without issues. As a reference, for my previous two cards I had 2/245 and 2/388 images that could not be loaded properly. This is of course not conclusive evidence that 2.0.6.0 fixes the issue, so I will continue testing and report back.
Leica Customer Care also confirmed today that my report and a link to this issue has been passed to their development team. I will keep them posted as well.
Shot 251 images yesterday with the 2.1.0.0 firmware and they could all be loaded without issues. As a reference, for my previous two cards I had 2/245 and 2/388 images that could not be loaded properly
In other words, it looks like a bug in previous Leica firmwares that caused incorrect files to be created under certain (rare, thankfully) conditions. It's good that Leica fixed it, but it's bad that the reason is not a failure of a specific card or card reader, and therefore other users of buggy versions of the firmware also have this problem.
In any case, this is not a darktable issue, since the decision to discard the file or try to be forgiving of data errors that can be corrected (as Adobe DNG Converter does) is rawspeed's business. So this issue should be closed IMO.
