rawspeed
rawspeed copied to clipboard
darktable-org
Canon CR3 support.
Though, until the production camera is actually released, and the full set of freely-licensed samples is contributed to the https://raw.pixls.us/, it won't be merged.
Does not appear to be TIFF.
Appears to be ISO Media / ISO/IEC 14496-12.
The actual content of the 'mdat' box appears to be JPEG-ish, but not old-good lossless-JPEG, because it contains M_DQT (define-quantization-tables marker)
$ file canon_eos_m50_01.cr3
canon_eos_m50_01.cr3: ISO Media
$ mpv canon_eos_m50_01.cr3
Playing: canon_eos_m50_01.cr3
(+) Video --vid=1 (*) ( 6000x4000 1.000fps)
Video --vid=2 (*) ( 1624x1080 1.000fps)
Video --vid=3 (*) ( 6288x4056 1.000fps)
Failed to initialize a video decoder for codec ''.
Hmm, ok, this is strange. Got the ISO Media parser working, and hooked-up dummy Cr3Decoder.
The mdat chunk is NOT lossless jpeg. But it just loads via libjpeg.
Those CR3 appear to be plain demosaiced JPEG's...
(Yes, this is an actual decoded image)
According to ffprobe, mdat chunk contains four substreams. First one is, exactly, JPEG preview, so from libjpeg's point of view this is JPEG file + ~30Mb of garbage (real raw data + makernotes) on tail.
unfortunately, two data chunks (1624x1080 and 6288x4056, first one probably is low-res preview) are not lossless jpegs too :(
There are 0xFF0xFF sequences in files 05, 11, 19, (looking at the second half of the file in okteta hex editor)
Thank you folks. I'm going to keep out of a discussion about the pixels and keep to my domain of metadata. Thanks to your comments, I've now heard of ffprobe.
822 rmills@rmillsmbp:~/gnu/github $ exiv2 -pR foo.jpg
STRUCTURE OF JPEG FILE: foo.jpg
address | marker | length | data
0 | 0xffd8 SOI
2 | 0xffdb DQT | 132
136 | 0xffc0 SOF0 | 17
155 | 0xffc4 DHT | 418
575 | 0xffda SOS
823 rmills@rmillsmbp:~/gnu/github $ ffprobe foo.jpg
ffprobe version 3.4.1 Copyright (c) 2007-2017 the FFmpeg developers
built with Apple LLVM version 9.0.0 (clang-900.0.39.2)
configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-librsvg --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-jack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-indev=jack --enable-opencl --disable-outdev=xv --enable-audiotoolbox --enable-videotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-x86asm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libavresample 3. 7. 0 / 3. 7. 0
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100
libpostproc 54. 7.100 / 54. 7.100
Input #0, image2, from 'foo.jpg':
Duration: 00:00:00.04, start: 0.000000, bitrate: 7605132 kb/s
Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown), 6000x4000, 25 tbr, 25 tbn, 25 tbc
824 rmills@rmillsmbp:~/gnu/github $
I'll work on the metadata. We've found the XMP data and I'm confident the Exif (and IPTC) data is in the Canon uuid box at the top of the file.
Later, I will investigate previews and ICC profiles.
I'll work on the metadata. We've found the XMP data and I'm confident the Exif (and IPTC) data is in the Canon uuid box at the top of the file. Later, I will investigate previews and ICC profiles.
Thank you, @clanmills! Sounds awesome! :)
Phil Harvey exiftool 10.82 can parse Canon CR3 it seems there are at least 3 pictures: THMB, PRVW, and other in mdat
for http://img.photographyblog.com/reviews/canon_eos_m50/photos/canon_eos_m50_01.cr3 mdat contains "trak" picture at 0x6d940, 0x37b030, 0x546c70, and TIFF metadata at 0x2565b98
Yep, that much was understood. Need to continue working on that code.. Hopefully tomorrow?
I started a CR3 format description here https://github.com/lclevy/canon_cr3 if you are interested in contributing... I've done it for CR2 here http://lclevy.free.fr/cr2/ and https://github.com/lclevy/libcraw2
Laurent
This is very helpful indeed. I wrote an email to you to ask if you were considering working on CR3. Then I made good progress and decided not to send the email.
I've documented my progress: https://github.com/Exiv2/exiv2/issues/236
Team Exiv2 will have a meeting on May 5 at my home in England. This project will be discussed. There are many demands on the team and only a few contributors. In honestly, I don't think we have the resources to implement support for CR3 in 2018. https://github.com/Exiv2/exiv2/issues/225
Hi, a lots of updates at https://github.com/lclevy/canon_cr3, you can subscribe to the project to see updates
Thanks for doing this. It's looking good.
I'm puzzled by the claim (also made by @lclevy and @LibRaw) that there are 4 images in MDAT. I only see a single JPEG. I see there are 4 trak objects which are a hierarchy of items (CRAW, stsz, co64). Perhaps you could document the meaning of the objects in the trak hierarchy.
532 rmills@rmillsmbp:~/gnu/github/isobmff/pyke369/isobmffdump $ ./isobmffdump --dump stsz ~/Downloads/canon_eos_m50_01.cr3
@0 | ftyp [24]
@24 | moov [28440]
@32 | uuid [26216]
@26248 | mvhd [108]
@26356 | trak [484]
@26364 | tkhd [92]
@26456 | mdia [384]
@26464 | mdhd [32]
@26496 | hdlr [33]
@26529 | minf [311]
@26537 | vmhd [20]
@26557 | dinf [36]
@26565 | dref [28]
@26593 | stbl [247]
@26601 | stsd [128]
@26617 | CRAW [112]
@26729 | stts [24]
@26753 | stsc [28]
@26781 | stsz [20]
00000000 00 00 00 00 00 31 ea 2f 00 00 00 01 .....1./....
@26801 | free [15]
@26816 | co64 [24]
...
There are 4 parts in mdat: 3 pictures (JPEG, plus 2 with new compression), then metadata like in tiff
@lclevy I do not agree that there is a tiff in the MDAT. Here's my evidence: https://github.com/Exiv2/exiv2/issues/236#issuecomment-372625849
We agree that there is at least one JPEG in the MDAT. I'm not convinced there are 3 pictures, however I have no proof to assert there is only 1 picture.
@lclevy Apologies. I am mistaken. There is a valid tiff in the MDAT and ExifTool reads it.
@LebedevRI Thanks for pointing this out. By coincidence I discovered and documented this before your email arrived. Two heads thinking of the same thing! That's better than no heads thinking! https://github.com/Exiv2/exiv2/issues/236#issuecomment-372633645
Steady progress. Great Teamwork. Thanks everybody.
Well done, @lclevy. Very impressive progress. It's possible I now have sufficient information to parse all the CR3 metadata and thumbnails in Exiv2.
@lclevy Happy Easter. We were on vacation last week and now I'm home and would like to look at parse_cr3.py. Where have you put it?
Please allow me this one question: Is this still worked on? I have recently bought a M50 and was hoping for linux support since Lightroom already had it. But for now i purely rely on JPEG output, which is a bit sad.
Presently - no. It will eventually be done. I should probably take a stab at this.
I haven't been able to work on this either. Since July 2017, Exiv2 has been swamped by security issues. I'm glad to say that every issue has been fixed and we will release Exiv2 0.27 on 28 December 2018. We intend to offer "dot" releases as necessary during 2019 and 2020 to address security issues.
It has been a major undertaking to deal with security and very little feature development has been possible. CR3 support is on the "highlight" features for v0.28 and will be discussed at a team meeting in December. https://github.com/Exiv2/exiv2/projects
The Team may, or may not, agree to undertake this work in 2019.
For sure, I will not do this work. I will be 68 in January and have been contributing daily to Exiv2 for more than 10 years. During 2019 and 2020, I will be the release engineer (to do the dot releases), and deal with user questions. However I want to retire and no longer contribute to the C++ code.
I'm delighted to say that the future of Exiv2 is secure. In 2017 I recruited two outstanding engineers and they have dealt with most of the security issues. They have also rewritten the build and test infrastructure.
Several observations about open source:
- Many people request changes
- Few folks volunteer to help
- User contributed patches usually result in lots of work concerning platforms and testing
Adobe have 10,000+ people including several thousand C++ Engineers. I know, I worked for Adobe in San Jose, California for more than 10 years. Team Exiv2 has been mostly a single volunteer. Surprise, Lightroom Supports CR3 before Roman, Laurent and Exiv2 can catch up.
If you have any good ideas about how to recruit engineers to join Team Exiv2, I'll be delighted to receive suggestions.
I do have a practical suggestion to make about this. Canon should be contributing to open-source. When they decide to launch a new format on their users, I believe they have engineers in "Developer Relations" who will engage with Adobe, Google, Apple, Microsoft and others to ensure that PhotoShop and other major applications provide support for their format.
I have never received any communication from Canon or any other camera manufacturer. The MakerNote Support in Exiv2 has been reversed engineered. Laurent has done very good work to reverse-engineer CR3. Canon sold the camera and contributed nothing to the open-source community.
So, @simonszu Would you write to Canon and express your disappointment that they have not supported the open-source Community to adopt CR3.