RawTherapee
RawTherapee copied to clipboard
Published
20 hours ago •
Beep6581
Beep6581
False colors under tungsten illumination
Originally reported on Google Code with ID 2747
After Revision: bf5be3ed66b8 - Issue 2691: No chromatic adaptation in DCP code => wrong
colors
there is a serious regression of color rendering on scenes under tungsten illumination
.. at least when we use single dcp profiles built with x-rite's color passport utility.
These profiles are usually built using a single illuminant (daylight).
I had only tested the patch with samples illuminated by close to daylight illumination
and the issue2691 patch was an improvement there. Unfortunately with samples under
tungsten we have wrong colors.
A useful data set is Dpreview's studio samples which are shot under both "daylight"
and tungsten (push the lamp icon at the top for tungsten) http://www.dpreview.com/reviews/image-comparison/fullscreen?attr18=daylight&attr13_0=sony_a7&attr13_1=canon_eos5dmkiii&attr13_2=nikon_df&attr13_3=sony_a7s&attr15_0=raw&attr15_1=raw&attr15_2=raw&attr15_3=raw&attr16_0=100&attr16_1=100&attr16_2=100&attr16_3=100&normalization=full&widget=1&x=-0.042789977043272844&y=0.664669889676727
As RT comes bundled with 5DIII dcp I used the Dpreview 5DIII raw samples from the above
link .. find screenshots
https://drive.google.com/folderview?id=0B0NqktTgc54sfkJ1aGlXRTdGbEVBQ3pNU0dENnp1ODBmaUJTNUN4Nzl6TWVvaHN2OGdPR3c&usp=sharing
Reported by iliasgiarimis on 2015-04-21 09:42:43
Most easily detected are redish colors ..
Reported by iliasgiarimis on 2015-04-21 09:43:51
Good find! This explains why I kept getting pink toys in a photo I knew had red ones
in a color-managed workflow...
Reported by entertheyoni on 2015-04-21 13:17:21
-
Status changed:
Accepted
DrSlony, do you have any cc24 shots under tungsten ? it would be useful for some measures
.. Dpreview's tungsten shots are not lighted properly for this (light fall off right
to left).
Reported by iliasgiarimis on 2015-04-21 15:22:24
I will make one tonight. Does it have to be a clear tungsten bulb? Can it be a white
tungsten bulb? Can it be a candle?
Btw. I just learned how to make ICC profiles using ArgyllCMS. Ah I'll paste it here
for reference:
argyll-scanin -v -p -G 1.0 -dipn /tmp/old_wb.tif /usr/share/argyllcms/ref/ColorChecker.cht
/usr/share/argyllcms/ref/ColorChecker.cie /tmp/old_wb_ColorChecker_diag.tif
then
argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast matrix" -qh -am -nc
-u old_wb && mv -v old_wb{,_matrix}.icc
argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast Lab cLUT" -qh -al
-nc -u old_wb && mv -v old_wb{,_Lab_cLUT}.icc
argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast gamma+matrix" -qh
-ag -nc -u old_wb && mv -v old_wb{,_gamma_matrix}.icc
argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast singlegamma+matrix"
-qh -aG -nc -u old_wb && mv -v old_wb{,_singlegamma_matrix}.icc
argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast shaper+matrix" -qh
-as -nc -u old_wb && mv -v old_wb{,_shaper_matrix}.icc
-a is algorythm type. Choices are:
l = Lab cLUT (def.), x = XYZ cLUT, X = display XYZ cLUT + matrix
g = gamma+matrix, s = shaper+matrix, m = matrix only,
G = single gamma+matrix, S = single shaper+matrix
The resulting Lab cLUT ICC leads to poor results as predicted because my chart only
has 24 patches. The rest all look good, and don't exhibit this issue when applied to
this 2507 Kelvin photo: http://rawtherapee.com/shared/test_images/pentax_k10d_09_20.pef
I will upload the tungsten cc24 shot here too, tonight or in the morning: http://filebin.net/rt2747xdcp
Reported by entertheyoni on 2015-04-21 15:33:17
I still have to learn how to make DCP profiles using ArgyllCMS, so we can eliminate
the X-Rite program as being the cause.
Reported by entertheyoni on 2015-04-21 15:35:04
# 6, anything that gives full spectrum 2500-3000K light. I think the white is better
due to diffused light.
nice with agryll, up to now I relied on frontends (coca or makeinputicc) for this :).
The first step exports *.ti3 .. yes ?.
BTW I am in search for an "Old_colorchecker.cht" to be compatible with Ted's cc24 target
(see issue_2729) ..
#7, Anders could give a hand I suppose :)
I think it's not x-rite's utility at fault, but RT's dcp code, building an icc and
converting to dcp will again have the same issue as #0, something goes wrong with the
bradford adaptation .. it was OK before.
Reported by iliasgiarimis on 2015-04-21 19:00:48
I'm working intensively with a camera profiling software which I hope to release open
source in the coming weeks. It can do DCP profiles, both from targets or direct by
using spectral sensitivity functions, it will be useful for making profiles for RT.
Until release I won't be doing much context switching, so I can't look into this issue
now.
I'd suggest to test the problematic profiles in Lightroom as a reference. It can be
the problem of the profile rather than the DCP handling.
Reported by [email protected] on 2015-04-21 20:03:56
Correct, it created the .ti3 I uploaded.
I will try to help with his profile.
Reported by entertheyoni on 2015-04-21 20:24:44
Sorry for the delay. Real tungsten clear bulb ("milky white" would change the spectrum).
http://rawtherapee.com/shared/test_images/pentax_k10d_2015-04-20_passport_daylight.dng
http://rawtherapee.com/shared/test_images/pentax_k10d_2015-04-22_passport_tungsten.dng
I will compare this to candlelight tonight, I'm thinking four candles some distance
away at each side to minimize falloff.
Reported by entertheyoni on 2015-04-24 15:49:44
Off topic but related; I have now made my first release of my open-source digital camera
profiling tool, you can make DCPs with this (you need Argyll too):
http://www.ludd.ltu.se/~torger/dcamprof.html
Reported by [email protected] on 2015-04-30 17:28:14
Anders, Congratulations !!.
I wish a win32 complilation will be available to test it's results.
BTW are there any profiles built by Dcamprof available for testing ..
Reported by iliasgiarimis on 2015-04-30 22:36:08
Thanks :-).
You really should get a Linux box running, everything is just soooo easy to compile
there, development environment is builtin. A virtual box with ubuntu in it running
inside your windows perhaps?
Well, maybe I'll look into building a windows version. I have windows 8 running in
a virtual box on my Linux box :-).
I haven't actually had the time yet to play around that much with DCamProf other than
testing during development to see that it works. I'll start making profiles soon.
One cool thing with it is that you can make profiles directly from the camera's spectral
sensitivity functions, that its color filter responses. Unfortunately it's not so easy
to come by that kind of data, and it requires costly gear to measure yourself. For
my Canon 5Dmk2 there is SSFs to get though, but for my Hasselblad I'll have to make
profiles the traditional way with a target.
Reported by [email protected] on 2015-05-01 07:20:00
Here are two profiles generated with the default settings of DCamProf using the images
you attached in #11
Workflow:
1. rotate crop and export for profiling *without white balance*
2. argyll: scanin -v -dipn rawfile.tif ColorChecker.cht cc24_ref.cie
(ColorChecker.cht comes with Argyll, cc24_ref.cie comes with DCamProf)
3. dcamprof make-profile -i D50 rawfile.ti3 profile.json
Replace -i D50 with -i StdA for the tungsten
4. dcamprof make-dcp -c "Pentax K10D" profile.json profile.dcp
Let me know what you think.
Reported by [email protected] on 2015-05-01 13:43:57
- _Attachment: [k10d-tungsten.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-15/k10d-tungsten.dcp)_ - _Attachment: [k10d-daylight.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-15/k10d-daylight.dcp)_
...and here's a DCP with both DCPs from #15 merged into one dual-illuminant profile.
Workflow:
1. make two separate profiles as described in #15
2. convert both profiles to JSON, "dcamprof dcp2json profile.dcp dcp-profile.json"
3. merge both profiles using a text editor, the lower temp profile gets index 1, the
higher index 2.
4. convert back to DCP, "dcamprof json2dcp edited.json dual.dcp"
Reported by [email protected] on 2015-05-01 13:54:53
- _Attachment: [k10d-dual.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-16/k10d-dual.dcp)_
Great work, Anders!
If you (or someone else) wants to do some comparisons or quality checks, I uploaded
all the CC24 Raw files that were used for the creation of the current DCP profiles
in RT for the following cameras: Pentax K-5, Pentax K-5II, Canon Powershot S110.
The DCP profiles were created with the X-Rite software and had their quality checked
with the colorchartcheck.exe utility. For comparison I also include an alternative
profile created with Adobe DNG Profile Editor for the K-5II.
http://filebin.net/u6jmuy4dio
Reported by stefan.ittner on 2015-05-01 19:19:35
I just released a version 0.5.1 of my software which fixes some issues, so the profiles
in #15 and #16 should be regenerated, but it would make only a tiny (probably invisible)
change.
Yes it would be nice if someone could do some qualitive testing, I haven't had much
time for that. Note that DCamProf does not only take precision into account, there's
no problem to match 100% each patch when you have a LUT, but it can lead to sharp bends
in the LUT which leads to bad gradients. DCamProf makes averaging automatically if
patches are too close in chromaticity which happens for a few patches in the cc24.
Therefore it may seem like the profile is "worse" if the only evaluation is DE on patch
matching.
DCamProf also makes a larger LUT than needed to only match the patches; as DCP HSM
LUT uses linear interpolation and DCamProf has spline-based native LUT it creates the
DCP HSM LUT by sampling the spline-based LUT to get smoother interpolation and thus
better gradients. This is also an aspect that is not easily seen when just evaluating
patch matching.
Reported by [email protected] on 2015-05-01 20:58:22
Anders great! Can't wait to test it, but its 2AM now so I'll have to sleep on it :)
I tried the DCPs from #15 and by eyeball the patch colors look more as they should
than when I use DCPs made using X-Rite or ICCs made using ArgyllCMS. More specifically,
the red looks more red and less pink, green is more green, magenta is not oversaturated,
and orange is more vibrant. Not sure about the blue and purple.
How can we quantify the accuracy?
I read that creating LUT profiles from CC24 targets leads to bad results because there
are too few sample points, but it sounds like your spline approach works around this.
What is HSM?
Reported by entertheyoni on 2015-05-02 00:07:39
With DCamProf you can through its SSF support test how good or bad a specific target
is when comparing them to others. I compared it to 1600 patch Munsell and the difference
is not that large, it's hard to say which one's best.
The thing is that the profiling problem is "impossible", each vision color can be represented
by an infinite amount of spectra, and I've seen in my SSF simulations that contradicting
results are not uncommon, that is a green leaf may pull the LUT in one direction and
a green printed color of the similar vision color may pull it in a different, so you
can't correct for both.
That said it would not hurt if the cc24 had more saturated colors and some more purple.
I'm less worried about cc24 profiles now than I was before I ran through my simulations
though.
Of course a profile will be good at fulfilling accuracy of the test target patches
though, so testing if a cc24 target matches cc24 patches is more of a sanity check,
good to do though. DCamProf won't match all 100% as it makes a 2.5D LUT and therefore
groups together the neutral row for example into one group and make and average correction
for those.
HSM = HueSatMap, the DNG Profile LUT.
Spline probably doesn't improve accuracy, but it improves smoothness, ie transitions
from one color to another. This paper does show some advantages of thin plate spline
though (which is the what DCamProf is using):
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3435966/
Reported by [email protected] on 2015-05-02 06:25:12
I created a dual-illuminant DCP from those tungsten and daylight shots using your latest
version 0.5.2.
For reference:
argyll-scanin -v -dipn daylight.tif /usr/share/argyllcms/ref/ColorChecker.cht ~/programs/dcamprof-0.5.2/data-examples/cc24_ref.cie
argyll-scanin -v -dipn tungsten.tif /usr/share/argyllcms/ref/ColorChecker.cht ~/programs/dcamprof-0.5.2/data-examples/cc24_ref.cie
~/programs/dcamprof-0.5.2/bin/dcamprof make-profile -i D50 daylight.ti3 daylight.json
~/programs/dcamprof-0.5.2/bin/dcamprof make-profile -i StdA tungsten.ti3 tungsten.json
~/programs/dcamprof-0.5.2/bin/dcamprof make-dcp -c "Pentax K10D" daylight.json daylight.dcp
~/programs/dcamprof-0.5.2/bin/dcamprof make-dcp -c "Pentax K10D" tungsten.json tungsten.dcp
~/programs/dcamprof-0.5.2/bin/dcamprof dcp2json tungsten.dcp dcp-tungsten.json
~/programs/dcamprof-0.5.2/bin/dcamprof dcp2json daylight.dcp dcp-daylight.json
cat dcp-tungsten.json dcp-daylight.json > dual.json
~/programs/dcamprof-0.5.2/bin/dcamprof json2dcp dual.json "Pentax K10D 2015-04-22 daylight
tungsten dcamprof.dcp"
Reported by entertheyoni on 2015-05-02 11:33:06
I attached the DCP along with one made using Adobe's DNG Profile Editor, another from
X-Rite's Passport program, and two ICC profiles from ArgyllCMS, all made using the
same shots, just in case anyone wants to compare.
Anders I sometimes shoot in light from LED stage cans, if I was to shoot the passport
under white LED light can you advise what options to pass to DCamProf, how to specify
the right illuminant (there is no LED standard illuminant I'm aware of) and whether
I could still use the cc24_ref.cie file?
Reported by entertheyoni on 2015-05-02 11:42:11
- _Attachment: [Pentax K10D 2015-04-22 daylight tungsten DPE.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/Pentax K10D 2015-04-22 daylight tungsten DPE.dcp)_ - _Attachment: [Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp)_ - _Attachment: [Pentax K10D 2015-04-22 daylight tungsten xrite.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/Pentax K10D 2015-04-22 daylight tungsten xrite.dcp)_ - _Attachment: [PentaxK10D 2015-04-20 passport daylight.icc](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/PentaxK10D 2015-04-20 passport daylight.icc)_ - _Attachment: [PentaxK10D 2015-04-22 passport tungsten.icc](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/PentaxK10D 2015-04-22 passport tungsten.icc)_
Anders RawTherapee's "DCP Illuminant" combobox is grayed-out when I load this DCamProf
DCP. Did I merge them incorrectly?
Reported by entertheyoni on 2015-05-02 12:19:17
- _Attachment: [dcp-tungsten.json](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-23/dcp-tungsten.json)_ - _Attachment: [dcp-daylight.json](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-23/dcp-daylight.json)_
You can use the cc24_ref.cie for any illuminant as it contains reflectance spectra,
that is a specification on how each patch reflects light. So DCamProf can calculate
exactly what light in the full visible spectrum that reflects back to the observer
(eye).
I don't know what the spectrum from your led stage lights are. Probably the spectrum
is a little bit peaky just like fluorescents and not really ideal for profiling. Halogen
lamps are still better. However if you really want to make a profile to be used under
that led light, it's best to use that as light source of course.
DCamProf uses illuminant spectrum for two things, 1) to calculate XYZ reference values
for the color matrix and 2) to assign calibration illuminant to the DCP. Actually it
does not matter that much if it's correct. The color correction takes place with the
forward matrix which is always related to D50. Specifying the calibration illuminant
is more for informational use. Calibration illuminant matters when calculating the
color matrix and it's only used to figure out which color temperature we're shooting
at (white balance), and has some value when making a dual illuminant matrix too as
it will decide how to mix the two matrices and LUTs.
With DCamProf you can specify illuminant StdA, D50 and D65, plus any blackbody. One
suitable illuminant for your led could be to find out what it's correlated color temperature
is and use that as illuminant.
If the led's correlated color temperature is reasonably close to StdA (~2850K) or D50
(~5000K) or D65 (6500K) I'd specify that as illuminant to make-profile. Otherwise I'd
specify with a blackbody say "-i 3700K" if the CCT would be 3700K. The illuminant name
in the DCP will then be the boring "Other", but the profile should work as it should.
Haven't really tested how RT reacts to "Other" illuminant, as the dual-illuminant files
are nearly always StdA + D65.
Another way is to simply see if the light is redder than daylight and then pretend
it's StdA. If it's a lot wrong it will only be a potential slight problem in a dual
illuminant profile.
Note that RT never get its white balance temperature from the DCP matrices, it gets
it from the DCraw/Camconst D65 matrix. In a longer term we should change this so RT
uses the DCP.
Reported by [email protected] on 2015-05-02 14:33:36
I tried your Pentax K10D 2015-04-22 daylight tungsten DPE.dcp in #22 and it is dual
illuminant and it works, combo box is not greyed out.
Reported by [email protected] on 2015-05-02 14:37:30
Yes, I want to use the LED profile only for those shots, as I shoot there quite frequently.
The profile you tried with "DPE" in the filename comes from DNG Profile Editor. It
is the one named "Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp" which has
the combobox grayed out. I see now that concatenating doesn't work, I did it like this
now and it works http://i.imgur.com/gfl74Bw.png
Fixed dual-illuminant DCamProf DCP attached.
Reported by entertheyoni on 2015-05-02 15:55:33
- _Attachment: [Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-26/Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp)_
If you have access to a spectrometer you can read the LED light's spectra using Argyll's
illumread. I've just added ability to read Argyll's SPECT format directly when specifying
illuminant, coming in the next patch release. In the current one have to reformat it
to JSON to read it.
But as said it's not that important that it's right, if you're making a single-illuminant
profile and using it with RT it's not used at all as all color correction is done with
the ForwardMatrix which is locked to D50, and white balance derivation is done from
the camconst/dcraw color matrix.
Reported by [email protected] on 2015-05-02 16:11:57
Released anew version 0.5.3 where you don't need to merge dual illuminant profiles manually.
dcamprof make-dcp profile1.json profile2.json output.dcp
One can also get a full list with all exif lightsources and their temperatures, useful
for choosing calibration illuminant.
Reported by [email protected] on 2015-05-03 17:32:34
Do these DCP profiles support the perceptual rendering intent?
Reported by entertheyoni on 2015-05-03 21:40:14
#29, uh I don't think I understand. What does it mean to support perceptual rendering
intent? As far as I know DCP profiles is not supposed to do any gamut mapping.
I've seen custom profiles that actually do gamut mapping (that is move in saturated
colors outside sRGB so they get inside) made by people that is not satisfied how Lightroom
does gamut mapping.
However this means you get a static mapping so that even if no colors are outside sRGB
you still get compression. It's better to let the raw converter color pipeline handle
gamut mapping, which usually just means pure clipping and then you as a user use the
color adjustment tools to adjust to your liking. I prefer that way to work.
Reported by [email protected] on 2015-05-04 06:34:07