raspberrypi
[BUG] IMX477 on Raspberry Pi5 colors in JPEG mode (no-AWB)
Something appears to be wrong with non-AWB JPEG processing on the Raspberry Pi5 platform with the IMX477 camera.
I am not sure if the blue channel is being boosted or the red channel is diminished, the balance of colors is very wrong.
I am ignoring the green bias of the images, that is expected.
JPEG mode (no AWB)
JULIETT
JPEG mode histogram
DNG mode (no AWB)
DELTA
DNG mode histogram
You can see in the JPEG histogram that the blue levels are considerably higher than red.
I mentioned this issue in #715, but this issue affects all images, not just random images.
Can you provide the exact command line you used to capture both images please?
JPEG mode:
rpicam-still \
--immediate \
--nopreview \
--camera 1 \
--encoding jpg \
--quality 95 \
--gain 1 \
--shutter 133437 \
--metadata /tmp/tmpk25kbn0d.json \
--metadata-format json \
--awbgains 1,1 \
--tuning-file /usr/share/libcamera/ipa/rpi/pisp/imx477.json \
--output /tmp/tmprq_iu29o.jpg
DNG mode:
rpicam-still \
--immediate \
--nopreview \
--camera 1 \
--raw \
--denoise off \
--gain 1 \
--shutter 133437 \
--metadata /tmp/tmpsidbzfie.json \
--metadata-format json \
--awbgains 1,1 \
--tuning-file /usr/share/libcamera/ipa/rpi/pisp/imx477.json \
--output /tmp/tmpr3541uf4.dng
The command lines looks sensible to me. And unless I've misunderstood the problem, I don't think there is anything really wrong here.
The "JPG mode" output goes through our ISP pipeline which applies lots of processing on the pixel data, including WB gains, colour correction matrices, tonemapping/gamma, etc. Whereas the "DNG mode" output, which presumably you have used an external application like RawTherapee to visualize, will likely not have these processing stages applied to the image. So I would expect the samples to be quite different.
Just a note for your dng mode command - If you use the following (note the change in filename extension for the --output argument):
rpicam-still \
--immediate \
--nopreview \
--camera 1 \
--raw \
--denoise off \
--gain 1 \
--shutter 133437 \
--metadata /tmp/tmpsidbzfie.json \
--metadata-format json \
--awbgains 1,1 \
--tuning-file /usr/share/libcamera/ipa/rpi/pisp/imx477.json \
--output /tmp/tmpr3541uf4.jpg
this command will save both a JPEG (tmpr3541uf4.jpg) and a DNG (tmpr3541uf4.dng), and the JPEG will be fully processed by the ISP. This will allow you to compare our processing with the external DNG processing that was applied by your DNG viewer.
I think the green channel is masking the problem. I will also post the same scene with AWB applied to see the normal levels.
Normal scene
NOVEMBER
JPEG output (minus green channel) - I boosted the midpoint to brighten things up a little.
FOXTROT
DNG output (minus green channel) - boosted for contrast
GOLF
The walls are an [unfortunate] shade of brown (which has a greater red than blue component). You can see the DNG output has the appropriate red levels, but the JPEG output has blue way over represented.
This issue appears to be unique to the Pi5. The same camera module on a Pi4 will not exhibit this behavior.
Adding @davidplowman for his thoughts.
So how was the normal scene in the pictures above captured? And what app are you using to visualize the DNG file?
Hi, thanks for all these images. I think we're still not 100% clear what the question is, perhaps it would help if I explain a bit what happens.
When you capture a JPEG (fully-processed image), we take the raw image from the sensor and (ignoring pixel-level processing that doesn't really affect the overall look) apply vignetting and colour-shading correction, then white balance, a colour matrix (to give sRGB) and finally gamma and tonemapping curves.
The DNG contains only the raw camera data, with none of that post-processing. When comparing the two images, therefore, it will depend entirely on what your raw converter is doing. But in general, a converted DNG file is going to look somewhere between "quite different" to "completely different", as compared to our JPEG. We also note that this will affect every colour channel.
Can you say how you are converting your DNG file and what processing the converter might be doing? Thanks!
@naushir @davidplowman The images were captured through indi-allsky (I am the main developer). indi-allsky can ingest RAW (DNG/FITS) or JPEG/PNG data from all sky cameras. OpenCV is used for debayering. In the DNG photos, there was no processing performed other than debayering and downsampling the data to 8-bits. The JPEG photos did not have any processing other than the last two which were just adjusted for brightness after I removed the green channel.
Problem Statement
On the Pi5 platform, JPEG images without AWB applied via rpicam-still have serious color balancing issues. The unprocessed DNG output is actually better. And just for clarity, this is NOT about the color balance not being perfect. I expect the output to be less ideal. The issue here is the data has been compromised to the point that it is unusable. The same configuration on Pi4 does not exhibit this behavior.
I have labeled the images above to make it easier to reference.
- NOVEMBER - Normal JPEG output from rpicam-still (with AWB)
- JULIETT - JPEG output no AWB (--awbgains 1,1)
- DELTA - DNG output no AWB (--awbgains 1,1)
- FOXTROT - JPEG output (juliette) minus the green channel
- GOLF - DNG output (delta) minus the green channel
If you look at FOXTROT, there is an overall blue bias that is not present in GOLF/DELTA. This blue bias should not be present.
Thanks for the update. Perhaps you can confirm that I'm understanding everything correctly.
- NOVEMBER was a standard capture using rpicam-still, with AWB running, and it's what we expect.
- JULIETT is another capture from rpicam-still, the only difference being that both colour gains were set to 1.
- DELTA is what you get when you load the DNG into OpenCV and de-bayer it there, doing nothing else (is that right?).
And the problem you're reporting is that JULIETT and DELTA look quite different. Again, is that correct?
Now, if I've got all that right, then I would say that the issue is that rpicam-still (the imaging pipeline on the Pi) is performing multiple operations above and beyond just de-bayering the raw data. Specifically we have:
- (Before de-bayering) Black level subtraction (but perhaps you are doing this too?).
- Lens shading and vignetting correction.
- (After de-bayering) Application of a colour matrix.
- A gamma/tonemapping curve is applied.
Again, if this is indeed the case, then it might be worth turning off the colour matrix and gamma adjustment in the Pi's imaging pipeline. You can do this by finding the camera tuning file (probably /usr/share/libcamera/ipa/pisp/imx477.json) and replacing "rpi.ccm" by "x.rpi.ccm" and "rpi.contrast" by "x.rpi.contrast" (which effectively comments them both out). I think you can probably also disable the lens shading adjustment by replacing "rpi.alsc" by "x.rpi.alsc".
Alternatively, given that the NOVEMBER picture looks OK, is it possible to send us a short Python script that processes the DNG file to give the same results that you are getting? Thanks!
@davidplowman You are right on all counts. I did forget about the black level offsets, indi-allsky automatically subtracts this if found in the metadata. But the data is otherwise unprocessed.
I would not say that the problem is that JULIETT and DELTA look different. I understand that libcamera is processing the JPEG and I expect them to be different. The problem appears to be libcamera incorrectly adjusting the colors.
Based on the info above, by disabling the rpi.ccm processing appears to bypass the issue, so I suppose something in the color correction matrix might be negatively impacting the output.
Attached is a MVP for converting a DNG to a JPEG in Python. I included a small function to remove the green bias called SCNR Average Neutral which is something used in the astrophotography world. The color green is not prevalent in astronomy, so a synthetic green channel is created from the blue and red pixels. It does a decent job of removing green bias quickly.
#!/usr/bin/env python3
import numpy as np
import cv2
import rawpy
import logging
DNG_FILE = 'input.dng'
MAX_BITS = 16 # some cameras return data in 10/12/14 bit space
BLACK_LEVEL = 4096
CFA = 'BGGR' # IMX477 CFA
logging.basicConfig(level=logging.INFO)
logger = logging
class DNG2JPEG(object):
__cfa_bgr_map = {
'RGGB' : cv2.COLOR_BAYER_BG2BGR,
'GRBG' : cv2.COLOR_BAYER_GB2BGR,
'BGGR' : cv2.COLOR_BAYER_RG2BGR,
'GBRG' : cv2.COLOR_BAYER_GR2BGR,
}
def main(self):
### start
logger.info('Read %s', DNG_FILE)
raw = rawpy.imread(DNG_FILE)
raw_data_16 = raw.raw_image
logger.info('Subtract offset: %d', BLACK_LEVEL)
black_level_scaled = BLACK_LEVEL >> (16 - MAX_BITS)
raw_data_16 = cv2.subtract(raw_data_16, black_level_scaled)
logger.info('Debayer: %s', CFA)
debayer_algorithm = self.__cfa_bgr_map[CFA]
bgr_data_16 = cv2.cvtColor(raw_data_16, debayer_algorithm)
logger.info('Downsample to 8 bits')
shift_factor = MAX_BITS - 8
bgr_data_8 = np.right_shift(bgr_data_16, shift_factor).astype(np.uint8)
### if you want to read a jpeg instead
#bgr_data_8 = cv2.imread('input.jpg')
### remove green bias
#bgr_data_8 = self.scnr_average_neutral(bgr_data_8)
logger.info('Write output.jpg')
cv2.imwrite('output.jpg', bgr_data_8, [cv2.IMWRITE_JPEG_QUALITY, 90])
def scnr_average_neutral(self, data):
### https://www.pixinsight.com/doc/legacy/LE/21_noise_reduction/scnr/scnr.html
logger.info('Applying SCNR average neutral')
b, g, r = cv2.split(data)
# casting to uint16 (for uint8 data) to fix the magenta cast caused by overflows
m = np.add(r.astype(np.uint16), b.astype(np.uint16)) * 0.5
g = np.minimum(g, m.astype(np.uint8))
return cv2.merge((b, g, r))
if __name__ == "__main__":
DNG2JPEG().main()
The colour matrix is of course a very simple thing, so there's no way it can be "applied wrongly", but when you set explicit colour gains there is some uncertainty about which colour matrix will get chosen. It will use the one for the colour temperature that has gains "closest" to (1, 1), which is a bit hard to predict in cases like this.
You always have the option of deleting the colour matrices in the tuning file and replacing them by one of your own choosing. Then you know that's the one that it will use. Though this does mean that the colours in fully processed images, using AWB, will presumably be off now - a problem that could be mitigated be keeping two alternative tuning files.
Also, and I don't know if it helps, you can reverse engineer what it did by fishing the colour matrix out of the DNG file. Of course that's a camera-RGB-to-XYZ matrix, so you'd have to munge it a bit to get back to the camera-RGB-to-sRGB matrix that it started from the tuning file.
But anyway, do we think we've got to a point where you can get the behaviour you want, or is there still something to explain? Thanks!
I suppose the real question is: ~~Why is there a change in color behavior only on the IMX477 between the Pi4 and the Pi5? THIS ONLY AFFECTS THE IMX477.~~ [edit: affects both Pi4 and Pi5] Every other camera module I have tested performs pretty much the same between the Pi4 and Pi5. Even the IMX378 is not affected on the Pi5.
I am not the only person who has noted this behavior. I linked one issue from indi-allsky here, but there have been others. It is becoming a larger problem because the Pi5 is really starting to gain popularity and the IMX477 is the best camera module for the purposes of an allsky camera. (I can go into more detail why this is the case, but that is not really important here.)
I am assuming this is not widely reported because most people using the IMX477 would use AWB. However, in the use-case of an allsky camera, the most common mode of operation is low-light, high-gain, and long-exposures with a need to minimize the delay between exposures (the libcamera AWB causes long delays between exposures).
Perfect color balance is not a requirement, but it should still be within reason.
Hi again, I'm still really struggling with this one, mostly because I don't know how to determine that something is "wrong". Here's what I tried.
I ran the following on both a Pi 4 and a Pi 5. From the discussion above, I understand that Pi 4 "works" and Pi 5 doesn't:
rpicam-still --shutter 30000 --gain 1.0 --awbgains 1,1 -r -o test.jpg
I pushed the DNG files through the code you posted above (adjusting for 12-bit data in the Pi 4 case), and compared those against the JPEGs produced by rpicam-still.
The top row shows the Pi 4 pictures, the bottom row is Pi 5. The left column are the images produced by the code above; the right column are the JPEGs directly from rpicam-still. These are clearly much brighter and more saturated than the left column images, but we expect this. Beyond that, the Pi 4 and 5 images look broadly similar to me.
Can you confirm that you're using the exact same tuning files as the ones we ship, namely this one for Pi 5 and this one for Pi 4.
If you compare the EXIF data for a Pi 4 and Pi 5 DNG, what differences are there (besides the 12/16 "bit-ness" of course)? Perhaps you could post the output of something like exiftool for both. Are you in a position to perform the same experiment as I did with a Pi 4 and a Pi 5 image - then perhaps we can describe some clear way to determine that something is "wrong" in the Pi 5 version?
I ran a full gamut of tests with the IMX477 on the Pi4 and Pi5 and there is consistent output from both systems. I can only guess that when I tested months ago, there was a difference between the Pi4 and Pi5 systems. I can say with certainty that the blue bias exists on both Pi4 and Pi5 on the IMX477 when disabling AWB in JPEG mode. There was a point in the past where it did not on the Pi4, there has definitely been a change in behavior.
I also performed a bit more testing with the IMX378. There is a slight blue bias produced, but it is nowhere near as severe as the IMX477. (The IMX378 image is at least usable.)
I have also been testing manually applying the Color Correction Matrix. I have had OpenCV code available that is able to apply the CCM values provided in the metadata to the DNG raw file. I never widely used it because the output never made sense (more on this below).
With the OpenCV CCM code, I can manually apply the CCM to the DNG raw file and I can produce the same blue bias with similar effect to what libcamera produces. The CCM is 100% the cause of the blue bias. I am starting to suspect that the CCM should not be applied if AWB is disabled.
Is there established order of operations that libcamera uses to process images?
Hi, and thanks for the update.
I ran a full gamut of tests with the IMX477 on the Pi4 and Pi5 and there is consistent output from both systems. I can only guess that when I tested months ago, there was a difference between the Pi4 and Pi5 systems. I can say with certainty that the blue bias exists on both Pi4 and Pi5 on the IMX477 when disabling AWB in JPEG mode. There was a point in the past where it did not on the Pi4, there has definitely been a change in behavior.
We certainly haven't done anything much to the tuning files recently, other than unrelated changes relating to camera synchronisation, or metering for HDR.
The colour matrices for Pi 4 were last changed back in summer 2023 as far as I can tell. Do you think there has been a change of behaviour since then?
I also performed a bit more testing with the IMX378. There is a slight blue bias produced, but it is nowhere near as severe as the IMX477. (The IMX378 image is at least usable.)
All the colour matrices are produced by the same procedure, namely a least squares fit of the raw data to the canonical Macbeth chart values, all in the CIELAB colour space. There is of course some uncertainty in which colours match most accurately and which don't. The gamma curve is also an issue here, because we don't apply a standard 2.2 gamma, there is some additional tonemapping which may accentuate the errors in some places.
You might prefer to try the imx477_scientific.json tuning file, which produces more colour accurate results, though for everyday photography, you get slightly flatter and less "punchy" images. Failing that, there's always the option of producing your own colour matrices.
I have also been testing manually applying the Color Correction Matrix. I have had OpenCV code available that is able to apply the CCM values provided in the metadata to the DNG raw file. I never widely used it because the output never made sense (more on this below).
With the OpenCV CCM code, I can manually apply the CCM to the DNG raw file and I can produce the same blue bias with similar effect to what libcamera produces. The CCM is 100% the cause of the blue bias. I am starting to suspect that the CCM should not be applied if AWB is disabled.
I can certainly see an argument that, if your AWB gains are not chosen to give you "real" output colours, then there's no point in applying a CCM either. It's very easy to disable the colour matrices entirely. I think replacing "rpi.ccm" in the tuning file by "x.rpi.ccm" effectively comments it out. libcamera has no runtime control for disabling it, so far as I can recall.
Is there established order of operations that libcamera uses to process images?
There is an established order in which the hardware processes the images. For Pi 5 you can find the information here, probably start with the diagrams on pages 64 (the Bayer part of the pipeline) and 67 (the RGB part of the pipeline). Colour matrices are pretty much the first thing after demosaic.
After reading the document, my conclusion is CCM does appear to be the cause of the problem, however the CCM is NOT wrong The CCM is only valid with proper color balance, ie if the color gains have been properly calibrated.
The green bias of the RAW image causes the CCM to over-correct the data. There was definitely a change in behavior. Perhaps the CCM was previously not applied to non-AWB data by accident and the behavior was "fixed".
I realize the tuning file can be edited to disable the CCM, but I feel like this places the burden on the user to fix a mistake libcamera is making. The CCM should certainly not be applied when the color gains are set to (1.0, 1.0).
I realize the tuning file can be edited to disable the CCM, but I feel like this places the burden on the user to fix a mistake libcamera is making.
Slightly disagree here. A common use case for our users is to provide manual R/B gains since they have fixed and known lighting conditions. In such cases, they would still want CCM applied to the image.
I do agree that having to edit the tuning file to disable CCMs may not be very intuitive. Perhaps we can consider a libcamera control that could be used at runtime to disable the CCM?
Slightly disagree here. A common use case for our users is to provide manual R/B gains since they have fixed and known lighting conditions. In such cases, they would still want CCM applied to the image.
I would agree with this. That is why I only mentioned the case where awbgains are (1.0, 1.0).
A runtime control to disable the CCM would be great.
@naushir, @davidplowman,
As you may recall, I support the Allsky software, which is similar to Aaron's indi-allsky. Of the 500+ Allsky users I have data for, over half are using the RPi HQ camera. I don’t know how many "other" Allsky users there are or how many indi-allsky users there are, but suspect our total user base is over 1000, with about 600 HQ cameras. And they all use the cameras primary for long-exposure nighttime images. 12% of Allsky users have a Pi 5.
I think the Allsky users tend to be less technical than the indi-allsky users, and asking most of the Allsky users to edit a tuning file simply won't work. Having A runtime control to disable the CCM would work, since Allsky (and I assume indi-allsky) can make that change without impacting users.
Eric C
@EricClaeys having a runtime control for CCM ought to give Allsky users what they need. However, we cannot add this control directly without talking to the libcamera developers. However, this may take some time to implement and release.
I wonder if the best interim solution would be for RPi to add a new tuning file called imx477_allsky.json that removes the AWB algorithm entirely, and sets the CCM to unity. You can then switch this tuning file in with --tuning-file /usr/share/libcamera/ipa/rpi/pisp/imx477_allsky.json. Would that work for you?
The change to allow manual CCMs will soon be merged into libcamera: https://patchwork.libcamera.org/patch/24046/.
I'll resolve this shortly.
@naushir, Is there a new argument to rpicam-still to enable this change, or will it always be enabled?
Eric
We will implement a new command line arg to override the default CCM. It will not be enabled by default.
With https://github.com/raspberrypi/rpicam-apps/commit/962af1ef9945f8b2999a5e5259df2a0f3260cbfc merged, you can now use the --ccm command line option to disable the CCM operation if needed.
Resolving now.