raspberrypi
Problems with colours in AllSky software
Hallo all,
As discribed in the AllSky Github issues, some users experience a problem with the Rpi HQ camera.
Mostly at night, some pictures are “too blue”. Maintainer @EricClaeys suggested to open an issue here because he thinks it is a libcamera software issue.
I’m running a fully updated Bookworm on the RPI5 with the Raspberry HQ Camera. Attached you can see a timeline of the images taken with a 60 seconds interval.
Could this be a libcamera software issue?
Regards
Can you provide us the exact rpicam-apps command line used to capture the image and the actual JPG/DNG images as well please? Without this, there's not much we can investigate I'm afraid.
Hello @naushir , sorry I’m really not very experienced with Linux. What do you mean with ”rpicam-apps command line”?
rpicam-apps is the application that captures the images for allskycam. The application is configured by a number of parameters on the command line. This would be useful to know exactly how the system is configured to capture images. Perhaps speak with the package authors and they can give you this information?
Hello @naushir ; I will ask them, thanks.
In att 2 images taken 60 seconds apart.
Are you able to attach the original images taken from the camera - the once you posted in the comment above don't have any metadata in them.
Unfortunately these do not have metadata either :(
Perhaps you can ask the authors if the original camera image is stored anywhere on the filesystem?
OK, I've asked them to provide you the required information. Thanks for now!
I suspect Github may strip metadata off images. It'd be worth trying ZIPing them and attaching the archive.
@babsenfred, to get the command line used to take the pictures, run:
grep "Running:" /var/log/allsky.log > /tmp/rpicam.txt
There will be one line for every image. You can attach /tmp/rpicam.txt.
All the images are stored in ~/allsky/images/YYYYMMDD where YYYYMMDD is something like 20240912
I also have the same issue, so maybe I can share my own images and command line...
Here are two images taken about 60 seconds apart. From what I can see, there is no metadata in these images. They have been downloaded directly from the RPI5 using WinSCP (/Home/pi/allsky/images/20240913/)
and here are the commands line used for each image....
2024-09-13T22:23:32.191317-04:00 Allsky allsky[16016]: STARTING EXPOSURE at: 2024-09-13 22:23:32 @ 60.0 sec 2024-09-13T22:23:32.207545-04:00 Allsky allsky[16016]: > Running: libcamera-still --output '/home/pi/allskyold/allsky/tmp/image-20240913222332.jpg' --timeout 1 --nopreview --width 4056 --height 3040 --shutter 60000000 --analoggain 8 --quality 100 2024-09-13T22:24:34.137377-04:00 Allsky allsky[16016]: > Saving NIGHT image 'image-20240913222332.jpg' 2024-09-13T22:24:34.145002-04:00 Allsky allsky[16016]: > Sleeping: 10.0 sec 2024-09-13T22:24:44.139692-04:00 Allsky allsky[16016]: ----- 2024-09-13T22:24:44.139831-04:00 Allsky allsky[16016]: STARTING EXPOSURE at: 2024-09-13 22:24:44 @ 60.0 sec 2024-09-13T22:24:44.155749-04:00 Allsky allsky[16016]: > Running: libcamera-still --output '/home/pi/allskyold/allsky/tmp/image-20240913222444.jpg' --timeout 1 --nopreview --width 4056 --height 3040 --shutter 60000000 --analoggain 8 --quality 100 2024-09-13T22:25:46.165833-04:00 Allsky allsky[16016]: > Saving NIGHT image 'image-20240913222444.jpg' 2024-09-13T22:25:46.166357-04:00 Allsky allsky[16016]: > Sleeping: 10.0 sec 2024-09-13T22:25:56.168243-04:00 Allsky allsky[16016]: -----
No, Auto white balance is disabled for both day and night captures, with default settings for the RPI HQ camera.
I tried to enable Auto white balance few weeks ago, but I didn't see any change.
Thanks
@jfharvey unfortunately I still cannot see any exif data in the images. As @6by9 said in an eariler comment, it might be GitHub stripping metadata when attaching the files. Could you zip up the image and attach the file instead please?
Also, has this only recently started happening, or was the problem always visible?
Hi naushir
Here are the two files zipped. But as I said, I checked on my PC before uploading them, and I can't see any metedata in the pictures, but you can double check, maybe I missed something @EricClaeys , should metadata be present in images from /Home/pi/allsky/images/yyyymmdd/ ?
The issue started when I switched to a RPI5. On my RPI4 (using the same camera), there is no "blue image" issue.
Here are the last night thumbnails. We can clearly see the issue.
and here is the zip file test.zip
@jfharvey, Allsky doesn't add any metadata to the images. I don't know if libcamera-still adds any - none of the command-line arguments tell it to.
@naushir, several other people also see the "blue images". I asked them to comment in this thread.
@davidplowman for visibility.
@jfharvey and @EricClaeys can you run rpicam-hello --version and let me know the version string that is printed. Additionally, can you enable RAW output in the command line by adding the -r option? This will produce a DNG which might be useful for debugging.
If we are sure that AWB is disabled, can I suggest that we disable the ALSC algorthim and see if that fixes the issue for you. To do this, edit /usr/share/libcamera/ipa/rpi/pisp/imx477.json. Under the rpi.alsc section, can you set the value of n_iter to 0, and try running again?
Thank you for your help in debugging this with us!
Hello naushir,
rpicam-apps build: 6202c09b2bfa 16-02-2024 (17:15:39) libcamera build: v0.2.0+46-075b54d5
I will check with @EricClaeys if it is possible to enable the RAW output and I will try again.
I will also disable the ALSC algorithm later today and see if it is better...or not.
Regards
Those builds are quite old, would it be possible to update your libcamera libraries as well please?
Thanks naushir
I'm 100% sure I did a sudo apt update/upgrade last week, after the OS installation, but I just did another one to be sure, and now, the version is the following
rpicam-apps build: d7a1a13b041e 05-09-2024 (14:07:57) libcamera build: v0.3.1+50-69a894c4
Regards
@jfharvey, you can add additional command-line arguments via the EXTRA PARAMETERS setting.
Hello, I disabled the ALSC algorithm as requested. Unfortunately, The blue image issue is still present, as you can see in the attached files.
I also included the RAW files. Maybe you will find something useful in the metadata.
Here is an excerpt from the log showing the command line used for each picture.
2024-09-17T20:20:49.579876-04:00 Allsky allsky[5081]: STARTING EXPOSURE at: 2024-09-17 20:20:49 @ 10.0 sec 2024-09-17T20:20:49.595971-04:00 Allsky allsky[5081]: > Running: libcamera-still --output '/home/pi/allsky/tmp/image-20240917202049.jpg' --raw --timeout 1 --nopreview --width 4056 --height 3040 --shutter 10000000 --analoggain 8 --quality 100 2024-09-17T20:21:01.707044-04:00 Allsky allsky[5081]: > Got: shutter_us: 10.0 sec, gain: 8.000, mean: 0.203, target mean: 0.200, diff (target - mean): -0.003 2024-09-17T20:21:01.707163-04:00 Allsky allsky[5081]: > index: 0, meanHistory[]=0.189 exposureLevelHistory[]=226, newNean=0.189 2024-09-17T20:21:01.707184-04:00 Allsky allsky[5081]: > index: 1, meanHistory[]=0.208 exposureLevelHistory[]=226, newNean=0.606 2024-09-17T20:21:01.707201-04:00 Allsky allsky[5081]: > index: 2, meanHistory[]=0.203 exposureLevelHistory[]=226, newNean=1.215 2024-09-17T20:21:01.707218-04:00 Allsky allsky[5081]: > New mean target: 0.201, mean_forecast: 0.198, mean_diff (newMean - target mean): 0.001, idx=2, idxN1=1 2024-09-17T20:21:01.707234-04:00 Allsky allsky[5081]: > ExposureChange clipped to 3 (diff from last change: 0) 2024-09-17T20:21:01.707248-04:00 Allsky allsky[5081]: > ++++++++++ Prior image mean good - no changes needed, mean=0.203, target mean=0.200 threshold=0.100 2024-09-17T20:21:01.707266-04:00 Allsky allsky[5081]: > Saving NIGHT image 'image-20240917202049.jpg' 2024-09-17T20:21:01.707515-04:00 Allsky allsky[5081]: > Sleeping: 80.0 sec
2024-09-17T20:22:21.709236-04:00 Allsky allsky[5081]: STARTING EXPOSURE at: 2024-09-17 20:22:21 @ 10.0 sec 2024-09-17T20:22:21.725597-04:00 Allsky allsky[5081]: > Running: libcamera-still --output '/home/pi/allsky/tmp/image-20240917202221.jpg' --raw --timeout 1 --nopreview --width 4056 --height 3040 --shutter 10000000 --analoggain 8 --quality 100 2024-09-17T20:22:33.834788-04:00 Allsky allsky[5081]: > Got: shutter_us: 10.0 sec, gain: 8.000, mean: 0.204, target mean: 0.200, diff (target - mean): -0.004 2024-09-17T20:22:33.834916-04:00 Allsky allsky[5081]: > index: 1, meanHistory[]=0.208 exposureLevelHistory[]=226, newNean=0.208 2024-09-17T20:22:33.834944-04:00 Allsky allsky[5081]: > index: 2, meanHistory[]=0.203 exposureLevelHistory[]=226, newNean=0.614 2024-09-17T20:22:33.834962-04:00 Allsky allsky[5081]: > index: 0, meanHistory[]=0.204 exposureLevelHistory[]=226, newNean=1.227 2024-09-17T20:22:33.834979-04:00 Allsky allsky[5081]: > New mean target: 0.205, mean_forecast: 0.205, mean_diff (newMean - target mean): 0.005, idx=0, idxN1=2 2024-09-17T20:22:33.834995-04:00 Allsky allsky[5081]: > ExposureChange clipped to 3 (diff from last change: 0) 2024-09-17T20:22:33.835010-04:00 Allsky allsky[5081]: > ++++++++++ Prior image mean good - no changes needed, mean=0.204, target mean=0.200 threshold=0.100 2024-09-17T20:22:33.835027-04:00 Allsky allsky[5081]: > Saving NIGHT image 'image-20240917202221.jpg' 2024-09-17T20:22:33.835293-04:00 Allsky allsky[5081]: > Sleeping: 80.0 sec
@naushir, here's a summary of the log output above, showing only the commands used to take the two pictures. Notice the commands are identical, yet one image was blue:
libcamera-still --output '/home/pi/allsky/tmp/image-20240917202049.jpg' --raw --timeout 1 --nopreview --width 4056 --height 3040 --shutter 10000000 --analoggain 8 --quality 100
libcamera-still --output '/home/pi/allsky/tmp/image-20240917202221.jpg' --raw --timeout 1 --nopreview --width 4056 --height 3040 --shutter 10000000 --analoggain 8 --quality 100
Thanks, these show some interesting results.
image-20240917202049.dng EXIF:
ExifTool Version Number : 12.40
File Name : image-20240917202049.dng
Directory : .
File Size : 24 MiB
File Modification Date/Time : 2024:09:18 01:21:01+01:00
File Access Date/Time : 2024:09:18 08:04:49+01:00
File Inode Change Date/Time : 2024:09:18 08:04:49+01:00
File Permissions : -rwxrwxrwx
File Type : DNG
File Type Extension : dng
MIME Type : image/x-adobe-dng
Exif Byte Order : Little-endian (Intel, II)
Make : Raspberry Pi
Camera Model Name : imx477
Orientation : Horizontal (normal)
Software : rpicam-still
Subfile Type : Full-resolution image
Image Width : 4056
Image Height : 3040
Bits Per Sample : 16
Compression : Uncompressed
Photometric Interpretation : Color Filter Array
Samples Per Pixel : 1
Planar Configuration : Chunky
CFA Repeat Pattern Dim : 2 2
CFA Pattern 2 : 2 1 1 0
Black Level Repeat Dim : 2 2
Black Level : 4096 4096 4096 4096
White Level : 65535
Strip Offsets : 0
Strip Byte Counts : 0
Exposure Time : 10
ISO : 800
Date/Time Original : 2024:09:17 20:21:01
DNG Version : 1.1.0.0
DNG Backward Version : 1.0.0.0
Unique Camera Model : Raspberry Pi imx477
Color Matrix 1 : 0.7559810877 -0.1972008348 -0.04402547702 -0.1264587343 0.896514833 0.205433175 0.003762362292 0.0730144456 0.1681934893
As Shot Neutral : 0.4733974636 1 0.2597243786
Calibration Illuminant 1 : D65
CFA Pattern : [Blue,Green][Green,Red]
Image Size : 4056x3040
Megapixels : 12.3
Shutter Speed : 10
ISP output:
dcraw output:
image-20240917202221.dng EXIF:
ExifTool Version Number : 12.40
File Name : image-20240917202221.dng
Directory : .
File Size : 24 MiB
File Modification Date/Time : 2024:09:18 01:22:33+01:00
File Access Date/Time : 2024:09:18 08:04:53+01:00
File Inode Change Date/Time : 2024:09:18 08:04:53+01:00
File Permissions : -rwxrwxrwx
File Type : DNG
File Type Extension : dng
MIME Type : image/x-adobe-dng
Exif Byte Order : Little-endian (Intel, II)
Make : Raspberry Pi
Camera Model Name : imx477
Orientation : Horizontal (normal)
Software : rpicam-still
Subfile Type : Full-resolution image
Image Width : 4056
Image Height : 3040
Bits Per Sample : 16
Compression : Uncompressed
Photometric Interpretation : Color Filter Array
Samples Per Pixel : 1
Planar Configuration : Chunky
CFA Repeat Pattern Dim : 2 2
CFA Pattern 2 : 2 1 1 0
Black Level Repeat Dim : 2 2
Black Level : 4096 4096 4096 4096
White Level : 65535
Strip Offsets : 0
Strip Byte Counts : 0
Exposure Time : 10
ISO : 800
Date/Time Original : 2024:09:17 20:22:33
DNG Version : 1.1.0.0
DNG Backward Version : 1.0.0.0
Unique Camera Model : Raspberry Pi imx477
Color Matrix 1 : 0.5130448341 -0.1162250116 -0.02393050492 -0.1746720523 0.9088459015 0.236192748 0.0329060331 0.1008179113 0.4090979695
As Shot Neutral : 0.3453533351 1 0.5775325894
Calibration Illuminant 1 : D65
CFA Pattern : [Blue,Green][Green,Red]
Image Size : 4056x3040
Megapixels : 12.3
Shutter Speed : 10
ISP output:
dcraw output:
From the above results, it seems like the blue tint is present in the RAW image as well? I assume dcraw is doing it's own processing for colours without relying on any EXIF data from the capture.
Ive run the DNGs through RawThreapee and get the same results using the built-in AWB and get exactly the same results. It seems the blue tint is genuinely present in the RAW data.
Can you share what power supply you are using on your Pi5? Is it an official Raspberry Pi supply?
Thanks nausir for your help on that issue.
I'm using a POE hat for the raspberry PI 5.
This is pure speculation, but I wonder if this is a power supply issue causing a glitch in the sensor. Would you be able to run with a regular Raspberry Pi 5 power supply and see if it eliminates the problem? You can probably just run this indoors for the test.