Toupcam flats issue

[ajeb78]

Certain durations of flats cause extreme gradients that do not correspond to the vignetting actually present in the optical system. My hardware is a toupcam IMX571 mono camera. With my flat panel, if I take a blue flat of 0.199s the flat is as expected, but not quite up to the ADU level I would want. If I change the exposure to 0.200s, the flat has a very extreme gradient, with the top half of the image completely saturated. I get the same issue with Oiii flats (these take around 3s with my flat panel) but with a less extreme gradient. There is no light leak and the pattern is completely different to the 0.199s exposure. I can go as close as I like to 0.200s and I get behaviour as expected, but as soon as I go to 0.200s the behaviour is pathological. Strangely for very long flats (my Ha and Sii flats are about 30s as my EL panel doesn't put out much light at those wavelengths) the results are fine, and if I turn the brightness on my panel right down so that Oiii flats take 30s they are fine too, so it doesn't seem to be a time-related issue.

The issue definitely isn't caused by a light leak, as I can obtain correct (but slightly too dim) flats at a very slightly shorter exposure, and then for a 0.5% increase in exposure length half the image is saturated.

Toupcam settings: gain 160, high full well on, low noise mode on, high conversion gain, heater on level 3, cooler on set to -10C, speed 0, FPS limit 0.

This appears to be a new issue, I'm not sure when I first noticed it (I've had very few clear nights this year but it was all working fine in the spring).

Expected behavior Flats look correct, gradually increasing in brightness until they become saturated.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Kubuntu 23.04
• indi-toupcam 1.54.2+202309301853~ubuntu23.04.1
• kstars-bleeding 3.6.7+202310142136~ubuntu23.04.1

[knro]

@touptek Your thoughts on this issue?

[OGMAvision]

Hi, Juan from OGMA here. There are two elements in play here. Before assuming that it is related to the camera, let's test whether the problem comes from the flat panel. Could you take a few flats with a t-shirt against the sky and report back?

[ajeb78]

Hi, I've done that. The issue is the same with flats taken with a t-shirt, both with the telescope pointed at the sky (brighter) and with it pointed at the ceiling (dimmer). The gradient is harsher when the overall illumination is brighter, but there is still a qualitative change in the pattern of the flats when going from 200ms to 201ms that cannot be explained simply by a 0.5% increase in incident light during the exposure. (Sorry, I think I posted incorrect times in my initial post, I said 199ms -> 200ms but it's actually 200ms ->201ms. Even a value of 0.200001s gives the same incorrect gradient.)

I've saved some examples as FITS so you have all the metadata:

1. Flat taken with the panel at maximum brightness at 200ms: no gradient, flat looks as expected. here
2. Flat taken with the panel at maximum brightness at 201ms: extreme gradient, half of flat is saturated. here
3. Flat taken with the panel at minimum brightness at 200ms: no gradient, flat looks as expected (but dimmer). here
4. Flat taken with the panel at minimum brightness at 201ms: gradient. It's not as extreme as before but you can see it is qualitatively different to the actual vignetting of the optical train. here
5. Flat taken with a t-shirt at 200ms: no gradient, flat looks as expected. here
6. Flat taken with a t-shirt at 201ms: gradient is back. here
7. Brighter flat taken with a t-shirt pointed at the sky at 200ms: bright but not saturated, and looks as expected. here
8. Same flat taken at 201ms: very harsh gradient is back again, image is mostly saturated. here

From ekos, here is some more data on my camera: Model: ATR3CMOS26000KMA Production date: 20220524 SN: TP22054112418ED9B1787DC3C400AA Firmware version: 3.6.1.20210702 Hardware version: 3.0 Revision: 1 SDK Version: 54.23231.20230823

And I hadn't spotted that the indi toupcam driver has 2 packages in ubuntu: I mentioned the indi-toupcam version above, but my indi-toupbase version is 2.1+t202309302047~ubuntu23.04.1.

I wondered whether it might be related to this issue that was posted on the APT forums, which also relates to Touptek cameras producing very similarly erroneous flats - the camera in that thread looks to be a color one, but the pattern looks very similar to what I'm seeing. The user in that case has identified that it was introduced by some version of the Toupcam SDK that changed between what is bundled with APT v4.01 and v4.10, but I'm not sure what that corresponds to in terms of Touptek SDK numbers. The date of that thread also chimes with my feeling that this is something new that has appeared over the summer.

Any more data I can provide, let me know.

[ajeb78]

I just noticed there was an updated toupbase package available (2.1+t202311131302~ubuntu23.04.01). Just so that I've been testing with the latest available versions, I've installed it and can confirm that it makes no difference.

And just for completeness I turned on logging, and redid the 0.20000 and 0.200001s flats: here is the log output. As before the 0.20000s flat is perfect, the 0.200001s flat is saturated with the extreme gradient.

[2023-12-01T11:48:37.572 GMT INFO ][   org.kde.kstars.ekos.capture] - "Capture exposure = 0.2 sec, type = Flat, filter = G, upload mode = 0, batch mode = false, seq prefix = Flat_G_0.200_secs, format = Mono 16, encoding = FITS, binning = 1x1, ROI = (0+6224, 0+6224)"
[2023-12-01T11:48:37.572 GMT INFO ][   org.kde.kstars.ekos.capture] - "Capturing 0.200-second G image..."
[2023-12-01T11:48:37.636 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] eventCallBack: 0x00000001 "
[2023-12-01T11:48:40.050 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] eventCallBack: 0x00000004 "
[2023-12-01T11:48:40.063 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] Image received. Width: 6224, Height: 4168, flag: 3, timestamp: 467044736 "
[2023-12-01T11:48:40.064 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] Exposure complete "
[2023-12-01T11:48:40.182 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] Uploading file. Ext: fits, Size: 51891840, sendImage? Yes, saveImage? No "
[2023-12-01T11:48:40.187 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] BLOB transfer took 0.00447118 seconds "
[2023-12-01T11:48:40.187 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] Upload complete "
[2023-12-01T11:48:41.821 GMT DEBG ][           org.kde.kstars.indi] - Image received. Mode: "Normal" Size: 51891840
[2023-12-01T11:48:42.371 GMT INFO ][   org.kde.kstars.ekos.capture] - "Captured /tmp/image.fits"
[2023-12-01T11:48:42.378 GMT INFO ][   org.kde.kstars.ekos.capture] - "Download Time: 2.19 s, New Download Time Estimate: 2.08 s."
[2023-12-01T11:48:46.309 GMT INFO ][   org.kde.kstars.ekos.capture] - "Capture exposure = 0.200001 sec, type = Flat, filter = G, upload mode = 0, batch mode = false, seq prefix = Flat_G_0.200_secs, format = Mono 16, encoding = FITS, binning = 1x1, ROI = (0+6224, 0+6224)"
[2023-12-01T11:48:46.310 GMT INFO ][   org.kde.kstars.ekos.capture] - "Capturing 0.200-second G image..."
[2023-12-01T11:48:46.535 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] eventCallBack: 0x00000001 "
[2023-12-01T11:48:48.780 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] eventCallBack: 0x00000004 "
[2023-12-01T11:48:48.795 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] Image received. Width: 6224, Height: 4168, flag: 3, timestamp: 475783233 "
[2023-12-01T11:48:48.796 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] Exposure complete "
[2023-12-01T11:48:48.933 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] Uploading file. Ext: fits, Size: 51891840, sendImage? Yes, saveImage? No "
[2023-12-01T11:48:48.937 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] BLOB transfer took 0.00347081 seconds "
[2023-12-01T11:48:48.937 GMT DEBG ][           org.kde.kstars.indi] - ToupTek ATR3CMOS26000KMA : "[DEBUG] Upload complete "
[2023-12-01T11:48:50.597 GMT DEBG ][           org.kde.kstars.indi] - Image received. Mode: "Normal" Size: 51891840
[2023-12-01T11:48:51.060 GMT INFO ][   org.kde.kstars.ekos.capture] - "Captured /tmp/image.fits"
[2023-12-01T11:48:51.069 GMT INFO ][   org.kde.kstars.ekos.capture] - "Download Time: 2.13 s, New Download Time Estimate: 2.09 s."

[knro]

@touptek Any update on this issue?

[ajeb78]

@touptek

I've been doing a bit of experimentation and can narrow down the bug to a problem with the change to the Touptek SDK between indi-3rdparty v2.0.2 and v2.0.3.

With the Toupcam SDK v54.22587.20230516 the flats work perfectly, the symptoms of the bug as described above do not occur at all.

If I change to the Toupcam SDK v54.23041.20230731 the bug described above appears as described: the symptoms remain the same with all subsequent versions of the SDK. By switching back and forth between the SDK versions I can make the bug appear / disappear with 100% reliability.

I hope that helps narrow it down further. If it helps, I can test any intermediate versions of the library if you can make them available to me: that might help narrow it down to the commit that caused the bug.

[ajeb78]

I've just upgraded to the latest indi version (indi-toupbase/mantic,now 2.1+t202404120657~ubuntu23.10.1) and can confirm that the problem is resolved in this version of the firmware. I can take flats of any length without issue. Thank you!