How to save high-quality static images with controls i.e. auto white-balance, focus and exposure?

[mmortazavi]

The following piece of code works by simply saving a frame on a disk.

import cv2
import time
import depthai as dai

pipeline = dai.Pipeline()

camRgb = pipeline.createColorCamera()

xoutRgb = pipeline.createXLinkOut()
xoutRgb.setStreamName("rgb")
camRgb.preview.link(xoutRgb.input)

device_info = dai.DeviceInfo()
device_info.state = dai.XLinkDeviceState.X_LINK_BOOTLOADER
device_info.desc.protocol = dai.XLinkProtocol.X_LINK_TCP_IP
device_info.desc.name = "169.254.1.222"

with dai.Device(pipeline, device_info) as device:
    # current datetime
    now = time.strftime("%Y%m%d-%H%M%S")
    filename = f"Test_{now}.jpg"
    qRgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
    frame = qRgb.get().getCvFrame()
    print(f"{frame.shape}")
    cv2.imwrite(f"data/{filename}", frame)

But image has a low quality, it is blur, and exposure is not good. What I seek to achieve is to control the camera e.g. activating auto white-balance, auto focus, auto expose, and set the resolution to maximum 12 MP and save the frame to the disk. Whatever I do so far doesn't work, I have tried:

import cv2
import time
import depthai as dai


# Create pipeline
pipeline = dai.Pipeline()

# Define sources and outputs
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
stillEncoder = pipeline.create(dai.node.VideoEncoder)

controlIn = pipeline.create(dai.node.XLinkIn)
stillMjpegOut = pipeline.create(dai.node.XLinkOut)
previewOut = pipeline.create(dai.node.XLinkOut)

# Stream Names
controlIn.setStreamName('control')
stillMjpegOut.setStreamName('still')
previewOut.setStreamName('preview')

# Properties
camRgb.setPreviewSize(1200, 800)
stillEncoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)

# Linking
camRgb.still.link(stillEncoder.input)
camRgb.preview.link(previewOut.input)
controlIn.out.link(camRgb.inputControl)
stillEncoder.bitstream.link(stillMjpegOut.input)

# Connect to device and start pipeline
with dai.Device(pipeline) as device:

    # Get data queues
    controlQueue = device.getInputQueue('control')
    stillQueue = device.getOutputQueue('still')

    print("Autoexposure enable")
    ctrl = dai.CameraControl()
    ctrl.setAutoExposureEnable()
    controlQueue.send(ctrl)

    print("Auto white-balance enable")
    ctrl.setAutoWhiteBalanceMode(dai.CameraControl.AutoWhiteBalanceMode.AUTO)
    controlQueue.send(ctrl)

    print("Autofocus trigger (and disable continuous)")
    ctrl = dai.CameraControl()
    ctrl.setAutoFocusMode(dai.CameraControl.AutoFocusMode.AUTO)
    ctrl.setAutoFocusTrigger()
    controlQueue.send(ctrl)

    frame = stillQueue.get().getCvFrame()

    if frame is not None:

        # current datetime
        now = time.strftime("%Y%m%d-%H%M%S")
        filename = f"Test_{now}.jpg"
        print(f"Frame Shape: {frame.shape}")
        cv2.imwrite(f"data/{filename}", frame)

The script just hangs after setting the controls!

[Luxonis-Brandon]

We will circle back with help here. Sorry about the trouble.

[alex-luxonis]

Hi @mmortazavi,

For the first issue (first code snippet), 3A (auto- exposure, focus and white-balance) needs some time to converge. First images won't look good.

For the hang issue with the second snippet, the still output of ColorCamera is conditioned by sending a setCaptureStill(True) command on the control input. Adding this code should fix it:

    ctrl = dai.CameraControl()
    ctrl.setCaptureStill(True)
    controlQueue.send(ctrl)

But there's one more issue writing the file with cv2.imwrite, as getCvFrame in this case doesn't actually decode the JPEG frame, but returns it plain (can be observed from checking its shape). Maybe we should fix that in the host code.

Could you check if the code below works for you. Can change getJpeg = False to receive an uncompressed 12MP frame.

import cv2
import time
import depthai as dai

# When set True, will use VideoEncoder to compress on-device.
# Otherwise, will receive an uncompressed NV12 frame from `ColorCamera.still`,
# that getCvFrame() will convert to BGR
getJpeg = True  # or False

pipeline = dai.Pipeline()

camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setPreviewSize(1200, 800)

if getJpeg:
    stillEncoder = pipeline.create(dai.node.VideoEncoder)
    stillEncoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)

controlIn = pipeline.create(dai.node.XLinkIn)
stillOut = pipeline.create(dai.node.XLinkOut)
previewOut = pipeline.create(dai.node.XLinkOut)

controlIn.setStreamName('control')
stillOut.setStreamName('still')
previewOut.setStreamName('preview')

if getJpeg:
    camRgb.still.link(stillEncoder.input)
    stillEncoder.bitstream.link(stillOut.input)
else:
    camRgb.still.link(stillOut.input)
controlIn.out.link(camRgb.inputControl)
# Note: if not reading from the `preview` queue (as it happens below),
# it may block other ColorCamera outputs. Either do not link it (next line commented out)...
#camRgb.preview.link(previewOut.input)
# ... or have it linked, but set its device-side queue as non-blocking:
#previewOut.input.setBlocking(False)
#previewOut.input.setQueueSize(1)

with dai.Device(pipeline) as device:
    controlQueue = device.getInputQueue('control')
    stillQueue = device.getOutputQueue('still')

    # Wait a bit for 3A to converge, before requesting a still capture
    time.sleep(1)

    # Request a still image capture
    ctrl = dai.CameraControl()
    ctrl.setCaptureStill(True)
    controlQueue.send(ctrl)

    frame = stillQueue.get().getCvFrame()

    # current datetime
    now = time.strftime("%Y%m%d-%H%M%S")
    print(f"Frame Shape: {frame.shape}")
    if getJpeg:
        filename = f"Test_{now}.jpg"
        # Importantly, imwrite won't work here, as it's a plain JPEG
        frame.tofile(f"data/{filename}")
    else:
        # may as well be .jpg, but our received frame is uncompressed, so use lossless PNG
        filename = f"Test_{now}.png"
        cv2.imwrite(f"data/{filename}", frame)

[mmortazavi]

@alex-luxonis Thanks for your quick response. I have tested the second code snippet. It works, but there are a few remaining remarks/questions.

• Where do you in the code activate 3A? I see you the 'control', but nothing about 3A explicitly! Can you explain?
• 3A takes long time to converge, for instance here even 2 sec is not enough, I chose 3 sec. Which one of these 3A takes longer time? I am assuming the following order: auto focus > auto exposure > auto white-balance. How can deactivate auto-focus (fixed distance)? Shall I converge it once and then next images are focused? Would you please comment here?
• Field of View (FoV) can also be changed in the code (maybe by the preview size?)? Or it is changed by the lens distance?
• The image I get, even with 12MP, is not sharp enough. It looks like a 3-4 MP image. Of course that also related to the FoV i.e. pixel/mm (or pixel size X × Y µm or depth per inch (dpi)), which circles back to the previous question.

The primary application I have in mind using the camera is visual inspection in industry, usually done in less 500 msec. One scenario is capturing high-quality image of an object with 25x35 cm dimension, and decode a QR-code with 2x2mm dimension. So far, the above attempts doesn't give good enough pixel size quality. I need a minimum of 80 dpi.

[alex-luxonis]

• Where do you in the code activate 3A? I see you the 'control', but nothing about 3A explicitly! Can you explain?

3A are enabled by default on the camera, so explicitly sending the controls is not required. For auto-focus, it's set by default in CONTINUOUS_VIDEO mode. It can be changed to AUTO - in this case auto-focus will move the lens only once after initialization. Can be set also through initial controls when creating the pipeline (and not through inputControl at runtime):

camRgb.initialControl.setAutoFocusMode(dai.RawCameraControl.AutoFocusMode.AUTO)

• 3A takes long time to converge, for instance here even 2 sec is not enough, I chose 3 sec. Which one of these 3A takes longer time? I am assuming the following order: auto focus > auto exposure > auto white-balance. How can deactivate auto-focus (fixed distance)? Shall I converge it once and then next images are focused? Would you please comment here?

If the above (AutoFocusMode.AUTO) doesn't give better results, can set manual focus with the raw lens position as argument:

camRgb.initialControl.setManualFocus(140)

• Field of View (FoV) can also be changed in the code (maybe by the preview size?)? Or it is changed by the lens distance?

Through hardware, FoV/zoom change is not possible with these camera modules. Changing the focus / lens position will have though a very minor effect on the FoV. It's possible to set cropping with setVideoSize, and then the resulting area can be scaled with setPreviewSize - as it's done in the rgb_camera_control.py sample, but the image quality won't change through cropping.

• The image I get, even with 12MP, is not sharp enough. It looks like a 3-4 MP image. Of course that also related to the FoV i.e. pixel/mm (or pixel size X × Y µm or depth per inch (dpi)), which circles back to the previous question.

Try applying manual focus and see if the end results get better. If not, then maybe a different camera module (with a different lens and/or narrower FoV) could give better results. We support attaching custom camera modules (many from Arducam, or RPi HQ camera) with the OAK-FFC-3P baseboard.

The primary application I have in mind using the camera is visual inspection in industry, usually done in less 500 msec. One scenario is capturing high-quality image of an object with 25x35 cm dimension, and decode a QR-code with 2x2mm dimension. So far, the above attempts doesn't give good enough pixel size quality. I need a minimum of 80 dpi.

For the < 500ms aspect, the camera should be kept running (not init + single capture as in the scripts shared above), and extra interaction is possible from host, could also be aided by the Script node (on-device Python3).

Also, could you test with the latest release 2.14.1.0, we fixed a regression from 2.14.0.0 regarding initial controls.