Add support for OV5640 camera

[jasaw]

Add support for OV5640 5MP camera. The camera has a resolution of 2560 x 1920 but the output will be scaled down to the configured resolution e.g. 640 x 480. When zoom is enabled, a 640 x 480 window is cropped from the full 2560 x 1920 frame, therefore giving double the magnification compared to OV2640.

This is a zoom comparison between OV5640 and OV2640.

OV5640:

OV2640:

[caco3]

what is the impact on the RAM usage? Does it need to store a larger framebuffer for the higher resolution?

[jasaw]

@caco3 There's no impact on RAM usage at all. It works the same way as ov2640 where cropping is done on the camera module itself.

[SybexX]

Process:

1. The camera takes an image with a resolution of 2560x1920
2. The camera cuts out the desired area from the image
3. The camera scales the cropped area to 640x480, if needed (desired area is larger 640x480) This process applies both when using the zoom function and when zoom is deactivated.

However, further changes to the implementation are necessary, e.g. at the sharpness....

Ablauf:

1. Die Kamera nimmt ein Bild mit einer Auflösung von 2560x1920 auf
2. Die Kamera schneidet den gewünschten Bereich aus dem Bild aus
3. Die Kamera skaliert den ausgeschnittenen Bereich bei Bedarf auf 640x480 (wenn der ausgeschnittene Bereich größer 640x480 ist). Dieser Ablauf wird sowohl bei der Nutzung der Zoom-Funktion als auch bei deaktivierten Zoom benutzt/verwendet.

Allerdings sind noch weitere Änderungen an der Implementierung notwendig, z.B. bei sharpness.....

[jasaw]

However, further changes to the implementation are necessary

@SybexX May I know what you would like to change?

[SybexX]

e.g. so:

I made a few quick adjustments, but I don't know if it works because I don't have an OV5640. ClassControllCamera.zip

[jasaw]

@SybexX I've adjusted the sharpness handling to your version and it's working fine on my OV5640 system. I changed the sharpness range because the officially supported sharpness range is from -3 to +3. _sharpnessLevel = min(2, max(-2, _sharpnessLevel));

Your ClassControllCamera code enforces 4:3 ratio on the zoom parameters, but OV5640 is a lot more flexible. It doesn't have the 4:3 restriction. Currently this PR only supports zoom mode 0, but it might be possible to achieve other zoom modes by passing different parameters to set_res_raw with scaling and binning set to true. That needs more experimentation, which I think could be left for future enhancement.

[SybexX]

I specifically set the sharpnessLevel to 2. After longer tests, I found that a value of 3 sometimes causes problems.

I know that the OV5640 can do more, but we should adapt everything to the OV2640, otherwise we would need at least two pages, i.e. one for each camera model and it simply becomes too confusing for the user.

Sometimes less is more^^

[SybexX]

[jasaw]

@SybexX Thank you for reviewing. I've refactored my code based on your sample code but had to make a few changes to make it work with the OV5640.

only else, otherwise the other supported camera models will be ignored

My apologies, it is not clear to me that OV3660 (which is the other supported camera) behaves the same way as OV2640 and parts of the code are hitting the OV2640 registers directly (presumably the OV3660 has the same registers as OV2640?). Personally, I prefer to be more explicit in the code when it comes to handling hardware specific behaviour, like

if (sensor_info->model == CAMERA_OV5640)
{
}
else if ((sensor_info->model == CAMERA_OV2640) || (sensor_info->model == CAMERA_OV3660))
{
}

Either way, I am not particular with how the code is written as long as it is well documented via comments or code itself so other developers understand the intention of the author.

[SybexX]

The registers don't matter, the properties of the camera are more important. The https://github.com/espressif/esp32-camera relieves us of having to pay attention to the registers. Since no distinction was made between the camera models before I made my changes, I assumed that the OV2640 and OV3660 were very similar. After I quick look, the OV3660 is very close to the OV5640 in terms of functions.

[jasaw]

The registers don't matter

ov2640_set_sharpness and ov2640_enable_auto_sharpness functions are called for OV3660 camera as well and eventually those functions call sensor->set_reg which writes directly to the OV3660's registers. Keep in mind that OV2640 sharpness is not officially supported by Espressif SDK. I haven't looked into the OV3660 datasheet, so I don't know what the camera will do if those registers don't exist or worse, have a different function. I don't have an OV3660 to test, so can't validate.

Since no distinction was made between the camera models before I made my changes, I assumed that the OV2640 and OV3660 were very similar

That was an oversight on my part as well. We should probably test sharpness against OV3660 and update the code accordingly?

[SybexX]

If something is not available on a camera model, the esp32-camera library actually returns a -1

https://github.com/espressif/esp32-camera/blob/master/sensors/ov2640.c https://github.com/espressif/esp32-camera/blob/master/sensors/ov3660.c https://github.com/espressif/esp32-camera/blob/master/sensors/ov5640.c

[jasaw]

I propose this sharpness handling code:

void CCamera::SetCamSharpness(bool _autoSharpnessEnabled, int _sharpnessLevel)
{
    sensor_t *s = esp_camera_sensor_get();

    if (s != NULL)
    {
        _sharpnessLevel = min(2, max(-2, _sharpnessLevel));

        camera_sensor_info_t *sensor_info = esp_camera_sensor_get_info(&(s->id));

        if (sensor_info != NULL)
        {
            if (sensor_info->model == CAMERA_OV5640 || sensor_info->model == CAMERA_OV3660)
            {
                if (_autoSharpnessEnabled)
                {
                    // autoSharpness is not supported, default to zero
                    s->set_sharpness(s, 0);
                }
                else
                {
                    s->set_sharpness(s, _sharpnessLevel);
                }
            }
            else if (sensor_info->model == CAMERA_OV2640)
            {
                // The OV2640 does not officially support sharpness, so the detour is made with the ov2640_sharpness.cpp.
                if (_autoSharpnessEnabled)
                {
                    s->set_sharpness(s, 0);
                    ov2640_enable_auto_sharpness(s);
                }
                else
                {
                    ov2640_set_sharpness(s, _sharpnessLevel);
                }
            }
        }
    }
    else
    {
        LogFile.WriteToFile(ESP_LOG_ERROR, TAG, "SetCamSharpness, Failed to get Cam control structure");
    }
}

[jasaw]

@SybexX Looks like OV3660 set_res_raw function is the same as OV5640 as well. I should probably handle OV3660 explicitly.

[SybexX]

or so, then the ESP is spared the second comparison ^^

void CCamera::SetCamSharpness(bool _autoSharpnessEnabled, int _sharpnessLevel) { sensor_t *s = esp_camera_sensor_get();

if (s != NULL)
{
    _sharpnessLevel = min(2, max(-2, _sharpnessLevel));

    camera_sensor_info_t *sensor_info = esp_camera_sensor_get_info(&(s->id));

    if (sensor_info != NULL)
    {
        if (sensor_info->model == CAMERA_OV2640)
        {
            // The OV2640 does not officially support sharpness, so the detour is made with the ov2640_sharpness.cpp.
            if (_autoSharpnessEnabled)
            {
                s->set_sharpness(s, 0);
                ov2640_enable_auto_sharpness(s);
            }
            else
            {
                ov2640_set_sharpness(s, _sharpnessLevel);
            }
        }
        else
        {
			// for CAMERA_OV5640 and CAMERA_OV3660
            if (_autoSharpnessEnabled)
            {
                // autoSharpness is not supported, default to zero
                s->set_sharpness(s, 0);
            }
            else
            {
                s->set_sharpness(s, _sharpnessLevel);
            }
        }			
    }
}
else
{
    LogFile.WriteToFile(ESP_LOG_ERROR, TAG, "SetCamSharpness, Failed to get Cam control structure");
}

}

[SybexX]

        if (sensor_info->model == CAMERA_OV2640)
        {
            // The OV2640 does not officially support sharpness, so the detour is made with the ov2640_sharpness.cpp.
            if (_autoSharpnessEnabled)
            {
                s->set_sharpness(s, 0);	<<<<<<<<<< oh yes, this is actually not necessary, as it only returns a -1(not supported)
                ov2640_enable_auto_sharpness(s);
            }
            else
            {
                ov2640_set_sharpness(s, _sharpnessLevel);
            }
        }

[jasaw]

@SybexX I have updated the code to your suggested sharpness handling and also updated zoom handling code to better deal with OV3660 but I don't have an OV3660 so can't test. I have only tested with OV5640 so far.

[SybexX]

            case CAMERA_OV5640:
                frameSizeX = 2560;
                frameSizeY = 1920;
                // max imageSize = ((frameSizeX - CCstatus.ImageWidth) / 8 / 4) - 1
                // 59 = ((2560 - 640) / 8 / 4) - 1
                if (imageSize < 59)
                {
                    _imageSize_temp = (59 - imageSize);
                }
                SanitizeZoomParams(_imageSize_temp, frameSizeX, frameSizeY, _imageWidth, _imageHeight, _offsetx, _offsety);
                SetCamWindow(s, unused, frameSizeX, frameSizeY, _offsetx, _offsety, _imageWidth, _imageHeight, CCstatus.ImageWidth, CCstatus.ImageHeight);
                break;

            case CAMERA_OV3660:
                frameSizeX = 2048;
                frameSizeY = 1536;
                // max imageSize = ((frameSizeX - CCstatus.ImageWidth) / 8 / 4) -1
                // 43 = ((2048 - 640) / 8 / 4) - 1
                if (imageSize < 43)
                {
                    _imageSize_temp = (43 - imageSize);
                }
                SanitizeZoomParams(_imageSize_temp, frameSizeX, frameSizeY, _imageWidth, _imageHeight, _offsetx, _offsety);
                SetCamWindow(s, unused, frameSizeX, frameSizeY, _offsetx, _offsety, _imageWidth, _imageHeight, CCstatus.ImageWidth, CCstatus.ImageHeight);
                break;

            case CAMERA_OV2640:
                // ov2640_sensor_mode_t _mode = OV2640_MODE_UXGA; // 1600x1200
                // ov2640_sensor_mode_t _mode = OV2640_MODE_SVGA; // 800x600
                // ov2640_sensor_mode_t _mode = OV2640_MODE_CIF;  // 400x296
                _mode = 0;
                frameSizeX = 1600;
                frameSizeY = 1200;
                // max imageSize = ((frameSizeX - CCstatus.ImageWidth) / 8 / 4) -1
                // 29 = ((1600 - 640) / 8 / 4) - 1
                if (imageSize < 29)
                {
                    _imageSize_temp = (29 - imageSize);
                }
                SanitizeZoomParams(_imageSize_temp, frameSizeX, frameSizeY, _imageWidth, _imageHeight, _offsetx, _offsety);
                // _mode sets the sensor resolution (3 options available),
                // _offsetx and _offsety set the start of the ROI,
                // _imageWidth and _imageHeight set the size of the ROI,
                // CCstatus.ImageWidth and CCstatus.ImageHeight set the output window size.
                SetCamWindow(s, _mode, frameSizeX, frameSizeY, _offsetx, _offsety, _imageWidth, _imageHeight, CCstatus.ImageWidth, CCstatus.ImageHeight);
                break;

[jasaw]

@SybexX The new imageSize code that you shared above appears to be working well on my OV5640 setup and thanks for adding the comments to explain how the imageSize numbers came from because I had no idea previously.

I've also increased the OV5640 full frame size to match the datasheet.

[SybexX]

still necessary adjustment in the edit_reference.html:

        <td>
                <input required type="number" id="TakeImage_CamZoomSize_value1" value="0" min="0" max="59" step="1" onchange="cameraParameterChanged()"
                oninput="(!validity.rangeOverflow||(value=59)) && (!validity.rangeUnderflow||(value=0)) && (!validity.stepMismatch||(value=parseInt(this.value)));">
        </td>

still necessary adjustment in the edit_config_template.html:

		<td>
			<input required type="number" id="TakeImage_CamZoomSize_value1" value="0" min="0" max="59" step="1" onchange="cameraParameterChanged()"
			oninput="(!validity.rangeOverflow||(value=59)) && (!validity.rangeUnderflow||(value=0)) && (!validity.stepMismatch||(value=parseInt(this.value)));">
		</td>
		<td>$TOOLTIP_TakeImage_CamZoomSize</td>			
	</tr>

	<tr class="expert" unused_id="TakeImage_CamZoomOffsetX_ex3">
		<td class="indent1">
			<class id="TakeImage_CamZoomOffsetX_text" style="color:black;">Zoom Offset X</class>
		</td>
		<td>
			<input required type="number" id="TakeImage_CamZoomOffsetX_value1" value="0" min="-960" max="960" step="8" onchange="cameraParameterChanged()"
			oninput="(!validity.rangeOverflow||(value=960)) && (!validity.rangeUnderflow||(value=-960)) && (!validity.stepMismatch||(value=parseInt(this.value)));">Pixel
		</td>
		<td>$TOOLTIP_TakeImage_CamZoomOffsetX</td>
	</tr>

	<tr class="expert" unused_id="TakeImage_CamZoomOffsetY_ex3">
		<td class="indent1">
			<class id="TakeImage_CamZoomOffsetY_text" style="color:black;">Zoom Offset Y</class>
		</td>
		<td>
			<input required type="number" id="TakeImage_CamZoomOffsetY_value1" value="0" min="-720" max="720" step="8" onchange="cameraParameterChanged()"
			oninput="(!validity.rangeOverflow||(value=720)) && (!validity.rangeUnderflow||(value=-720)) && (!validity.stepMismatch||(value=parseInt(this.value)));">Pixel
		</td>
		<td>$TOOLTIP_TakeImage_CamZoomOffsetY</td>
	</tr>

[SybexX]

@jasaw Have you already tested it with an OV2640? For me it always comes to a reboot at take_image, there must be something wrong with the calculation.

[SybexX]

@jasaw I made a few changes so that the OV2640 works again. test.txt

[jasaw]

@SybexX Thank you for testing with OV2640. I have been testing with OV5640 and haven't got time to test against OV2640.

I have incorporated your recommended changes. Thanks for fixing it and adding comments to explain how imageSize is supposed to work. I have added your comments in the code to help other developers.

I have tested your multi-step zoom and it is a very nice idea. I have to use scale and binning for OV5640 to work. I don't have an OV3660 so I can only assume it works the same way as OV5640.

I have also been testing the OV5640 in less than perfect lighting conditions and have to bump the max jpeg quality down to 18 because the camera is unstable below 18. I noticed you changed the max jpeg quality to 6 for OV2640. Is the OV2640 stable at 6 in less than perfect lightning conditions?

[SybexX]

I always test on the water meter and there is no problem with the quality = 6. Have you tried increasing the frequency, it is set to 20MHz, but I think the OV5640 can go up to 27MHz

[jasaw]

Have you tried increasing the frequency, it is set to 20MHz, but I think the OV5640 can go up to 27MHz

The OV5640 datasheet recommends 24MHz and I tried that but it didn't work properly. The image was always half black with some ghosting and the other half of the image was normal.

[jasaw]

@SybexX I've checked other examples of running OV5640 on ESP32 and they are running 20MHz clock from what I can see. Not sure what's the maximum the ESP32 + OV5640 combo can handle.

I have just found out that both OV3660 and OV5640 are different from OV2640 in these areas as well:

1. OV3660 and OV5640 support de-noise. The range is from 0 to 8 where zero is auto-denoise. OV2640 does not support de-noise.
2. OV3660 and OV5640 auto-exposure range is from -5 to +5 whereas OV2640 range is from 1 to 5.
3. OV3660 and OV5640 sharpness range is from from -3 to +3 but no auto-sharpness whereas we limit our OV2640 range to -2 to +2 and ov2640_set_sharpness takes -3 to +3.

What would you like to do? I'm leaning towards increasing the sharpness range to -3 to +3 (clip it to -2 to +2 for OV2640 with a comment saying -3 or +3 are unstable), increasing the auto-exposure range to -5 to +5 but clip it internally for OV2640, and add de-noise option on the web interface.

[SybexX]

Personally, I would limit myself to the functions of the OV2640. Users won't change most of the settings anyway if the image quality is right. That's why I only put the most necessary settings on the reference page and the rest is only available on the settings page. It is best if @jomjol or @caco3 comments on what and how he imagines the implementation of the available settings and their setting range.

[friedpa]

I personnaly think that the AutoSharpness (if that is the same as AutoFocus) is most important, as a lot of users have difficulties with the manual lens adjustment. So it could be better to change to the new camera as a default hardware.

[SybexX]

AutoSharpness and AutoFocus are two different functions. The problem is that every auto function can change the image quality every time and this making the evaluation a game of chance (in addition, this may place more strain on the ESP during alignment and evaluation). For a reasonable and consistent evaluation, the image quality must always remain the same (In theory you would have to remove and deactivate every auto function).

[jasaw]

For my gas meter and water meter set up, I can't have a fully enclosed housing because meter readers come around every couple of months to read the meters and they need to be able to look at the digits on the meters. What this means is my image is highly subjected to ambient lighting (sun light), so auto-exposure needs to be on. Other auto functions may need to be turned on to get an optimal image. I have also turned on negative effect (I used to have both grayscale and negative). I'm quite impressed that the neural net has been getting 95% of the readings correctly. Long story short, auto features are important if lighting is not constant.

Regarding the focus issue, I agree that it's hard to get the focus right, which is why I have been researching the OV5640 with autofocus. I think it will be a more user friendly solution. The user adjusts the focus through the web interface during edit-reference step and leave the focus at the configured level during normal operation (no auto-focus). We would need to proof that focus can be adjusted manually first. I have shared my research here: https://github.com/jomjol/AI-on-the-edge-device/issues/2162#issuecomment-2102013159

As for the OV3660 and OV5640 specific features (de-noise and bigger auto-exposure range), I'm happy to just have them on the config page as expert settings.