Porting of OpenMV

[krzychb]

This is to open discussion how to port OpenMV to ESP32 and what it entails.

Initial ideas:

1. Write shims for the STM32 DCMI peripheral functions, translating them into I2S calls
2. Abstract away hardware-specific part of working with DCMI, using some generic API. Such an API could be implemented both for the ESP32 and STM32 targets. - https://github.com/igrr/esp32-cam-demo/pull/8#issuecomment-264704990

3. Initially we can start porting just the camera and image processing related parts of OpenMV - https://github.com/igrr/esp32-cam-demo/issues/9#issuecomment-265139008

Basing on overview of DCMI, here is the list of DCMI features (add more if missing):

1. From 8 to 14-bit parallel interface, data formats: 8/10/12/14-bit progressive scan; YCbCr 4:2:2 progressive scan; RGB 565 progressive video and compressed data: JPEG
2. Continuous or snapshot capture mode
3. Crop feature (except JPEG)
4. DCMI_PIXCLK up to 54MHz
5. AHB bus
6. 8x32 (8-level) FIFO to accommodate for any DMA response latency
7. H/W image format conversion by extracting specific bytes from FIFO
8. Interrupts: IT_LINE, IT_FRAME, IT_OVR, IT_VSYNC and IT_ERR
9. Low-power mode: Run, Sleep, Stop and Standby

Basing on my understanding of ESP32 internals, I would implement DCMI features in ESP32 as follows:

1. All numbers of bits are supported by I2C. Potential limitations: not enough memory to store bigger or >8 bit images, GPIOs already used for other peripherals
2. Snapshot is already implemented as camera_run function. I assume continuous capture shouldn't be an issue unless there are some perforce limitations of RTOS.
3. Should be implemented in s/w as post processing of already captured image
4. ESP32 supports up to 40MHz pixel clock
5. I am not sure what of AHB bus may be replicated by ESP32 h/w and what should be supplemented by s/w. I think they key is data streaming over Wi-Fi / ethernet, LCD and SD card support
6. There is no I2S documentation yet to say what intermediate storage is provided
7. Should be implemented in s/w like already done line_filter_task
8. IT_LINE, IT_FRAME, IT_VSYNC interrupts may be programmed for pins receiving these signals from camera. IT_OVR, IT_ERR may be replaced by s/w events
9. I assume ESP32 can support these modes but did not check specific functionality.

How do you see replication of DCMI functionality in ESP32?

[Oitzu]

1.) OpenMV is currently only using the 8-bit interface, or? How much free memory do we actually have? I have no idea how to determine that.

DCMI isn't that deeply implemented in OpenMV, as far as i see, some sort of abstraction should do the trick, for dcmi.

[igrr]

Very nice list! Yes, i think we can limit the implementation to 8 bit for now.

You can determine the amount of free heap using xPortGetFreeHeapSize function.

[Oitzu]

You can determine the amount of free heap using xPortGetFreeHeapSize function.

What are the actual numbers in case of the demo code? As said... i'm unfortunately still waiting on delivery of my camera. :(

[krzychb]

printf("Free heap: %u\n", xPortGetFreeHeapSize());

• before camera_init - 263,864 bytes
• after camera_init (QVGA) - 181,668 bytes

[igrr]

I would expect that once we add Micropython (or god forbid, WiFi) to the mix, we will have much less memory. On the other hand, we may just as well do some porting in advance, and then place framebuffer into the external SRAM once we have better support for it in the ESP-IDF.

[Oitzu]

Just for the protocol: With WiFi enabled i got a free heap of 106.676 bytes (QVGA)