Improve ESP32 camera frame rates when streaming video to HA

[Rudd-O]

Describe the problem you have/What new integration you would like

Current frame rates with 640x480 of an OV2640 are > 10 FPS when streaming with MJPEG web camera component, but 2.5 FPS when watching the same feed through HA. Rates remain high when both videos are playing, however when the MJPEG component ceases to stream, rates drop to 2.5 FPS.

Similar effect is seen with the OV5640, where the MJPEG streaming rate is 6.5 FPS average, and drops to 1.5 when streaming through HA API.

(These tests were performed using a Freenove device, although other ESP devices were tested with similar results. Freenove needed a custom component with minimal edits compared to the upstream component: https://github.com/Rudd-O/esphome_freenove_camera_component )

Please describe your use case for this integration and alternatives you've tried:

A higher frame rate would enable much more uses of OV cameras with ESP32 devices.

Additional context

No additional context. I do have this code to calculate the frame rate:

camera:
# ...
    on_image:
      - component.update: camera_frame_rate

globals:
  - id: last_update_times
    type: unsigned int[$frame_rate_buffer_size]
    restore_value: no

sensor:
- platform: template
  name: "Camera frame rate"
  id: camera_frame_rate
  lambda: |-
    float now = millis();
    unsigned int max_ = $frame_rate_buffer_size;
    unsigned int i;
    for (i = max_ - 1; i > 0; i--) {
      id(last_update_times)[i] = id(last_update_times)[i-1];
    }
    id(last_update_times)[0] = now;
    float avgdiff = 0.0;
    float mostrecentmillis = id(last_update_times)[0];
    for (i = 1; i < max_; i++) {
      if (mostrecentmillis == 0) {
        ESP_LOGD("Camera", "No more updates, breaking at %u", i);
        break;
      }
      avgdiff = mostrecentmillis - id(last_update_times)[i];
      if ((id(last_update_times)[i-1] - id(last_update_times)[i]) > 9500) {
        ESP_LOGD("Camera", "Big diff, breaking at %u", i);
        if (i > 1) {
          // This is at least value number three from most
          // recent to least recent.  Undo its use.
          i = i - 1;
          avgdiff = mostrecentmillis - id(last_update_times)[i];
          ESP_LOGD("Camera", "Compensating avgdiff");
        }
        break;
      }
    }
    if (avgdiff == 0.0) {
      return {};
    } else {
      avgdiff = avgdiff / i;
      float rate = 1000.0 / avgdiff;
      //if (rate >= 5.0) {
      //  rate = floor(rate + 0.5);
      //} else {
      //  rate = floor(rate * 10 + 0.5) / 10;
      //}
      return rate;
    }
  update_interval: never
  entity_category: diagnostic
  state_class: measurement
  device_class: frequency
  filters:
    - heartbeat: 5s
    - throttle_average: 5s
    - delta: 0.25

[nagyrobi]

Don't forget that with HA API you normally have a real time encrypted connection, that may slow down a bit such transfers of bigger data packs.

[Rudd-O]

By that much? I suspect something else beyond CPU execution time might be wrong.