raspberrypi
[BUG] Get empty frames with "start_recording" on custom output
Describe the bug In picamera(1) I could easily output frames to a custom "output" object.
With picamera2 I'm getting mostly empty frames.
To Reproduce I have this non-optimal script that can reproduce the issue. It creates a custom output that just pushes raw frames through a queue to be displayed by opencv
import cv2
import time
import numpy as np
import queue
from picamera2 import Picamera2
from picamera2.encoders import Encoder
from picamera2.outputs import Output
class RawFrameProcessor(Output):
def __init__(self, frame_queue):
super().__init__()
self.frame_queue = frame_queue # Queue to send raw frame bytes to the main thread
def outputframe(self, frame, keyframe=True, timestamp=None, packet=None, audio=False):
try:
raw_data = frame.read() # Read raw RGB888 bytes
if len(raw_data) == 0:
print("empty frame") # lands here most of the time
return
print("got frame of len:", len(raw_data))
if not self.frame_queue.full(): # Avoid blocking if the queue is full
print("pushing frame")
self.frame_queue.put(raw_data)
except Exception as e:
print(f"Error in outputframe: {e}")
# Initialize frame queue
frame_queue = queue.Queue(maxsize=1) # Keep only the latest frame
# Initialize Picamera2
picam2 = Picamera2()
width, height = 320, 240
config = picam2.create_video_configuration(main={"format": "RGB888", "size": (width, height)})
picam2.configure(config)
encoder = Encoder()
frame_processor = RawFrameProcessor(frame_queue)
cv2.startWindowThread()
# Start camera
picam2.start_recording(encoder, output=frame_processor)
# Main loop for processing frames
while True:
try:
raw_frame = frame_queue.get(timeout=1) # Get raw bytes from the queue
np_frame = np.frombuffer(raw_frame, dtype=np.uint8) # Convert bytes to NumPy array
np_frame = np_frame.reshape((height, width, 3)) # Reshape to (H, W, 3)
cv2.imshow("Frame", np_frame)
if cv2.waitKey(1) & 0xFF == ord('q'): # Press 'q' to exit
break
except queue.Empty:
continue # Skip if no frame is available
except Exception as e:
print(f"Error in frame processing: {e}")
# Cleanup
picam2.stop_recording()
cv2.destroyAllWindows()
picam2.stop()
Expected behaviour Expect no empty frames
Console Output, Screenshots
[22:04:20.455549985] [7052] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+53-29156679
[22:04:20.529303103] [7058] WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[22:04:20.534684441] [7058] INFO RPI vc4.cpp:447 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0
[22:04:20.534918399] [7058] INFO RPI pipeline_base.cpp:1121 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[22:04:20.552437410] [7052] INFO Camera camera.cpp:1202 configuring streams: (0) 320x240-RGB888 (1) 640x480-SGBRG10_CSI2P
[22:04:20.553181783] [7058] INFO RPI vc4.cpp:622 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
got frame of len: 230400
pushing frame
here?
got frame of len: 230400
pushing frame
got frame of len: 230400
got frame of len: 230400
got frame of len: 230400
got frame of len: 230400
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
^Cempty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
^Cempty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
^Cempty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
empty frame
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
empty frame
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
^Cempty frame
empty frame
Hardware : RaspberryPi 3
Additional context
Maybe I'm just doing something incorrectly. I know I can use .capture_array and the performance is decent. But from my experience with the original picamera library using start_recording was significantly faster.
Hi, indeed a good little puzzle. The problem, I think, is that the frames in your output are mmap objects that come directly from the camera system where they get written to by hardware blocks. The mmaps, however, are persistent and there's only a limited number of them that get re-used over and over.
So what happens is that you read out all the data, but when that buffer comes back you've already read everything, so there's now zero bytes left, even though the part of the buffer that you read previously has been overwritten. The easiest workaround in your case would be to add frame.seek(0) just before raw_data = frame.read().
There's clearly an argument that something else in the system should be resetting you back to the start of the buffer before handing them to you, so I'll have a look at that. I guess you must be the first person to try this sort of thing, normally buffers just get turned into numpy arrays, which doesn't seem to have the same problem. Oh well. Thanks for reporting the problem!
Oh interesting! thanks for the tip.
I actually developed (Picameleon)[https://github.com/Esser50K/PiCameleon] in the past to make it easier for multiple processes to use the camera at the same time while also kinda turning raspberrypi into a capable IP camera.
I'm very happy to see that some of the concepts I thought of then are being used here (like having a list of outputs for start_recording).
Will probably rewrite it to use picamera2 and it should be a lot less work this time around.