raspberrypi
[HOW-TO] Use Hardware JPEG encoder
Describe what it is that you want to accomplish I want to use the hardware JPEG encoder to encode single images (in contrast to MJPEG) which is used in the former picamera1 stack.
Describe alternatives you've considered I thought of using the MJPEG encoder which claims to use the hardware encoders, but i want to use it for single frames instead of video frames.
Additional context
Unfortunately we haven't written code that uses the MJPEG encoder for still images. With things like bitrates and framerates it's not an ideal match. That said, would a workaround like this be any use?
import io
import threading
from picamera2 import Picamera2
from picamera2.encoders import MJPEGEncoder
from picamera2.outputs import FileOutput
picam2 = Picamera2()
picam2.start()
main_stream = picam2.stream_map["main"]
class MyOutput(io.BufferedIOBase):
def __init__(self, filename):
self.filename = filename
self.event = threading.Event()
def write(self, buf):
with open(self.filename, 'wb') as f:
f.write(buf)
self.event.set()
def wait_for_frame(self):
self.event.wait()
self.event.clear()
mjpeg_encoder = MJPEGEncoder()
mjpeg_encoder.framerate = 30
mjpeg_encoder.size = config["main"]["size"]
mjpeg_encoder.format = config["main"]["format"]
mjpeg_encoder.bitrate = 5000000
my_output = MyOutput("out.jpg")
mjpeg_encoder.output = FileOutput(my_output)
mjpeg_encoder.start()
request = picam2.capture_request()
mjpeg_encoder.encode(main_stream, request)
request.release()
my_output.wait_for_frame()
It's a little tricky because the video encoders run asynchronously, so we have to find a way to get a signal from them when it's all finished. Though once it's set up, you should be able just to repeat those last 4 lines every time you want to do a capture.
i am interested in using this
can anyone explain how i would decide on a bit rate
i want to use the lores stream at 820 by 486
i am currenlty using opencv to encode as jpeg and it is too slow, i am using this on a robot that sends jpeg over network for proccesing on desktop using opencv
here is what i am doing at the moment.
import cv2
from picamera2 import Picamera2
import time
import pprint
picam2 = Picamera2()
print("sensor_modes _________________________________")
pprint.pp(picam2.sensor_modes)
mainW = 1640
mainH = 1232
picam2.preview_configuration.main.size = (mainW, mainH)
picam2.preview_configuration.main.format = "RGB888"
picam2.preview_configuration.controls.FrameRate = 10
loresW = 820
loresH = 486
picam2.preview_configuration.enable_lores()
picam2.preview_configuration.lores.size = (loresW, loresH)
picam2.preview_configuration.align()
picam2.configure("preview")
picam2.start()
print("__________________________________________________________\n")
pprint.pp(picam2.camera_properties)
print("__________________________________________________________\n")
picam2.controls.AeExposureMode = 1
fps = 0
pos = (30, 60)
font = cv2.FONT_HERSHEY_SIMPLEX
height = 1.5
weight = 3
myColor = (0, 0, 255)
cal_image_count = 0
frame_count = 1
counter = 0
while True:
tStart = time.time()
im = picam2.capture_array('lores')
im = cv2.cvtColor(im, cv2.COLOR_YUV420p2RGB)
frame_count += 1
if frame_count % 30 == 0:
counter += 1
if counter == 2:
cal_image_count += 1
print("saving image")
cv2.imwrite("cal_image_" + str(cal_image_count) + ".jpg", im)
cal_image_count += 1
counter = 0
cv2.putText(
im,
str(int(fps)) + " FPS, counter:" + str(counter),
pos,
font,
height,
myColor,
weight,
)
cv2.imshow("Camera", im)
if cv2.waitKey(1) == ord("q"):
break
tEnd = time.time()
loopTime = tEnd - tStart
fps = 0.9 * fps + 0.1 * (1 / loopTime)
cv2.destroyAllWindows()
Hi, and thanks for the question. I think a lot of the answers depend on how you want to send the JPEG over the network, how it's going to be used and displayed, and so on. So what follows may seem like a slightly random list of suggestions, but maybe some of them will be useful.
- To be honest, the best suggestion will be to set everything up as a regular video stream. Probaby use a PyavOutput or a custom output that streams directly to the network. For instance, this example will achieve 30fps easily. A PyavOutput will stream 1080p30 H.264 without batting an eyelid.
- I think the MJPEGEncoder will accept YUV420 directly, so that would probably be more efficient.
- As regards bitrate and framerate, set the framerate to any number you like. If the image quality is poor, double the bitrate. If the image looks really good, maybe reduce the bitrate. I think the only purpose of the framerate is to normalise the bitrate, I don't think the codec measures the true framerate, it just believes what you tell it!
- Obviously writing to files is really bad. Hopefully you can avoid it completely, but if not, at least write to a file in /dev/shm.
- If you're using OpenCV as the receiver, it's quite likely to support MPEG2 Transport Streams and RTSP, so I'd be tempted to give one of those a try. A PyavOutput would generate an MPEG2-TS directly, though depending on exactly what you want to do, I've found MediaMTX to be a very effective streaming solution, either by itself or in conjunction with Picamera2.
oops i posted the wrong code, just noticed what i posted writes to a file, i will correct it with correct script. the publisher that sends the image uses zmq under the hood.
thanks for your answer thats a lot for me to look into.
using opencv to work with mjpeg streams is an interesting idea, i had not realised that might be an option.
i was planning on using a file like object with the example above. my hope is that with the example above it will use the native jpeg encoding and speed things up.
i dont have note of how long the jpeg encoding above took but i will measure it and post it here
what i can say is that at 1640 by 1232 it can only do 7 fps. which i think is pretty slow. i have tried using simplejpeg but it doesnt speed things up its not that i need this higher resolution its all i can remember the speed for of the top of my head
the problem for me that it is not just the fps i need to worry about, i need to do computer vision in between the frames and encoding as jpeg is eating in to the time i have to do it. at the moment i run the computer vision algos on the desktop and it works fine. but eventually i will need to run some algos on the pi and will still need to stream to desktop. So its not so much that i need to find a solution for my current use, more that i know i am wasting proccessing time that doesnt need to be wasted and that i have a project with continously increasing demands.
this means i need the image as a numpy array on the pi, and an mjpeg stream for the network, do you know if the above example would give me access to the numpy array on the pi. or would i need to decode it thus slowing things down again?
one thing i had done previously was to use the command line utility from libcamera by running it as a process from nodejs, (nodejs is super fast for sending stuff over the network) and streaming it from there. i then just decoded it manually looking for the end bytes of each jpeg. i never noticed any speed problems when doing this but didnt measure it, so i cant be sure. this still leaves me with the problem of not having numpy arrays on the pi.
thanks, again for your reply it is very helpfull.
The example I gave above doesn't give you both the numpy array and the JPEG at the same time. You could obviously ask for numpy arrays in the main thread, but they wouldn't necessarily be synchronised (particularly if you're dropping frames).
Another thing to try might be to follow this example. Strip out the H.264 related stuff (don't forget to start the camera when you remove start_recording) and you're left with a single main loop that can give you numpy arrays and pass them to the MJPEGEncoder. The MJPEGEncoder guarantees to output every frame you give it, in order, so you can map them accurately to any numpy arrays (if you want to).
I think it will still probably make sense to choose your "main" and "lores" streams carefully, so that the Pi isn't processing more pixels than it has to. On non-Pi 5s, of course, the "lores" stream has to be YUV420.
thats great thanks, i think that is what i was looking for, I will try it out.