OctoPi-UpToDate Feedback for the latest camera-streamer based webcam stack builds (builds 2023.07.20.144556, 2023.10.09.154319)

This is a ticket to collect feedback on the latest build of the new camera stack (some background on that here) for OctoPi-UpToDate.

Currently, that is 2023.10.09.154319. Find the image here: https://github.com/OctoPrint/OctoPi-UpToDate/releases/tag/1.0.0-1.9.3-20231009154319

Config docs are available here: https://faq.octoprint.org/camera-streamer-config

The source tree used for the build is here: https://github.com/OctoPrint/OctoPi-UpToDate/tree/camera-streamer

Changelog

Changes in build 2023.10.09.154319 from 2023-10-09

OctoPrint updated to 1.9.3

Changes in build 2023.07.20.144556 from 2023-07-20

camera-streamer updated to 0.2.5
- Heads-up: camera-streamer 0.2.5 added a new parameter --http-listen to define which network interfaces to listen on, and by default only listens on localhost. If you have cameras configured that are currently not behind haproxy on the same device, you'll need to add --http-listen=0.0.0.0 to the OPTIONS of your camera config. The docs have been updated accordingly.
user-fix fixed #8

❗ Please note

This image needs to be flashed to your SD card, wiping it in the process. So if you want to keep anything on there, make a backup first! You can also create an OctoPrint backup, download that, flash, restore from backup. There's no way to update to the contents of this image without a full reflash.

Please provide feedback on your experience with this image in this ticket, whether things work for you or not. Make sure to at least include the following:

Everything working?
Pi model
used cameras
in case of issues: related logs (journalctl -u camera-streamer\* | pb and share the generated paste.octoprint.org URL)

Known issues:

ayufan/camera-streamer#94 (this is currently considered a blocker for general roll-out of this camera stack, as it causes the camera stream to crash)

Common pitfalls:

RPiV3 camera not auto-focusing? Make sure any kind of case you've added to it is not interfering with the lens & sensor

Jul 20 '23 15:07 foosel

Just out of curiosity (while I download and install the new image), I thought there was a package being created for camera-streamer that could be installed/updated with apt so that libcamera updates didn't require a new image. What happened to that project?

Jul 20 '23 15:07 b-morgan

There is. This new image was needed because there was an issue in the basic image I built two days ago. The update-able package is alive and well (and used here), but since this is the first image to ship with the 0.2.5 version out of the box, I figured this merits a new feedback ticket.

Jul 20 '23 16:07 foosel

Hi Gina,

Thanks for the blazing fast response. I have good and possibly bad news.

Good News: camera service is still there after doing a full upgrade
Bad news: cameras services are now bound to localhost

All I did was change ports of libcamera and usb camera to 8081 and 8082 respectively and rebooted. Journal shows no errors and services listening in respective ports. However, netstat shows the following which explains why i cannot connect from another computer but only from rpi itself.

pi@octopi2:~ $ netstat -a --numeric-ports | grep LIST
tcp        0      0 localhost:8082          0.0.0.0:*               LISTEN
tcp        0      0 localhost:8081          0.0.0.0:*               LISTEN
tcp        0      0 localhost:5000          0.0.0.0:*               LISTEN

Snippets from journal below:

-- Boot 3e0689d38a4f4aef9df1285433eaa6c3 --
Jul 20 20:14:02 octopi2 systemd[1]: Starting camera-streamer default...
Jul 20 20:14:02 octopi2 systemd[1]: Started camera-streamer default.
Jul 20 20:14:06 octopi2 sh[686]: util/http/http.c: ?: HTTP listening on 127.0.0.1:8082.
Jul 20 20:14:06 octopi2 sh[686]: device/v4l2/device.c: CAMERA: Device path=/dev/v4l/by-id/usb-SHENZHEN_AONI_ELECTRONIC_CO.__LTD_Full_HD_webcam_20200729001-video-index0 fd=5 opened
Jul 20 20:14:06 octopi2 sh[686]: device/v4l2/device_options.c: CAMERA: The 'horizontal_flip=0' was failed to find.

-- Boot 3e0689d38a4f4aef9df1285433eaa6c3 --
Jul 20 20:13:57 octopi2 systemd[1]: Starting camera-streamer libcamera...
Jul 20 20:14:01 octopi2 sh[604]: /base/soc/i2c0mux/i2c@1/imx708@1a
Jul 20 20:14:02 octopi2 systemd[1]: Started camera-streamer libcamera.
Jul 20 20:14:06 octopi2 sh[669]: util/http/http.c: ?: HTTP listening on 127.0.0.1:8081.
Jul 20 20:14:06 octopi2 sh[669]: /usr/bin/camera-streamer Version: 0.2.5 (41d8dfd)
Jul 20 20:14:06 octopi2 sh[669]: [0:00:18.170190026] [669]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+82-2783c8d8
Jul 20 20:14:06 octopi2 sh[669]: [0:00:18.363628174] [855]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media3 and ISP device /dev/me>

I will keep looking around to see if I missed something ....

Danke, Gaston

Jul 21 '23 03:07 gdombiak

I modified the systemd configuration file to add '--http-listen=0.0.0.0 ' and restarted the service. Cameras are now accessible from other computers since service bound to all interfaces. Not sure if this is the proper fix but I'm now unblocked. :D

Thanks, Gaston

Jul 21 '23 03:07 gdombiak

I put a heads-up into the changes list up in the very first post for that exact reason ;)

Jul 21 '23 04:07 foosel

LOL. RTFM for me then! :D Impatience got the "best" of me.

All seems to be working fine. Will keep testing and report if I came across anything else. I doubt it but you never know.

Thanks, Gaston

Jul 21 '23 04:07 gdombiak

Sems like the camera-streamer is missing in my case.

https://paste.octoprint.org/Scyq0nVuqz

Jul 21 '23 21:07 MrUpStry

Danke Gina for the very swift update. The update works for me on my RPi4B with Camera Module 3 noIR

Jul 21 '23 23:07 Bastillion1480

All the versions never worked for me with a Raspberry Pi Zero 2 W but I just tried same backup with a Pi 4B+ and it worked

Jul 23 '23 10:07 nicopasla

Can I update my way out of the this or do I have to reinstall octopi from the beginning?

The following packages have unmet dependencies:
 camera-streamer-raspi : Depends: libcamera0 (= 0~git20230707+2783c8d8-1) but 0~git20230720+bde9b04f-1 is to be installed
E: Unable to correct problems, you have held broken packages.

Jul 23 '23 15:07 oxivanisher

A new camera-streamer-raspi package should be available, try installing that.

Side note, hard pinning the libcamera0 version like this is sadly necessary as libcamera0 (currently?) happily changes its ABI on version updates, pulling the rug from underneath camera-streamer's feet and making a recompilation of the latter necessary to still work. It just has happened again with this update.

Jul 24 '23 08:07 foosel

All the versions never worked for me with a Raspberry Pi Zero 2 W

Define "never worked". As it is, that comment sadly is utterly unhelpful.

Jul 24 '23 08:07 foosel

A new camera-streamer-raspi package should be available, try installing that.

I don't see it:

prusamk3mmu ~ # apt update
Hit:1 https://apt.octoprint.org/debian bullseye InRelease
Hit:2 http://raspbian.raspberrypi.org/raspbian bullseye InRelease
Hit:3 http://archive.raspberrypi.org/debian bullseye InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
prusamk3mmu ~ # apt remove camera-streamer*
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'camera-streamer' for glob 'camera-streamer*'
Note, selecting 'camera-streamer-raspi' for glob 'camera-streamer*'
Note, selecting 'camera-streamer-stack' for glob 'camera-streamer*'
Package 'camera-streamer-raspi' is not installed, so not removed
Package 'camera-streamer-stack' is not installed, so not removed
Package 'camera-streamer' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
prusamk3mmu ~ # apt install camera-streamer-raspi
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 camera-streamer-raspi : Depends: libcamera0 (= 0~git20230707+2783c8d8-1) but 0~git20230720+bde9b04f-1 is to be installed
E: Unable to correct problems, you have held broken packages.

Jul 24 '23 08:07 oxivanisher

Ooooh, libcamera apparently pulled the rug from beneath our feet YET AGAIN. FFS... on it.

Jul 24 '23 10:07 foosel

New camera-streamer-raspi package now built and pushed, and also another day spent on improving automating things so that stuff like this in the future happens semi-automatically (final PR for package addition ton the repo needs to be merged manually) on every libcamera0 update detected on the RPi repo.

Jul 24 '23 14:07 foosel

Thank you very much for your effort! A simple apt install camera-streamer-raspi was enough to get it working again. Now I can again watch live how the MK4 runs out of filament and breaks my prints. 😒😂

Jul 24 '23 15:07 oxivanisher

me with a Raspberry Pi Zero 2 W Define "never worked". As

sorry I forgot to say that the camera stream never loaded on the PiZero2W, I had a lot of random reboot too

When using libcamera-hello, there was an error about the ressource already used

Jul 24 '23 17:07 nicopasla

I test against RPi3b, which is pretty much the same CPU, albeit in a different form factor. Not sure what you are running into there, everything green over here and I can also see at least 81 instances on an RPiZero2 running one of the images with the new camera build in the anonymous usage tracking data.

Random reboot sounds like insufficient power or some other hardware issue. libcamera-hello not being able to access the camera when the stream is already running is to be expected.

Jul 24 '23 17:07 foosel

Hi, no luck with fresh install ( Version 1.9.2 ) an AZDeliver Raspberry Cam v1.3.

libcamera --list-cams

Available cameras
-----------------
0 : ov5647 [2592x1944] (/base/soc/i2c0mux/i2c@1/ov5647@36)
    Modes: 'SGBRG10_CSI2P' : 640x480 [58.92 fps - (16, 0)/2560x1920 crop]
                             1296x972 [43.25 fps - (0, 0)/2592x1944 crop]
                             1920x1080 [30.62 fps - (348, 434)/1928x1080 crop]
                             2592x1944 [15.63 fps - (0, 0)/2592x1944 crop]

system log

Jul 26 14:57:37 octi sh[1406]: [0:22:10.182179708] [1424] #033[1;33m WARN #033[1;37mV4L2 #033[1;34mv4l2_videodevice.cpp:2007 #033[0;32m/dev/video0[23:cap]: #033[0mDequeue timer of 1000000.00us has expired!$
Jul 26 14:57:37 octi sh[1406]: [0:22:10.182389549] [1424] #033[1;31mERROR #033[1;37mRPI #033[1;34mpipeline_base.cpp:1333 #033[0mCamera frontend has timed out!$
Jul 26 14:57:37 octi sh[1406]: [0:22:10.182459132] [1424] #033[1;31mERROR #033[1;37mRPI #033[1;34mpipeline_base.cpp:1334 #033[0mPlease check that your camera sensor connector is attached securely.$
Jul 26 14:57:37 octi sh[1406]: [0:22:10.182524964] [1424] #033[1;31mERROR #033[1;37mRPI #033[1;34mpipeline_base.cpp:1335 #033[0mAlternatively, try another cable and/or sensor.

libcamera-hello

Made DRM preview window
[0:52:21.555484026] [1445]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[0:52:21.597288004] [1446]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media0 and ISP device /dev/media2
[0:52:21.597395555] [1446]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[0:52:21.598568671] [1445]  INFO Camera camera.cpp:1033 configuring streams: (0) 1296x972-YUV420
[0:52:21.599241269] [1446]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 1296x972-SGBRG10_1X10 - Selected unicam format: 1296x972-pGAA
[0:52:21.801866263] [1446]  INFO V4L2 v4l2_videodevice.cpp:1820 /dev/video0[13:cap]: Zero sequence expected for first frame (got 2)
[0:52:22.802831335] [1446]  WARN V4L2 v4l2_videodevice.cpp:2007 /dev/video0[13:cap]: Dequeue timer of 1000000.00us has expired!
[0:52:22.803041802] [1446] ERROR RPI pipeline_base.cpp:1333 Camera frontend has timed out!
[0:52:22.803118833] [1446] ERROR RPI pipeline_base.cpp:1334 Please check that your camera sensor connector is attached securely.
[0:52:22.803199613] [1446] ERROR RPI pipeline_base.cpp:1335 Alternatively, try another cable and/or sensor.

But the Camera works with stable Octopi release.

Any hints?

Jul 26 '23 12:07 djusHa

I noticed a small problem with camera-streamer. When I boot up my pi and access octoprint web interface at least once (so it loads the camera-streamer stream) everything works perfectly fine. (And keeps working, no matter for how long I'm not accessing the camera-streamer stream)

But when I boot up and not open the web interface for some time (it's reproduceable 100% of the time when I wait at least 60min.) and then load the web interface, I don't get any stream from camera-streamer. Opening "/webcam/stream" shows the text "Server Error". Opening "/webcam/" works, "/webcam/status" shows the expected json and "/webcam/control" shows the control interface.

"systemctl status camera-streamer" reports the service as active.

Logs: https://paste.octoprint.org/OYXd3AsGW5 (The entries after "Jul 28 23:56:32" are the ones when it fails) Cam: 'ov5647' on port 0 System: Raspberry Pi 4 Model B Rev 1.1 - 11 (bullseye) - Linux octopi 6.1.21-v7l+ #1642 SMP Mon Apr 3 17:22:30 BST 2023 armv7l GNU/Linux

This also happened before the recent libcamera0 & camera-streamer-raspi updates.

Restarting the camera-streamer service fixes the problem. ("systemctl restart camera-streamer")

Jul 28 '23 22:07 sarusani

@sarusani since that looks like it's rather an internal issue with camera-streamer than something related to the image including it, could you please report this on the camera-streamer repo?

https://github.com/ayufan/camera-streamer

Jul 29 '23 09:07 foosel

Sure, I'll do. 👍 Wasn't sure if it's camera streamer or a problem with the service.

https://github.com/ayufan/camera-streamer/issues/90

Jul 29 '23 10:07 sarusani

I noticed a small problem with camera-streamer. When I boot up my pi and access octoprint web interface at least once (so it loads the camera-streamer stream) everything works perfectly fine. (And keeps working, no matter for how long I'm not accessing the camera-streamer stream)

But when I boot up and not open the web interface for some time (it's reproduceable 100% of the time when I wait at least 60min.) and then load the web interface, I don't get any stream from camera-streamer. Opening "/webcam/stream" shows the text "Server Error". Opening "/webcam/" works, "/webcam/status" shows the expected json and "/webcam/control" shows the control interface.

"systemctl status camera-streamer" reports the service as active.

Logs: paste.octoprint.org/OYXd3AsGW5 (The entries after "Jul 28 23:56:32" are the ones when it fails) Cam: 'ov5647' on port 0 System: Raspberry Pi 4 Model B Rev 1.1 - 11 (bullseye) - Linux octopi 6.1.21-v7l+ #1642 SMP Mon Apr 3 17:22:30 BST 2023 armv7l GNU/Linux

This also happened before the recent libcamera0 & camera-streamer-raspi updates.

Restarting the camera-streamer service fixes the problem. ("systemctl restart camera-streamer")

Perhaps it's related to my issue at ayufan/camera-streamer#88?

Jul 29 '23 21:07 FacuM

did you consider switching to the actual video stream? Camera-streamer has both /video and /stream: /video is an h.264-stream, so much more efficient that the /stream, which is MJPEG (like the old stack). Octopi currently continues to use the mjpeg-stream, while it could have the much more efficient h.264 stream for nearly free. something has to be changed in the html for that, it's not quite as simple as replacing the action=stream with video in the url.

Jul 30 '23 08:07 gnurbs

did you consider switching to the actual video stream? Camera-streamer has both /video and /stream: /video is an h.264-stream, so much more efficient that the /stream, which is MJPEG (like the old stack). Octopi currently continues to use the mjpeg-stream, while it could have the much more efficient h.264 stream for nearly free. something has to be changed in the html for that, it's not quite as simple as replacing the action=stream with video in the url.

That's the whole point of this new stack. It's no longer using the MJPEG stream (only as a fallback). It's using the way more efficient WebRTC stream. 😃 (You can look at WebRTC as a browser friendly way to transport the h264)

Jul 30 '23 09:07 sarusani

This image doesn't view the h264 stream out of the box, it will still use the mjpg stream. I created https://github.com/cp2004/OctoPrint-CameraStreamer-Control to view those streams. It's incomplete, but it works and has a reasonable number of users despite not being published yet...

Jul 30 '23 13:07 cp2004

@sarusani, The command "systemctl restart camera-streamer" needs privilege so it should be "sudo systemctl restart camera-streamer". Unfortunately, that doesn't work very well if added to the OctoPrint UI (through config.yaml). Using "sudo service camera-streamer restart" works better (thanks @jneilliii).

Aug 09 '23 19:08 b-morgan

@sarusani, The command "systemctl restart camera-streamer" needs privilege so it should be "sudo systemctl restart camera-streamer". Unfortunately, that doesn't work very well if added to the OctoPrint UI (through config.yaml). Using "sudo service camera-streamer restart" works better (thanks @jneilliii).

The service command is very old and it would be better (and more future proof!) to use systemctl (aka systemd). I fixed this very easy for myself by adding the following sudo config which allows the user pi to run explicatively those two commands:

# cat /etc/sudoers.d/camera-streamer-control
pi ALL=NOPASSWD: /bin/systemctl start camera-streamer.service
pi ALL=NOPASSWD: /bin/systemctl stop camera-streamer.service

To ensure that you don't break the sudo config if you are not sure about it, you can verify your config like this:

# /usr/sbin/visudo -cf /etc/sudoers.d/camera-streamer-control
/etc/sudoers.d/camera-streamer-control: parsed OK

Aug 09 '23 19:08 oxivanisher

I'm actually using this in config.yaml and it works perfectly fine (pi user is in the passwordless sudo list):

system:
  actions:
  - action: restartwebcam
    command: sudo systemctl restart camera-streamer
    confirm: false
    name: Restart Webcam

But it still would be better if the service wouldn't break in the first place 😉

Aug 09 '23 19:08 sarusani

I'm actually using this in config.yaml and it works perfectly fine (pi user is in the passwordless sudo list):
system:
  actions:
  - action: restartwebcam
    command: sudo systemctl restart camera-streamer
    confirm: false
    name: Restart Webcam
But it still would be better if the service wouldn't break in the first place 😉

Absolutely! I have another use case. Since I sometimes print orders, I stream the feed (trough a mjpeg-proxy container!) to the internet. But I only want the feed running during prints, since uploading two 720p video streams (I use two printers) 24/7 is REALLY BAD for my upload. 😂 So my config for this use case looks like this:

events:
  enabled: true
  subscriptions:
  - command: sleep 30; sudo /bin/systemctl stop camera-streamer.service
    debug: false
    enabled: true
    event: PrintCancelled
    name: sleep 30; sudo /bin/systemctl stop camera-streamer.service
    type: system
  - command: sleep 30; sudo /bin/systemctl stop camera-streamer.service
    debug: false
    enabled: true
    event: PrintDone
    name: sleep 30; sudo /bin/systemctl stop camera-streamer.service
    type: system
  - command: sleep 30; sudo /bin/systemctl stop camera-streamer.service
    debug: false
    enabled: true
    event: PrintFailed
    name: sleep 30; sudo /bin/systemctl stop camera-streamer.service
    type: system
  - command: sudo /bin/systemctl start camera-streamer.service
    debug: false
    enabled: true
    event: PrintStarted
    name: sudo /bin/systemctl start camera-streamer.service
    type: system

But all this has nothing to do with this issue. 😉

Aug 09 '23 19:08 oxivanisher