Document using cgroup rules to start container without USB

[LongLiveCHIEF]

from https://discordapp.com/channels/704958479194128507/753995646218010674/792045233600856066

I have this problem where I can't restart the octoprint docker if my printer is not on. Anyone else struggled with this? In a nutshell: If my printer is OFF, the /dev/ttyACM0 is not available > octoprint container fails to mount it during startup > octoprint container fails to start. I currently have to turn the printer ON via smart socket and only then start/restart the octoprint container. It's fairly small annoyance, but I'd prefer octoprint to always be on and printer itself to be controlled via smart socket. Is this possible?

Powering off the printer octoprint controls during idle time is a common requirement of OctoPrint users. Docker will fail to start if you try to pass it a usb serial device if that device does not yet exist, and in addition, the device could be assigned a different binding depending on what usb port its plugged into or what order that usb device is booted.

The solution here will be to create a custom cgroup-rule, mount that, and create a udev rule which adds the device to the container when it is added. see: Dealing with dynamically created devices (--device-cgroup-rule)

The issue here of course is that the user may not be using a static name for the octoprint container, so this may only work in circumstances where you're using docker run or docker-compose with an container_name

[LongLiveCHIEF]

The real challenge here is not adding the cgroup rule, but the fact that you have to know the state of the desired device when the container starts. If it is connected, you need to include the device in the container startup. If it isn't, then having added the udev rule will cause the device to be added when it is detected.

This is just as bad as trying to start a container without the device, because it means you have to know the state of the printer prior to container startup regardless of if you have these device rules in place.

You could easily create a systemd unit file that uses a script to start the octoprint container, and that script checks for a device and includes it in container startup if it exists, but you've now once again tied the system to the container.

[LongLiveCHIEF]

Basically, we could support systemd type systems, and provide that script, and then tell anyone using portainer or any other tool that manages containers that they would be on their own to figure out how to do a dynamic container start?

[reverendj1]

I came across this Reddit post, https://www.reddit.com/r/octoprint/comments/klxzpr/docker_mapping_printer_to_usb_port/ and with your and wingjames' help was able to successfully set it up so that I can start Octoprint with the printer turned off. I was trying to think of a more elegant solution like you want and came across this, https://github.com/marthoc/docker-deconz/issues/298, where it seems that someone else is doing what you want. I couldn't quite figure out all the pieces to put it together or find in the code where they were setting up the devices or anything. Maybe it will point you in the right direction.

[Paraphraser]

@reverendj1 perhaps see this gist. It’s a brute force solution but it does actually work. I’m running OctoPrint in a Docker container on a Pi 3B+. I’ve also tested the approach on a Pi 4 with other detachable devices like a Zigbee adapter.

[LongLiveCHIEF]

I've already got the work for all this done, (it's a really simple single-line script for udev rules), just haven't had time to finish the PR to share docs.

[palsch]

Hello all, I had the same issue with my printer. Usually it is powered off. If I tried to restart my raspi or the octoprint container, it failed because of the missing usb device. So my current solution is also to use cgroup and mknod. I modified the Dockerfile and the docker-compose file with the following lines (my printer gets always ttyUSB0):

Dockerfile (take original Dockerfile and add following line before the ENTRYPOINT):

This command create a new script in the /etc/cont-init.d folder, that is executed automatically on container start (see also s6 overlay). The command mknod /dev/ttyUSB0 c 188 0 creates a new device file that can be used by octoprint to connect to the printer as soon as the printer is available. That means, the device file can be created while the printer is off.

....
RUN echo "#!/usr/bin/execlineb -P\nmknod /dev/ttyUSB0 c 188 0" >> /etc/cont-init.d/init-usb-device
ENTRYPOINT ["/init"]

Build the new docker image: docker build . -t octoprint-custom

docker-compose.yml (add following configuration changes to your existing file):

services:
  octoprint:
    image: octoprint-custom
    devices:
      - /dev/bus/usb # map the complete usb device bus
    volumes:
      - /run/udev:/run/udev:ro  # so that serial number vendor/product ids are known
    device_cgroup_rules:
      - 'c 188:* rmw' # allow creation of /dev/ttyUSBx nodes via mknod

[squirrel289]

I've already got the work for all this done, (it's a really simple single-line script for udev rules), just haven't had time to finish the PR to share docs.

Any chance you could share this one liner here so other less adept folks (like me) can use while you’re assembling the PR?

[LongLiveCHIEF]

https://github.com/OctoPrint/octoprint-docker/blob/udev-rules-for-dynamic-usb-serial-support/71-octoprint.docker.rules

[glennpierce]

https://github.com/OctoPrint/octoprint-docker/blob/udev-rules-for-dynamic-usb-serial-support/71-octoprint.docker.rules

Anything else needed other that adding this file to say /usr/lib/udev/rules.d and rebooting. When starting the container I get the same error about missing ttyACM0

[jjaeggli]

An alternative solution I'm using to this script above is to create a wrapper init script in conjunction to the cgroup rules. There's nothing which says the character devices need to be added to the filesystem dynamically, given that the printer is the only serial device connected to this system. I've placed the script in the octoprint volume and it looks something like this:

#!/bin/sh

mknod c 166 0 /dev/ttyACM0
mknod c 166 1 /dev/ttyACM1

/init $@

On my system the Rambo board uses device major 166. I then execute the script using docker flag --entrypoint=/octoprint/init_usb.sh. This means that no changes are required to the existing image, and I can upgrade without any further time commitment.

[bambuleee]

https://github.com/OctoPrint/octoprint-docker/blob/udev-rules-for-dynamic-usb-serial-support/71-octoprint.docker.rules

Anything else needed other that adding this file to say /usr/lib/udev/rules.d and rebooting. When starting the container I get the same error about missing ttyACM0

Hi, I tried a lot and finally got it working. You have to change the name of the dockercontainer (here octoprint-container) and the major and minor parameter (here 188 and 0-> ls /dev/ttyUSBDEVICE -l)

/etc/udev/rules.d/90-octoprint.rules

SUBSYSTEM!="tty", GOTO="end_octoprint_printers"
ACTION=="add|change", SUBSYSTEM=="tty", KERNEL=="ttyUSB[0-9]|ttyACM[0-9]", RUN+="/bin/bash /home/pi/drucker.sh"
ACTION=="remove", SUBSYSTEM=="tty", KERNEL=="ttyUSB[0-9]|ttyACM[0-9]", RUN+="/usr/bin/docker exec octoprint-container rm -rf /dev/ttyUSB0"
LABEL="end_octoprint_printers"

Script drucker.sh (chmod +x):

#!/bin/bash
minor=0
/usr/bin/docker exec octoprint-container rm -rf /dev/ttyUSB0
/usr/bin/docker exec octoprint-container mknod /dev/ttyUSB0 c 188 $minor

compose-file for octoprint

device_cgroup_rules:
  - 'c 188:* rmw'

[matt-laird]

I experienced some real pain and suffering trying to get this to work, but got there in the end.

• As @bambuleee mentioned in their post, the 71-octoprint.docker.rules should rather make a call to a script which executes the commands in the container, on an add|change event because I experienced problems with the RUN command that included both the removal and (&&) creation of the device in the container.

Notes for newbies:

Just a note for anyone finding this thread, here are a few unspoken things that are important, taking into account you have read the whole issue thread and are now here:

• The udev rules file needs to be saved on the host of your octoprint container.
• The number in the name of the udev rule does matter, just leave it as 71 you should be fine.
• Once the rules file is in place, reboot your OS for these to rules to take effect.
• Use the device_cgroup_rules directive in your compose to make sure the container can handle no solid references devices e.g. /dev/ttyACM0:/dev/ttyACM0

[piter-pit]

I struggle with this to work but without success. points I did:

1. added rule /etc/udev/rules.d/71-octoprint.rules and then tried also /etc/udev/rules.d/90-octoprint.rules with proper container name SUBSYSTEM!="tty", GOTO="end_octoprint_printers" ACTION=="add|change", SUBSYSTEM=="tty", KERNEL=="ttyUSB[0-9]|ttyACM[0-9]", RUN+="/bin/bash /home/pi/drucker.sh" ACTION=="remove", SUBSYSTEM=="tty", KERNEL=="ttyUSB[0-9]|ttyACM[0-9]", RUN+="/usr/bin/docker exec octoprint-container rm -rf /dev/ttyUSB0" LABEL="end_octoprint_printers"
2. created script /home/pi/drucker.sh with proper container name and chmoded it with +x -rwxr-xr-x 1 root root 158 Jun 20 13:57 drucker.sh
3. in docker-compose updated device_cgroup_rules: - 'c 188:* rmw' #devices: hashed out #- /dev/ttyUSB0:/dev/ttyUSB0 checked also without hashes but not starting without printer powered up

effects:

• after restart and with unpowered printer octoprint starts with "no serial port found" and ls /dev/ttyUSB0 -l reports cannot access '/dev/ttyUSB0': No such file or directory - ok...
• after power up printer and refresh /dev/ttyUSB0 is detected and ls /dev/ttyUSB0 -l reports crw-rw---- 1 root dialout 188, 0 Jun 20 16:05 /dev/ttyUSB0 - hurray
• when trying to connect it "Could not autodetect your printer No working connection parameters could be found." - not hurray
• set up /dev/ttyUSB0 and baudrate on 115200 not helps - "opening serial communication" displays forever

[LongLiveCHIEF]

are you sure your device uses ttyUSB and not ttyACM? Also, are you sure your printer is on USB0 and not USB1 or another enumerated USB device port?

The reason i haven't merged any of this documentation yet, is because there are generally no "universal" rules and instructions that work for everyone. There's not even a good set that works for the majority.

This is part of the container way of life unfortunately.

[piter-pit]

yes, it was always ttyUSB0, with devices:

• /dev/ttyUSB0:/dev/ttyUSB0

in addition to that after I connect printer: [ 324.035212] usb 1-1.4: new full-speed USB device number 3 using xhci_hcd [ 324.171512] usb 1-1.4: New USB device found, idVendor=1a86, idProduct=7523, bcdDevice= 2.63 [ 324.171543] usb 1-1.4: New USB device strings: Mfr=0, Product=2, SerialNumber=0 [ 324.171564] usb 1-1.4: Product: USB2.0-Serial [ 324.273915] usbserial: USB Serial support registered for generic [ 324.277125] usbserial: USB Serial support registered for ch341-uart [ 324.280937] usb 1-1.4: ch341-uart converter now attached to ttyUSB0

[piter-pit]

hurray, found the solution. I had to enable previliged mode so my modification to docker-compose looks like: device_cgroup_rules: - 'c 188:* rmw' privileged: true #devices: # - /dev/ttyUSB0:/dev/ttyUSB0

[bgala]

Hello all, I had the same issue with my printer. Usually it is powered off. If I tried to restart my raspi or the octoprint container, it failed because of the missing usb device. So my current solution is also to use cgroup and mknod. I modified the Dockerfile and the docker-compose file with the following lines (my printer gets always ttyUSB0):

Dockerfile (take original Dockerfile and add following line before the ENTRYPOINT):

This command create a new script in the /etc/cont-init.d folder, that is executed automatically on container start (see also s6 overlay). The command mknod /dev/ttyUSB0 c 188 0 creates a new device file that can be used by octoprint to connect to the printer as soon as the printer is available. That means, the device file can be created while the printer is off.

....
RUN echo "#!/usr/bin/execlineb -P\nmknod /dev/ttyUSB0 c 188 0" >> /etc/cont-init.d/init-usb-device
ENTRYPOINT ["/init"]

Build the new docker image: docker build . -t octoprint-custom

docker-compose.yml (add following configuration changes to your existing file):

services:
  octoprint:
    image: octoprint-custom
    devices:
      - /dev/bus/usb # map the complete usb device bus
    volumes:
      - /run/udev:/run/udev:ro  # so that serial number vendor/product ids are known
    device_cgroup_rules:
      - 'c 188:* rmw' # allow creation of /dev/ttyUSBx nodes via mknod

This worked well for the printer, how would this translate for the webcam?

[lictw]

There is more simple solution: (merge this with your docker-compose.yml)

    device_cgroup_rules:
      - 'c 1:* rw' # access to devices like /dev/null
      - 'c 188:* rmw' # change numbers to your printer
    volumes:
      - /dev:/dev

We are just mounting whole /dev (all nodes), but only to some of them granting access. No udev rules, no scripts, use with pleasure!

[LongLiveCHIEF]

We are just mounting whole /dev (all nodes), but only to some of them granting access. No udev rules, no scripts, use with pleasure!

There are many many reasons why this is a bad idea.

[sammcj]

There are many many reasons why this is a bad idea.

This is true if they don't have cgroup rules limiting access to only the specific device MAJOR and MINOR identifiers within /dev.

I believe with the correctly scoped cgroup rules it's OK as long as the devices being passed through are only in use by the container and not also the root user on the host.

[lictw]

Yep, there is normal, absolutely, we can mount all system tmpfs into container and until our process fully cgrouped, this mounts will grant only information, the containered process can't do anything with root resources. Btw, guys, it's home printing server, use normal router with firewall and relax, it must be comfortable, must!

PS @LongLiveCHIEF we with @sammcj know how to use containers, trust us!)

[lictw]

If anyone knows how to force udev to create second, real block-file for connected device (not just a symlink), it will be better! Any(!) base directory of block-device will fit my proposal, for now this is whole /dev. I have udev rule for set global rights and permanent symlink (without USB*) anyway, it will be very simple solution if udev will can create second (or move existing) block-device into, for ex., /dev/printers/ender3.

[sammcj]

Btw, guys, it's home printing server, use normal router with firewall and relax, it must be comfortable, must!

That's not good advice @lictw, you're making a lot of assumptions about how and where people host their Octoprint and how their Firewalls are setup.

People absolutely should be doing their best to secure wherever Octoprint runs.

"we with @sammcj know how to use containers, trust us!"

🤦

[sammcj]

Also, if you're allowlisting devices with a MAJOR of 1 - this includes /dev/mem which provides access to the memories entire physical address space, this might be risky (although I'm not 100% sure).

You might be better advising people to stick to something like:

device_cgroup_rules:
  - 'c 188:* rmw' # access to usb serial devices like /dev/ttyUSB0, /dev/ttyUSB1 etc... when using CH340 USB serial adapters with the MAJOR of 188
  - 'c 81:* rmw' # access to video and webcam devices like /dev/video0, /dev/video1 etc... when using Logitech webcams with the MAJOR of 81

If they experience problems they could try adding specific MAJOR 1 devices such as:

# Only ever use this as a LAST RESORT, NEVER in production and ALWAYS make sure cgroup rules are applied and set correctly!
device_cgroup_rules:
  - 'c 1:3 rw' # access to /dev/null
  - 'c 1:5 rw' # access to /dev/zero
  - 'c 1:8 rw' # access to /dev/random
  - 'c 1:9 rw' # access to /dev/urandom

Inside the container this limits /dev to:

# ls /dev/
ttyUSB0  core  fd  mqueue  null  ptmx  pts  random  shm  stderr  stdin	stdout	tty  urandom  video0  zero

A list of device names with their MAJOR and MINOR IDs can be obtained by running:

stat -c '%n major: %t minor: %T' /dev/*

[lictw]

You're right, /dev/mem was very reckless, it's really better to give personal access if the program needs special devices.

[lictw]

And okay, who want the security? I will add it for you:

echo "# merge output bellow with your personal mounts and add result into 'docker-compose.override.yml' under path 'services.NAME'"; \
  echo 'tmpfs:'; \
    find /dev -maxdepth 1 -type d | tail -n +2 | xargs -i echo '- {}'; \
  echo 'volumes:'; echo '- /dev:/dev'; \
    find /dev -maxdepth 1 -type b,c | grep -vE '/dev($|/(null|zero|random|urandom|ttyUSB|ttyACM|video))' | xargs -i echo '- /dev/null:{}'

What is it: it's a 'one-liner' that will create huge mount list, this mounts will override all real char/block-devices (except required) with /dev/null and mount tmpfs into all /dev/* directories, without a rights the access is useless, but after this it will be inaccessible in a principle.

PS: Yes it's a blacklist, but it's more then nothing!)

[12bchl]

Couple ideas:

1. Why can't dev mount point be more configurable? Seems so silly to go to this much trouble just because ttyUSB* (or other) is nested in with all other devices in /dev folder. If we could apply udev rule and mount to say /dev/rs232/ttyUSB* or something we could just pass that folder as a volume easily. You can halfway accomplish this with SYMLINK but obviously that doesn't help get the actual device into container.

2. Has anyone tried creating a persistent dummy usb device in host as a proxy? Lets say for however many devices you want access to inside container you create that many dummy interfaces. And you can use --device to easilly pass to container because they "exist" already. Then via udev rules bind the real devices as they appear to the "backend" of the existing dummy interface. I haven't played with gadgetfs too much, maybe this isn't possible and I'm out to lunch. Or maybe it would be easier in the case of usb-serial adapters to make a dummy serial device instead of dummy usb device.

[12bchl]

Hello all, I had the same issue with my printer. Usually it is powered off. If I tried to restart my raspi or the octoprint container, it failed because of the missing usb device. So my current solution is also to use cgroup and mknod. I modified the Dockerfile and the docker-compose file with the following lines (my printer gets always ttyUSB0):

Dockerfile (take original Dockerfile and add following line before the ENTRYPOINT):

This command create a new script in the /etc/cont-init.d folder, that is executed automatically on container start (see also s6 overlay). The command mknod /dev/ttyUSB0 c 188 0 creates a new device file that can be used by octoprint to connect to the printer as soon as the printer is available. That means, the device file can be created while the printer is off.

....
RUN echo "#!/usr/bin/execlineb -P\nmknod /dev/ttyUSB0 c 188 0" >> /etc/cont-init.d/init-usb-device
ENTRYPOINT ["/init"]

Build the new docker image: docker build . -t octoprint-custom

docker-compose.yml (add following configuration changes to your existing file):

services:
  octoprint:
    image: octoprint-custom
    devices:
      - /dev/bus/usb # map the complete usb device bus
    volumes:
      - /run/udev:/run/udev:ro  # so that serial number vendor/product ids are known
    device_cgroup_rules:
      - 'c 188:* rmw' # allow creation of /dev/ttyUSBx nodes via mknod

This is the ideal solution for me so far. Not sure about the author, but device is not hotpluggable. I did have to restart container after plugging in. I'm fine with that. It's a lot better than destroying container, rejigging the config and re-creating. I think you could make it fully hotpluggable by using a volume instead of device for /dev/bus/usb.

Some additional pointers on the original solution: -I commented out /run/udev volume and it still worked fine -I didn't have access to dockerfile so I do a manual docker cp to get that startup script in. Mine looks like this:

#!/bin/sh
mknod /dev/ttyUSB0 c 188 0
mknod /dev/ttyUSB1 c 188 0
mknod /dev/ttyUSB2 c 188 0
chown root:dialout /dev/ttyUSB0 /dev/ttyUSB1 /dev/ttyUSB2
chmod g+w /dev/ttyUSB0 /dev/ttyUSB1 /dev/ttyUSB2

^Note how I needed to change ownership and permissions to use the devices

[danpeig]

Hi! I just updated the full instructions in case anyone needs it...

Configuration for 3D Printer hot-plug in Docker containers

This configuration allows the container to start without a 3D printer plugged and will automatically detect and update the device list when the printer is turned on. It was tested on a brand new Raspian Bullseye 32-bit image (version 11.7) with Docker (24.0.6) from the official website (not the Rasperry PI repository). Octoprint version was 1.9.2.

The only change required for the host system is the creation of the udev rule. Container is not rebuilt, no privileged user permissions are required, there is no need to mount the /dev folder, there is no need to create any external shell script.

1. Create /etc/udev/rules.d/71-octoprint.docker.rules with the content below. Replace octoprint with the name of your container.

# allow usage/start of the container regardless of whether printer is connected
SUBSYSTEM!="tty", GOTO="end_octoprint_printers"
ACTION=="add|change", SUBSYSTEM=="tty", KERNEL=="ttyUSB[0-9]|ttyACM[0-9]", RUN+="/usr/bin/docker exec octoprint rm -rf /dev/3dprinter", RUN+="/usr/bin/docker exec octoprint mknod /dev/3dprinter c %M %m"
ACTION=="remove", SUBSYSTEM=="tty", KERNEL=="ttyUSB[0-9]|ttyACM[0-9]", RUN+="/usr/bin/docker exec octoprint rm -rf /dev/3dprinter"
LABEL="end_octoprint_printers"

2. Activate the udev rules sudo udevadm control --reload-rules && udevadm trigger . There is no need to reboot the computer.

3. Connect and turn on your printer and video camera.

4. Run stat -c '%n MAJOR: %t MINOR: %T' /dev/* , locate your printer and camera (if any) and take a note of the MAJOR ID. 3D printers are usually listed as /dev/ttyUSB0, /dev/ttyUSB1, /dev/ttyACM0 or /dev/ttyACM1.

5. Convert the MAJOR IDs from HEX to INT format ('a6' becomes '166', 'e' becomes '14', 'bc' is '188', etc...).

6. Edit your docker-compose.yaml:

   • Ensure version is 3.6 as cgroup rules behaviour changes between docker versions.
   • Add your MAJOR integer numbers to the device_cgroup_rules section.
   • Comment/delete the privileged permission (if enabled)
   • Comment/delete any references to the printer serial port in the devices section (if exist). If you added the cgroup code for the video camera, you can also comment it from devices.
   • Comment/delete any references to the host /dev folders in the volumes section (if exist).

Example:

version: '3.6'

services:
  octoprint:
    image: octoprint/octoprint
    container_name: octoprint #this should be equal to the udev rules you created
    restart: unless-stopped
    #privileged: true
    ports:
      - 5000:80 #Map to external port 5000
    device_cgroup_rules:
    #Insert your MAJOR ID numbers below
      - 'c 166:* rmw' # access to usb serial devices like /dev/ttyUSB0, /dev/ttyUSB1 etc... when using USB serial adapters with the MAJOR of 166
      - 'c 188:* rmw' # access to usb serial devices like /dev/ttyACM0, /dev/ttyACM1 etc... when using USB serial adapters with the MAJOR of 188
      - 'c 81:* rmw' # access to video and webcam devices like /dev/video0, /dev/video1 etc... when using Logitech webcams with the MAJOR of 81
      - 'c 1:3 rw' # access to /dev/null
      #- 'c 1:5 rw' # access to /dev/zero
      #- 'c 1:8 rw' # access to /dev/random
      #- 'c 1:9 rw' # access to /dev/urandom
    
#devices:
    # use `python -m serial.tools.miniterm` to see what the name is of the printer, this requires pyserial
    #  - /dev/ttyACM0:/dev/ttyACM0
    #  - /dev/video0:/dev/video0
    volumes:
     - ./octoprint:/octoprint
    environment:
     - TZ=America/Sao_Paulo
    #  - ENABLE_MJPG_STREAMER=true
  
  ####
  # uncomment if you wish to edit the configuration files of octoprint
  # refer to docs on configuration editing for more information
  ####

  #config-editor:
  #  image: linuxserver/code-server
  #  ports:
  #    - 8443:8443
  #  depends_on:
  #    - octoprint
  #  restart: unless-stopped
  #  environment:
  #    - PUID=0
  #    - PGID=0
  #    - TZ=America/Chicago
  #  volumes:
  #    - octoprint:/octoprint

#volumes:
#  octoprint:

7. Start your container from the folder where the yaml file is located sudo docker compose up -d.

8. Within Octoprint configuration menu, add /dev/3dprinter to the additional serial ports list. This will enable auto-detection.

9. Do not forget to update the OctoPrint restart command as well: s6-svc -r /var/run/s6/services/octoprint

[MaximumPotato]

@danpeig I had to dig around the web so much to find this thread, and eventually your comment. Thank you for providing instructions!

One thing I am unsure about: Is a webcam meant to function with the configuration as you've given it? Mine is not, and I've noticed the only "Additional Serial Port" your instructions mention is one identified as /dev/3dprinter. Not sure if that would be a bundle of all relevant devices from the udev script, or if something separate needs to be configured to get video output. If you don't beat me to it with an answer I'll come back to this when my headache goes away. :-)

Edit:

Forgot to mention that I'm running this in Portainer, rather than straight out of Docker. Maybe that has something to do with it? Not sure if that would interfere with step 7.