[Feature] Docker container overhaul

[feederbox826]

Is your feature request related to a problem? Please describe.

• Running as root in the docker container can cause issues with permissions
  • also is a minor security concern #684
  • while USER is an option, it breaks since stash looks under $HOME/.stash for config files
  • since it is not container-wide, ffmpeg might possibly be running as another user? #2797
• re-installing new python libraries on every new update is very troublesome

Describe the solution you'd like

• ~~A proper LSCR.io - like base for~~ handling common environment variables like
  • TZ #1238
  • PUID/ PGID #684
• Github Container Registry + DockerHub builds #3558
• a proper DockerHub README
• officially supported CUDA/ QSV images (eventually ™️) (#4091 related)
• an entrypoint script to install and update python dependencies specified by the user on startup
  • (also resolves closes #4102)
• a persistent volume for python dependencies
• a default /config directory instead of /root/.stash

Describe alternatives you've considered N/A

Additional context I want to work on this. Please assign me to this. I've done this for other projects and I am eager to finally do it for stash

[feederbox826]

High level overview and TODO tracker

developing in https://github.com/feederbox826/stash-s6

container

• entrypoint *ash script
  • [x] runtime python dep installation (r: #4102)
    • [x] automatic requirement finding, merging and upgrading
  • [x] migrations/ data storage
    • [x] stashapp/stash
    • [x] hotio/stash
    • [x] custom configurations
    • [x] /config mount
    • [x] mount detection
• rootless/ USER handling (r: #684)
  • [x] rootless mode
  • [x] k8s "security compatibility"
  • [x] permission checks & warnings
  • [x] permission corrections (r: #2797)
• root container restrictions
  • [x] SKIP_CHOWN flag
  • [x] PUID/ PGID user switching
• misc
  • [x] TZ ENV support (r: #1238)
  • [x] arm <v7 support
• hardware acceleration
  • ffmpeg flavors
    • [x] jellyfin-ffmpeg
    • [ ] lscr.io build
    • [ ] stock (debian)
  • hardware acceleration tests (r: #4091)
    • [ ] ~~lscr.io without drivers~~ removed (https://github.com/feederbox826/stash-s6/issues/2)
    • [x] base ffmpeg driver installs (across arch)
    • [x] jellyfin without drivers
    • [x] VAAPI iHD (gen8+)
    • [x] VAAPI i965 (gen5+)
    • [x] QSV (Intel)
    • [ ] Intel ARC
    • [x] NVENC (NVIDIA)
      • [x] multi-stream NVENC
    • [ ] V4L2 (rPi)

Building

• [ ] Makefile
• [ ] Github Actions
  • [x] simultaneous arch builds
  • [ ] Builds from source
• [x] publish to docker.io, ghcr.io (r: #3558)

Documentation

• updates
  • [ ] update dockerhub README
  • [x] add links to hwaccel flavors, make images links
  • [x] update docker-compose ENV, mounts, GPU bindings
• additions
  • [x] docker image
    • [x] add documentation for entrypoint variables
  • [x] hw acceleration
    • [x] breakdown of flavors/ hardware acceleration
    • [x] link test videos for reencoding
    • [x] example arguments
    • [x] device mounts

[Leopere]

The theory of why the changes need made is on par with what I’d want implemented. Privilege reduction proper installation of software requirements my biggest issue is that it needs to help users from before keep working ideally

[feederbox826]

it needs to help users from before keep working ideally

Yep, my plan right now is to test it in my own repository to iron out any bugs, and write mgirations for both hotio/stash and stashapp/stash so that users can migrate over seamlessly or as seamless as mv && chown -R can get

[stg-annon]

another problem people have with docker installs is python packages, might be worth taking a look at setting up a venv so that plugins and scrapers that require other packages don't need to reinstall them on every reboot

alternately we may be able to get away with just setting a couple environment variables to a mount from the host PIP_TARGET PYTHONHOME

[MaverickD650]

Could we keep a rootless container option without s6? It causes particular problems on kubernetes as it aggressively CHOWNs files and can cause a container to fail to start by exceeding the probes time limits. It must also be run as root on kubernetes to allow the inital s6 operations to take place which isn't ideal since we're aiming for rootless.

[feederbox826]

alternately we may be able to get away with just setting a couple environment variables to a mount from the host PIP_TARGET PYTHONHOME

https://github.com/feederbox826/stash-s6/blob/b71eb246ae412192e1d9426cc01b4829c546738f/alpine.Dockerfile#L16-L19

cache and PIP_INSTALL_TARGET are both set to a volume

[feederbox826]

Could we keep a rootless container option without s6? It causes particular problems on kubernetes as it aggressively CHOWNs files and can cause a container to fail to start by exceeding the probes time limits. It must also be run as root on kubernetes to allow the inital s6 operations to take place which isn't ideal since we're aiming for rootless.

the chown operations are done by the init scripts, not by s6 itself. Unfortunately my goal with this is to be as beginner-friendly as possible, following lscr.io conventions and s6 allows us some leniency without creating a excessively long docker-entrypoint script and adding in modular dropins for different flavors. I'll look into creating a minimal (alpine + python + binary) image, but I'm not a fan of fragmeting further from the base

[PrivatePuffin]

A proper LSCR.io

following lscr.io conventions

From a professional perspective, LSIO containers are not good containers. They are borderline problematic on anything but a docker(-compose installation), for example on kubernetes. Due to, essentually, requiring root. They essentially cut corners from good container practices, to build a container that "just works" for the average joi.

Calling LSIO a "convention", makes it seem like it's somehow a high grade container standard, which it's quite obviously not. It's better than the average container, but that comes at a cost.

also is a minor security concern https://github.com/stashapp/stash/issues/684

To be clear: The main container still runs as root in S6 and requires multiple(!) layers of security precautions in kubernets to be disabled. It also prevents things like supplemental groups to be used correctly.

while USER is an option, it breaks since stash looks under $HOME/.stash for config files

The solution for which is adapting this behavior, not migrating to S6 overlay based containers

[feederbox826]

Either way I'm discussing with @Leopere to drop s6 in favor of a (b)ash script but,

From a professional perspective, LSIO containers are not good containers.

They're not good but they work out of the box for 99% of users which imo is more important than accomodating someone who is already building their own helmchart and should know how to build their own container

k8s, as much as we would like to try and dance around, is not exactly docker, but that's a tangent for another time.

To be clear: The main container still runs as root in S6 and requires multiple(!) layers of security precautions in kubernets to be disabled. It also prevents things like supplemental groups to be used correctly.

iirc #684 is not to run a root-less container, is to run as a non-priviledged user within a rooted container.

[feederbox826]

I'll try to get the bash scripts out the door, but I have a some personal negative experiences with k8s but PRs are always welcome.

[PrivatePuffin]

imo is more important than accomodating someone who is already building their own helmchart and should know how to build their own container

Writhing a helm chart is a lot different than writhing containers as they are a LOT easier to standardise and maintain. Easier in terms of dev-time per application. Just because I can technically do something, doesn't mean the project I maintain has the manpower to do so on a scope of 900 help charts.

k8s, as much as we would like to try and dance around, is not exactly docker, but that's a tangent for another time.

Well, in terms of security: Yes it offers a lot more options to lock-down. But in general it has quite clear best-practices that aline with docker itself as well in certain areas:

But it's not rocker science:

• Accept USER being set
• Dont change user
• Don't chown (without copy)
• Dont chmod
• Don't write to directories already containing files (only write to folders that can be mounted)

[scruffynerf]

As someone who has written for and used both k8s and docker, in the context of Stash, k8s makes no sense for most people. Docker is a much more easily supported standard, and more so, the level of support it generates is already high. K8s would not improve matters. As much as I feel like some of the docker users could be better served by a native app, at least there are reasonable arguments in favor of Dockerizing.

[feederbox826]

Dropped s6 in favor of an ash entrypoint script

Limitations of root-less

• volume mounts are by default not accessible
• nvenc stream needs root to run ldconfig
• cannot migrate from root to non-root

but that's inevitable. If you have further suggestions, you seem much more versed in rootless and k8s compatibility so I look forward to your PRs

[jangrewe]

Please also add the stashapp-tools python module to fix some of the plugins. :-)

[feederbox826]

Please also add the stashapp-tools python module to fix some of the plugins. :-)

already bundled :) https://github.com/feederbox826/stash-s6/blob/main/stash/root/defaults/requirements.txt

[feederbox826]

Just a quick status update while I'm here - Right now it's down to figuring out mutli-arch workflow and makefile support for local and CI integration. Progress has been slow since I'm not used to this many layers of build dependencies and integrating into the existing CI for multiarch releases

[d3f113]

Would be great to add an environment variable to install packages like it is done in: https://github.com/jlesage/docker-jdownloader-2 Environment variable is called: INSTALL_PACKAGES

[feederbox826]

Would be great to add an environment variable to install packages like it is done in: https://github.com/jlesage/docker-jdownloader-2 Environment variable is called: INSTALL_PACKAGES

not sure if this is a general comment or a critique of the current method - the current method is to search the plugin and scraper dirs reursively for matching requirements.txt as well as it being a mappable override https://github.com/feederbox826/stash-s6/blob/main/stash/root/opt/entrypoint.sh#L231-L290

[toby666123]

Any news on this, do you have a rough assumption maybe for when the new docker images will become available?

Thanks a lot for your efforts!!

[feederbox826]

Sorry, I've been taking an extended break from stash et al. after getting burnt out. No promises but I'm aiming to get back on track by July 😓

[Maista6969]

Take care of your self @feederbox826, your expertise is appreciated but nothing worth burning out over! I'm sure someone else can pick up the great work you've already put in here

[feederbox826]

I'm close to being able to wrap up the project independently (build-from-source excluded)

I would love to have some testers that have

• Intel ARC
• Pi 4+ with V4L2

and some info if AMF (AMD) is officially supported

[toby666123]

I'm close to being able to wrap up the project independently (build-from-source excluded)

I would love to have some testers that have

• Intel ARC
• Pi 4+ with V4L2

and some info if AMF (AMD) is officially supported

Seems nobody here has the respective Hardware available. Could you also just finish the project without support for the 2 platforms?

[feederbox826]

I got an ARC myself and someone else verified V4L2, I haven't figured out the build process yet