image Add /etc/containers/auth.json and /etc/containers/auth.d/ dir

This is a follow-up of this PR https://github.com/containers/image/pull/1746 - original description follows (originally written by by @cgwalters ).

EDIT: Design discussion in https://github.com/containers/image/issues/2614

A long-running tension in the docker/podman land is around running as a system service versus being executed by a user. (Specifically a "login user", i.e. a Unix user that can be logged into via ssh etc.)

For login users, it makes total sense to configure the container runtime in $HOME.

But for system services (e.g. code executed by systemd) it is generally a bad idea to access or read the /root home directory. On image based systems, /root may be dynamically mutable state in contrast to /etc which may be managed by OS upgrades, or even be read-only.

A compounding problem today is that the podman login default is to write to $XDG_RUNTIME_DIR which is not persistent - but often we want persistent state.

For these reasons, let's introduce "system wide" persistent authfiles that follow the config files specification.

alignment with bootc/ostree

Many image based systems will want to run containers via system services, and may specifically mount /home and /root as tmpfs. Today of course, they could put authfiles elsewhere and copy them on boot into /root (or pick custom authfile locations and point at them directly) but I think many simple use cases will be happy with "default global pull secret" as a baseline starting point.

Kubelet

An important goal is that kubelet can use these locations in addition to /var/lib/kubelet/config.json as well. I am not aware of any work on kubelet upstream to change its model, but it's not required; we can recommend that kube users basically do ln -sr /var/lib/kubelet/config.json /etc/containers/auth.json perhaps.

But probably what we want is a way to tell the kubelet to also inherit the default CRI's authfile paths?

Oct 15 '24 14:10 ipanova

@mtrmac @vrothberg @rhatdan @cgwalters PTAL

This is a follow up PR of https://github.com/containers/image/pull/1746 It does not contain tests yet. There has been a lot of discussion but here's the summary.

What it does:

introduces new system-wide /etc/containers/auth.json and directory with drop-ins /etc/containers/auth.d/
these paths are preferred when process is ran in systemd as root
these paths are considered as last when process is ran by root ( not in systemd)
the precedence of drop-ins is being follow by these specs https://uapi-group.org/specifications/specs/configuration_files_specification/#drop-ins
error is raised if these paths are readle by g/o

By introducing system-wide auth file and auth.d dir it covers cases for:

system services, like bootc and its logically bound images where code is executed by systemd
where there is single node use with root access

Oct 23 '24 11:10 ipanova

I’m sorry, I am not silently ignoring this, but I’m also swamped with priority work now.

Please sign off per CONTRIBUTING.md to pass the DCO check and allow other tests to run. (Skopeo tests, are, as of today, failing on a version mismatch, that’s blocked on https://github.com/containers/storage/pull/2143 )

Oct 23 '24 18:10 mtrmac

Since there was some legitimate continuing debate about the desirability and design of this, let's split that out into https://github.com/containers/container-libs/issues/196

Oct 30 '24 15:10 cgwalters

An attempt at a summary of a video call, from my POV — so that we can see where there are more questions, and so that we can continue where we left off::

There’s a discussion about subscription-manager automatically providing registry credentials, in a way which is automatically available to everything on the node. I think that’s fine, and it should be read by truly everything (not restricted to files 0600 root:root).
Other than this, in the simple cases, adding more default paths specific to root or systemd services is not very attractive to me:
- A completely task-dedicated computer (like an OpenShift node) is probably managed by automation, and that automation can point at any credential files it wants, so hard-coding defaults in c/image is unnecessary
- A multi-purpose computer with multiple applications (“a pet”) should, typically, separate registry credentials used by different services, so that impact of vulnerabilities is constrained (both by ordinary credentials and by SELinux); making it attractive for users trying to run N services with N different credentials to put all N of them in a single file, and to make that file accessible by N services, would encourage bad security practices.
To me, the much more interesting proposal was to allow an “~immutable /etc”, or more precisely /etc (mostly?) “functionally” generated from ConfigMap-like inputs, in order to reduce the truly mutable state that exists on the system; and in that case those ConfigMap-like inputs need some destination to write into.
- Overwriting /root/.config/containers/auth.json by automation is obviously undesirable
- Overwriting /run/user/0/containers/auth.json would be a bit better, but, still, having the user-managed state from podman login interfere with / overwrite the credentials from the ConfigMaps is unintuitive [A]
So, something that might make sense:
- /etc/containers/[global-]auth.json{,.d} for the subscription use case, and other truly node-global credentials
- /etc/containers/root-auth.json{,.d} (.d to allow multiple ConfigMaps, probably — and because we would have that already implemented for global-auth.json.d anyway) for credentials only used by root (and we might enforce that they are only readable by root)
My questions:
- Do we, after all, want to read root-auth.json before /root/.config/, for the systemd services (and for $unknown other use cases??), to silence SELinux denial noise? Maybe the PR as proposed is actually what we want here.
- (Added after the call:) Looking at [A] again — do we ever want podman login and the ConfigMaps to interact / interfere? It seems to me that the answer is an obvious “no” … and that would point towards not having a default root-auth.* at all, again, and forcing services that do want to use this root-on-the-node set of secrets to use an explicit --authfile option, so that they choose between the “interactively-managed credential locations” and the “automation-managed credential locations”.

Nov 04 '24 14:11 mtrmac

There’s a discussion about subscription-manager automatically providing registry credentials, in a way which is automatically available to everything on the node. I think that’s fine, and it should be read by truly everything (not restricted to files 0600 root:root).

Link?

A completely task-dedicated computer (like an OpenShift node) is probably managed by automation, and that automation can point at any credential files it wants,

Yes but we have the papercut that anyone debugging things interactively needs to run podman --authfile=/var/lib/kubelet/config.json - which we arguably could and should fix today by just symlinking /run/user/0/auth.json -> /var/lib/kubelet/config.json by default.

This does relate though to the podman login discussion in that podman login becomes a trap in that it'd write to the machine managed state, but I guess we really just need to fix that ultimately by a configmap-style approach where the cluster-owned state is actually readonly anyways.

Do we, after all, want to read root-auth.json before /root/.config/, for the systemd services (and for $unknown other use cases??), to silence SELinux denial noise?

To be clear in practice I think there aren't any denials today in the Fedora policy; systemd units are effectively unconfined by default including reading/writing to home directories.

Anyways when you have a minute can you glance at https://github.com/containers/container-libs/issues/196 ?

Nov 04 '24 20:11 cgwalters

There’s a discussion about subscription-manager automatically providing registry credentials, …

Link?

I remember an in-person discussion at DevConf; I’m not sure there is a tracker.

Nov 04 '24 20:11 mtrmac

Hi, and thank you for your contribution!

We’ve recently migrated this repository into a new monorepo: containers/container-libs along with other repositories

As part of this migration, this repository is no longer accepting new Pull-Requests and therefore this Pull-Request is being closed.

Thank you very much for your contribution. We would appreciate your continued help in migrating this PR to the new container-libs repository. Please let us know if you are facing any issues.

You can read more about the migration and the reasoning behind it in our blog post: Upcoming migration of three containers repositories to monorepo.

Thanks again for your work and for supporting the containers ecosystem!

Aug 26 '25 14:08 jankaluza