[rfc] crun extension compatibility

[cyphar]

We've had a fair few requests from folks about the run.oci.* extensions and why runc doesn't support them. Given that some of these extensions are likely not going to be put into the runtime-spec (run.oci.keep_original_groups was suggested some time ago in opencontainers/runtime-spec#1020 but the core feature is basically a security anti-feature).

The two main features I see requests for quite often are:

• run.oci.keep_original_groups which lets you disable the setgroups step when using rootless containers with newgidmap. This lets the container access host groups (which is usually a security issue) which then allows you to access the GPU if the original user had access to the video group. podman has a magic flag to let you use this, but obviously it only works with crun at the moment since it's controlled by a non-standard annotation (benefits of vertical integration, I guess you could say).

• run.oci.handler is a generic mechanism for crun to run non-standard container workloads rather than doing a straight-forward exec of a binary. In particular run.oci.handler=wasm causes crun to run WASM code in the context of the pid1 through various WASM runtimes directly (crun can be compiled with wasm support, where a wasm runtime library is linked into crun and then crun runs the runtime on the target .wasm or .w. I'm a little worried that it seems that nobody can agree on what WASM runtime should be used (and crun itself supports four different runtimes AFAICS) but we could do something minimal with https://github.com/tetratelabs/wazero to see how many users want this feature.

Though there are probably others.

What should the policy be for supporting these? My view is that if users need these features we should support them even if they're non-standard because the alternative is that folks switch to crun. (One obvious point of confusion is that crun has called the annotations run.oci.* which users might think means they are official OCI annotations -- obviously only org.opencontainers.* annotations are official but that doesn't really help with public perception.)

[rata]

I agree with implementing both extensions.

I don't know if runtime-spec would like to have those things, I think we should implement the extensions first, though. Then we can see, as runtime-spec also has VM-runtimes, etc.

[AkihiroSuda]

(One obvious point of confusion is that crun has called the annotations run.oci.* which users might think means they are official OCI annotations -- obviously only org.opencontainers.* annotations are official but that doesn't really help with public perception.)

We should implement them under org.opencontainers.runc.* namespace, and ask crun to support the same convention

[cyphar]

I think org.opencontainers.runc.* isn't the right namespace for us, since that is reserved for spec-related things. It's a little complicated because we are also an OCI project... I think we would need to add something to the runtime-spec to specify that org.opencontainers.runc.* is reserved for runc's usage.

They used to call io.crun but they had to change it a long time ago (https://github.com/containers/crun/pull/199) and unfortunately they went with run.oci rather than something more reasonable like com.redhat.crun (I get that they don't want https://github.com/containers to be branded as "RedHat stuff" but like ... come on).

I think it's a bit late to ask them to change it to something else, and re-using the same annotations makes more sense for compatibility (podman currently sets these run.oci.* annotations and renaming them wouldn't help us with supporting podman features when using runc).

[AkihiroSuda]

I think org.opencontainers.runc.* isn't the right namespace for us, since that is reserved for spec-related things.

org.opencontainers.runc.* has been already used for runc-specific implementation details.

In runc features we already have:

• org.opencontainers.runc.checkpoint.enabled
• org.opencontainers.runc.commit
• org.opencontainers.runc.version

[AkihiroSuda]

Would it be possible to ask Red Hat to handover the oci.run. domain to OCI, and make it a part of the OCI Runtime Spec (with deprecation) ?

Even if not, we may still implement run.oci.* stuffs, but as an OCI project we should not encourage using that namespace at least.

[cyphar]

Anything is possible I guess, but I don't know how @giuseppe would feel about it. I do think the oci.run domain (and run.oci.* annotations) have lead to some confusion about whether their extensions are an "OCI thing", and I would prefer them to have used com.redhat.crun.* to make it clearer that these are vendor extensions (or even oci.run.crun like they did for their crun features annotations, but for the above reasons I would still prefer com.redhat.crun.*).

But this is a 6-year-old decision at this point, and they do legitimately own that domain, idk. I wouldn't fault @giuseppe for not wanting to do it, especially given that some of his proposals where annotations were meant to be stop-gaps still haven't been merged...

Also, from my perspective supporting the existing annotations is a runc problem while whether they should continue to use run.oci.* is a runtime-spec / OCI trademark board question.

org.opencontainers.runc.* has been already used for runc-specific implementation details.

runc features is a little different to annotations -- the spec explicitly says that

The org.opencontainers namespace for keys is reserved for use by this specification, annotations using keys in this namespace MUST be as described in this section. The following keys in the org.opencontainers namespaces MAY be used: [...]

For runc features it's kind of inevitable that we would need to use org.opencontainers.runc.*.

For what it's worth, I'm working on a spec PR for defining how vendors should reference their feature extensions in runc features and making sure that we don't end up in this situation is one of the goals...

[kolyshkin]

I think we should increase crun compatibility in general, and support those annotations in particular.

As for the naming, we can document that those are non-standard annotations and have nothing to do with OCI despite the name.

[giuseppe]

I own the oci.run domain, not Red Hat. I am fine to hand it over (one thing less to pay for 😄), and personally I'd even donate/move the entire crun repository under opencontainers.

I am currently using oci.run only for the annotations and for security reports (https://github.com/containers/crun/blob/main/SECURITY.md) where I tried to make clear that it is not affiliated with opencontainers.org. The email address is easy to fix, but what about the annotations? Can they still be used with the same meaning?

[AkihiroSuda]

Thanks @giuseppe cc @caniszczyk @cyphar WDYT about the domain?

Can they still be used with the same meaning?

I think this is fine for the existing annotations, but some of them could be promoted to a proper JSON structure? For newer experiments I'd suggest using the io.crun.* (as in the existing io.crun.keep_original_groups) or com.redhat.* namespaces.