OCI authentication support

[stgraber]

Currently our interactions with OCI registries doesn't support authentication.

We should add support for this, at minimum supporting HTTP basic auth encoded type credentials in the OCI registry URL. Possibly also supporting auth helpers on the client side, though that part gets a bit weird as it will prevent refreshes on the server side.

[cory-chang]

Hi @stgraber! @pi2chen and I are interested in working on this issue as part of Professor Chidambaram's Virtualization class. Could you assign this issue to us?

Looking forward to working in this repo, thanks!

[stgraber]

Done. Same as the other two, I'll add @pi2chen once he comments.

This is another issue that's not marked as Easy as we haven't quite determined how far we should take this right now.

I had a chat about some of that with @xnox and others before and in the Docker world, there is the concept of a credential helper than can be hooked into the registry interaction process, basically allowing for a temporary token of some kind to be retrieved and used to interact with a private registry.

We may be able to use something like this too, basically extending the config in ~/.config/incus/config.yml to indicate that the registry requires authentication and then have it use the helper to get that token.

This should be sufficient for the CLI as it will be able to use basically the same logic as Docker at that point. It would be more of a problem for other clients, like our web UI, where there's no access to such an helper, but that will be a problem for another day :)

@xnox any chance you can give us a few more details on the common way this is handled, and if possible at all, provide us either with a private registry we can test against (including credentials) or some pointers on how to setup a local private registry with the most common authentication mechanism so we can validate on that instead.

[xnox]

a cred helper just needs to emit json for ServerURL Username Secret, it can be a shell script that echos that.

the most trivial private registry is like k3d registy create.

To put it behind auth, one needs to front that with nginx password auth.

So a basic task could be:

1. install k3d
2. start k3d localhost registry with k3d registry create
3. install nginx
4. configure nginx, to have https self signed / snakeoil cert (or letsencrypt cert)
5. add basic username / password auth in nginx, and reverse proxy on localhost to said k3d registry
6. create "docker-credential-helper-myprivateregistry" that echos json of the requested server url, and that static username and password
7. configure docker to use that credhelper shell script for your private registry
8. test that with docker push/pull to that private registry works

Note, that this is a bit insecure, as there is no mechanism to enforce ACL => meaning if people can find that exposed registry, they can push gigabytes of data to it, and instantly use it as a dropbox to share files.

An alternative way, is to use authenticated access to dockerhub => meaning login to docker hub, and use creds to access it. The best way to test that it works, is to start off with invalid username/password, which docker pull should prevent from using. Then you know invalid auth is blocking access. If that gets rejected via incus too; swap in valid credentials => and it should then just work correctly.

@stgraber is this what you were looking for?

[xnox]

for webui

• provide input field for a token
• ask users to run the credhelper locally to get the json
• copy-paste json, and use it

Basically a slightly user-assisted api call; cause regular incus client would just pass the json to the daemon to unpack and use in the calls, until expiry.

[xnox]

Tokens can be valid for like 1h, 6h, or be long lived - for example a year. Thus when client sends a token, the daemon can store it, and keep on using. Once it stops working, create errors/messages asking the client to provide a fresh one (as in renewed) token.

[stgraber]

for webui

* provide input field for a token

* ask users to run the credhelper locally to get the json

* copy-paste json, and use it

Basically a slightly user-assisted api call; cause regular incus client would just pass the json to the daemon to unpack and use in the calls, until expiry.

Yeah, that's what I was thinking. We'd just offer the auth token field and ask that they submit it that way.

[stgraber]

Okay. I've gotten myself a private repository on my Google Workspace account.

The way this works is that I'm provided with a docker-credential-gcloud binary which supports a get argument and reads the domain from stdin (in my case northamerica-northeast1-docker.pkg.dev).

This then outputs a JSON with two fields:

• Username
• Secret

Internally, we're using skopeo for the image retrieval, so that's what will need to be able to consume those fields. We don't want to have the secrets be passed as arguments as that would allow anyone on the system to see them. So instead we need to jump through a few hoops.

So the result is that we need to:

• Extend the syntax in ~/.config/incus/config.yml to allow specifying a credentials_helper. That's in shared/cliconfig/remote.go
• Make it possible to set this through incus remote add with a new --credentials-helper option. That's in cmd/incus/remote.go
• Use the credentials_helper option in GetImageServer when in the OCI case. That's still in shared/cliconfig/remote.go. What should be done here is combine the existing remote.Addr with the Username and Secret values you get by running the credentials_helper, encoding it as the username and password field in the URL. So if the registry is https://docker.io and credentials helper gives Username=foo Secret=bar, this becomes https://foo:[email protected]. The net.url package can help do this cleanly.
• Have our invocation of skopeo inspect and skopeo copyhandle those credentials. This is inclient/oci_images.go`.

For that last point, here is a quick proof of concept I did here which works for me:

diff --git a/client/oci_images.go b/client/oci_images.go
index 72745a4512..d39f46d8f5 100644
--- a/client/oci_images.go
+++ b/client/oci_images.go
@@ -3,6 +3,7 @@ package incus
 import (
        "compress/gzip"
        "context"
+       "encoding/base64"
        "encoding/json"
        "fmt"
        "io"
@@ -360,6 +361,38 @@ func (r *ProtocolOCI) GetImageAlias(name string) (*api.ImageAliasesEntry, string
                }
        }
 
+       // Parse the URL.
+       uri, err := url.Parse(r.httpHost)
+       if err != nil {
+               return nil, "", err
+       }
+
+       creds, err := json.Marshal(map[string]any{"auths": map[string]any{fmt.Sprintf("%s://%s", uri.Scheme, uri.Host): map[string]string{"auth": base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s", uri.User.String())))}}})
+       if err != nil {
+               return nil, "", err
+       }
+
+       uri.Scheme = "docker"
+       uri.User = nil
+
+       authFile, err := os.CreateTemp("", "incus_client_auth_")
+       if err != nil {
+               return nil, "", err
+       }
+
+       defer authFile.Close()
+       defer os.Remove(authFile.Name())
+
+       err = authFile.Chmod(0o600)
+       if err != nil {
+               return nil, "", err
+       }
+
+       _, err = fmt.Fprintf(authFile, "%s", creds)
+       if err != nil {
+               return nil, "", err
+       }
+
        // Get the image information from skopeo.
        stdout, _, err := subprocess.RunCommandSplit(
                context.TODO(),
@@ -367,7 +400,8 @@ func (r *ProtocolOCI) GetImageAlias(name string) (*api.ImageAliasesEntry, string
                nil,
                "skopeo",
                "inspect",
-               fmt.Sprintf("%s/%s", strings.ReplaceAll(r.httpHost, "https://", "docker://"), name))
+               "--authfile", authFile.Name(),
+               fmt.Sprintf("%s/%s", uri.String(), name))
        if err != nil {
                logger.Debug("Error getting image alias", logger.Ctx{"name": name, "stdout": stdout, "stderr": err})
                return nil, "", err

Obviously this isn't clean code. We need to pull out the logic to run skopeo into a function we can share for both the skopeo inspect and skopeo copy and we'll need to only do the authFile logic if uri.User != nil.

With all that done, we'll have support for credentials helpers in the command line without having to do any API level change within Incus as we'll just use our existing ability to pass through HTTP authentication credentials through the URLs. Then when it comes time to support that in the UI, the UI can take the credentials JSON, parse it and include the values in the URL it sends to the server, easy enough to do (out of scope for this issue though, we only care about CLI here).

[stgraber]

To test this, you can:

• Install the Google SDK, this will give you the gcloud and docker-credential-gcloud commands
• Create a creds.json file with the content of https://dl.stgraber.org/creds-incus-1700.json
• Run gcloud auth login --cred-file=login.json
• Confirm that the login worked by running echo https://northamerica-northeast1-docker.pkg.dev | docker-credential-gcloud get

If that gets you one of those credentials JSON with a Username and Secret field, then you're all set to work on and test this.

Adding the registry should then be done by using the URL https://northamerica-northeast1-docker.pkg.dev/stgraber-1525358518329/test-registry. It contains a single image, alpine:latest.

With this feature fully implemented, I'd expect to be able to do:

incus remote add oci-stgraber https://northamerica-northeast1-docker.pkg.dev/stgraber-1525358518329/test-registry --credentials-helper=docker-credential-gcloud
incus image info oci-stgraber:alpine:latest
incus launch oci-stgraber:alpine:latest foo

[pi2chen]

Thank you @stgraber! Feel free to assign me to this issue.

Looking forward to working on it!

[xnox]

This then outputs a JSON with two fields:

• Username
• Secret

it should be three fields... it should be ServerURL too. Note that even with the same helper, one can have different tokens/auth for different url (one-to-many). I.e. in AWS ECR one can have the helper generate different tokens depending on how projects are setup.

[stgraber]

it should be three fields... it should be ServerURL too. Note that even with the same helper, one can have different tokens/auth for different url (one-to-many). I.e. in AWS ECR one can have the helper generate different tokens depending on how projects are setup.

At least for Google, the helper doesn't provide a URL in its response, but it takes a URL as its input (through stdin). That's how it would handle different tokens based on the URL of the registry.

So our current workflow is to write the registry URL through stdin to an helper using its get command. The response then gives us the expected username and secret for the URL we provided it.

[xnox]

the helper doesn't provide a URL in its response

Cute!

Will check code of other helpers and maybe provide sample outputs, as unit test cases.

[cory-chang]

Hi @stgraber, we've had a chance to dive a little deeper into this now, and had a few questions to clarify.

1. I'm looking at the feature fully implemented, so I assume the user needs to authenticate to the Google Cloud SDK before running the incus commands.
2. Can we assume that the functionality of all helpers will be the same? Ie. the command is <helper> get, and the output will be similar JSON?
3. For calling the credential helper, should we do that directly in GetImageServer, or is that something that should be done in ConnectOCI? I presume in GetImageServer, perhaps as a helper function, since we need to have the remote.Addr include username and secret already before passing into ConnectOCI.
4. Once we set remote.Addr with the username and secret, that address just gets passed through until skopeo needs it? We would then parse in a helper function for both GetImageAlias and GetImageFile into an authfile before passing to skopeo.

[stgraber]

I'm looking at the feature fully implemented, so I assume the user needs to authenticate to the Google Cloud SDK before running the incus commands.

Yeah. The exact process for doing that will vary based on registry. In the case of Google, it's gcloud login which handles that part.

Can we assume that the functionality of all helpers will be the same? Ie. the command is <helper> get, and the output will be similar JSON?

Yeah, the <helper> get and the JSON they return is standard. It's documented here: https://docs.docker.com/reference/cli/docker/login/#credential-helper-protocol

For calling the credential helper, should we do that directly in GetImageServer, or is that something that should be done in ConnectOCI? I presume in GetImageServer, perhaps as a helper function, since we need to have the remote.Addr include username and secret already before passing into ConnectOCI.

I think it makes most sense as part of GetImageServer since that has access to the remote config struct and therefore to the new field we'll be adding. It can then mangle the URL passed to ConnectOCI to include the token.

We may eventually move the logic into ConnectOCI by extending the args struct, but this will likely be done quite a bit later. Basically our Go client (client/) is available to external users and so is treated as stable API. Whatever gets added to it, can't really be changed or removed later. In contrast, we're more willing to extend, modify or even revert changes to the CLI and its helpers like cliconfig.

Once we set remote.Addr with the username and secret, that address just gets passed through until skopeo needs it? We would then parse in a helper function for both GetImageAlias and GetImageFile into an authfile before passing to skopeo.

Yep, exactly. The rest of the logic should work fine with the URL encoding the token, it just needs to be split out before feeding it to skopeo.

[cory-chang]

Perfect, I really appreciate you checking our assumptions. Thank you!

[cory-chang]

Hi @stgraber, one other point of clarification, we would need to specify the protocol as OCI when adding the remote right? Since the remote server is a OCI registry.

Therefore, the command would be: incus remote add oci-stgraber https://northamerica-northeast1-docker.pkg.dev/stgraber-1525358518329/test-registry --credentials-helper=docker-credential-gcloud --protocol=oci

[stgraber]

Yeah, that's correct. Sorry, my example from earlier was inaccurate.

[xnox]

Note that GAR supports storing arbitrary blobs, if one plumbs such authentication through, one would be able to store Incus / Images / Simplestream resources too, and have authentication. Thus after landing this initial support for the oci protocol only; one can expand this later to support other registries.

Also, it might be worth considering if OCI remotes can be used natively for regular incus container & VM images, as OCI registry allows to push and store arbitrary blobs.

[cory-chang]

@stgraber Sorry, one other question, after running make, I can run some incus commands successfully, such as the incus image info, but I cannot run incus admin init.

It errors out with Error: Failed to connect to local daemon: Get "http://unix.socket/1.0": dial unix /var/lib/incus/unix.socket: connect: no such file or directory

Do you have any ideas why?

[stgraber]

incus image info directly talks to the remote server from the client. incus admin init or incus list talks to the local Incus daemon instead.

The error you're showing suggests that you do not have the Incus daemon running on your system.

[rsyring]

FWIW, I was able to get authentication to ghcr.io private repository by setting up the login in skopeo (which Incus uses to fetch the image):

sudo /opt/incus/bin/skopeo login ghcr.io

Did I miss a better way?

FWIW, for the REST API in particular, I'd be content for something simple while waiting for all the credential helper logic architecture to be fleshed out and built:

# This issues a POST to  /1.0/instances
inc.instances.add(
        name,
        source={
            'type': 'image',
            'mode': 'pull',
            'protocol': 'oci',
            'server': 'https://ghcr.io/org/repo',
            'alias': 'foobar:latest',
            'username': 'me',
            'password': 'secret',
        },
        type='container',
        start=True,
        project=project_name,
    )

Maybe Incus can use that for just the first pull and then delete it? I realize that leaves the server unable to pull future images until it gets another API interaction but, in my case, that would be preferred as it ensures the person launching the instance has access to the credentials at the time the instance launch is requested.

[xnox]

Did I miss a better way?

i believe this is still the current best practice. As in sudo to the daemon user and provide credential helpers / config / auth there, as used by scopeo there.

[pi2chen]

Hi @stgraber, @cory-chang and I will no be working on this issue as our class has concluded.

We modified the CLI/YAML configurations to support specifying Docker credential helpers, updated the subprocess module to run credential helper binaries and handle their outputs, and implemented passing retrieved credentials into authenticated URIs for image pulls. We also began work on temporary authfile creation for skopeo but this part remains incomplete.

Our implementation can be found here: https://github.com/cory-chang/incus/tree/oci-auth.

Thank you again for the opportunity to contribute—it was a valuable and rewarding learning experience, and we are very grateful.

[stgraber]

@pi2chen thanks, I'll pick it up from there!