Could not resolve Private Feature when pre-building a Codespace (not a container image)

[metaskills]

So we have several private features at our org customink/features and I was hoping to do a prebuild with some projects an noticed the CLI raised a Could not resolve Feature error even though I granted the prebuild the needed permissions during creation time. Is this a bug?

2022-11-10T01:06:35.0172860Z * Processing feature: ghcr.io/devcontainers/features/common-utils
2022-11-10T01:06:35.2694805Z * Processing feature: ghcr.io/customink/features/debug
2022-11-10T01:06:35.5223311Z (!) WARNING: Falling back to deprecated GitHub Release syntax. See https://github.com/devcontainers/spec/blob/main/proposals/devcontainer-features.md#referencing-a-feature for updated specification.
2022-11-10T01:06:35.5224015Z Github feature.
2022-11-10T01:06:35.5224680Z Could not resolve Feature 'ghcr.io/customink/features/debug'.  Ensure the Feature is published and accessible from your current environment.
2022-11-10T01:06:35.5225246Z Error: Failed to process feature ghcr.io/customink/features/debug
2022-11-10T01:06:35.5226026Z     at Fse (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1804:3150)
2022-11-10T01:06:35.5226622Z     at processTicksAndRejections (internal/process/task_queues.js:95:5)
2022-11-10T01:06:35.5227374Z     at async bm (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1804:2294)
2022-11-10T01:06:35.5228223Z     at async Gf (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1806:1327)
2022-11-10T01:06:35.5229044Z     at async cF (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1832:176)
2022-11-10T01:06:35.5229873Z     at async eoe (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1849:2433)
2022-11-10T01:06:35.5230690Z     at async Qse (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1831:2396)
2022-11-10T01:06:35.5231491Z     at async voe (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1893:2222)
2022-11-10T01:06:35.5232321Z     at async Xf (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1893:3212)
2022-11-10T01:06:35.5233384Z     at async Jae (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:2013:15058)

[Chuxel]

Locally you just need to sign into the docker CLI for your private GHCR registry. @joshspicer @jkeech I assume this needs to be added to devcontainer.json to access private features for codeespaces? Might need better notifications about it in Codespaces if so.

{
    "customizations": {
        "codespaces": {
            "repositories": {
                "customink/features": {
                    "permissions": {
                        "packages": "read",
                        "contents": "read"
                    }
                }
            }
        }
    }
}

Also note that any container registry you are signed into (either via the Docker CLI locally, or Codespaces' support for private container registries) can also contain Features for you to use. The above json just does it without an explicit login for Codespaces specificially.

[metaskills]

Thanks, I already had this in place JSON metadata below, but noticed today that it might turn off "write" as indicated by the strikthrough in the screenshot. I tried adding your explicit example and still got the same error. Even after I remove my wildcard permissions. I'll do some more investigation to see what the root cause may be, but still feels like a bug.

Does it help the the features created at customink/features uses the CLI GitHubAction? Could it be a permission issue for some reason?

  "customizations": {
    "codespaces": {
      "repositories": {
        "customink/*": {
          "permissions": {
            "contents": "write",
            "packages": "read"
          }
        }
      }
    }
  },

[metaskills]

OK, I found the issue. My project is using dockerComposeFile and service. If I were to use the image from the service directly as the image in the devcontainer.json, things work. Does that help?

[Chuxel]

OK, I found the issue. My project is using dockerComposeFile and service. If I were to use the image from the service directly as the image in the devcontainer.json, things work. Does that help?

Hmmm. Not catching the scenario where it was and wasn't working since you're showing codespaces pre-builds and mentioning Actions.

The privs for the two products aren't unified is part of this since you, to your point, you likely want the "write" permission in some cases there ... like where the feature originates. You can set both from the configuration for the OCI Artifact in GHCR:

It's the same setup as images, so if you can access a private image in GHCR, you should be able to access features.

The Codespaces json is a shortcut to enable cross-repo access for Codespaces exclusively. I wouldn't expect Compose or image to have any impact.

If it does then it might be a Codespaces bug - not entirely sure. If that's what you are seeing though, @jkeech @joshspicer @edgonmsft @samruddhikhandale might have an idea of what could be going on there. Folks, are there any known issues with Docker Compose-based codespaces?

[metaskills]

and mentioning Actions.

Our customink/features repo uses the devcontainers/ci GitHub Action to publish the packages. Was just sharing that bit of info. I doubt it has something to do with it.

I think core issue is the prebuilds do not work with the private features if I am using dockerComposeFile. If it helps, here is the service in my compose file.

  kcollins:
    build: { context: ., dockerfile: Dockerfile }
    command: sleep infinity
    privileged: true
    user: vscode

[Chuxel]

Ok - managed to repro.

Works fine building the Codespace w/o a pre-build, so this it is very specific to pre-builds with private features like you said.

* Processing feature: ghcr.io/chuxel/feature-library/test-private-feature:latest
(!) WARNING: Falling back to deprecated GitHub Release syntax. See https://github.com/devcontainers/spec/blob/main/proposals/devcontainer-features.md#referencing-a-feature for updated specification.
Github feature.
Could not resolve Feature 'ghcr.io/chuxel/feature-library/test-private-feature:latest'.  Ensure the Feature is published and accessible from your current environment.

The interesting thing is (!) WARNING: Falling back to deprecated GitHub Release syntax is incorrect - so something strange is happening for pre-builds specifically. Looks like a bug within Codespaces itself.

@jkeech @joshspicer @edgonmsft @samruddhikhandale - LMK if you can repro or I can give you access to my repro. The fact that the same setup works for non-prebuilds in Codespaces means its likely not CLI related tho.

[edgonmsft]

@Chuxel Can you give me access to your repro?

[Chuxel]

@edgonmsft https://github.com/Chuxel/compose-private-feature-test is the repo - made it public, but added you as a collaborator so you can see the actions logs.

The feature referenced is in https://github.com/Chuxel/feature-library/tree/main/src/test-private-feature. I just have access on the resulting OCI Artifact set to private.

[edgonmsft]

Thanks @Chuxel it does look like a permissions issue in Codespaces, we are investigating.

@metaskills could you give us more information?

• If you go back to the docker compose config does the prebuild work now?
• What is the name of the repository you are trying to set prebuilds from.
• Can you delete and recreate the prebuild configuration if switching to compose still not works.
• Did you change visibility(private vs public) in the repositories or the packages?

[metaskills]

Thanks @Chuxel it does look like a permissions issue in Codespaces, we are investigating.

Agreed!

If you go back to the docker compose config does the prebuild work now?

No, nothing has changed since reporting the issue. Still not working with private features and docker compose.

What is the name of the repository you are trying to set prebuilds from.

customink/codespaces-kcollins

Can you delete and recreate the prebuild configuration if switching to compose still not works.

Yup, I tried this multiple times. Still does not work.

Did you change visibility(private vs public) in the repositories or the packages?

No, the repo I am using is private, and the feature packages are private. I would not feel comfortable switching them to public to test this. But I can say other public packages work. And private ones work too, just not with prebuilds.

[edgonmsft]

@metaskills does it work if you change the package security to internal? Not sure if that would be good enough for your scenario.

[metaskills]

Sorry, not sure what that means. The devcontainer/ci action publishes these packages on my behalf. If I view them in the repo they are listed as "private". What is "internal"?

[jkeech]

@metaskills, did you ever get this working? I did some searching and found that for private packages, you need to declare permissions bi-directionally in two places: in the consuming repo and in the publishing repo.

1. The consuming repo you are building needs to declare access to the repo where the package is published from in the devcontainer.json, and then you need to accept those permissions in the prebuild configuration.
   • https://docs.github.com/en/enterprise-cloud@latest/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces#setting-additional-repository-permissions
   • It sounds like there's a bug where wildcard permissions in the devcontainer for customink/* won't work for prebuilds currently.
   • In customink/codespaces-kcollins, add the following to the devcontainer and then edit the prebuild config to accept the permissions:

  "customizations": {
    "codespaces": {
      "repositories": {
        "customink/features": {
          "permissions": {
            "packages": "read"
          }
        }
      }
    }
  },

2. The private package also needs to grant access to the consuming repo
   • https://docs.github.com/en/enterprise-cloud@latest/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#ensuring-github-codespaces-access-to-your-package
   • In customink/features, go to the package settings for the various feature packages and grant access to the customink/codespaces-kcollins repository.

After those two steps are done, prebuilds should work. This flow is pretty complicated, and I'm following up to see if we can make it simpler. Let us know how it goes! 🙏

[metaskills]

Thanks for the followup to this, I have not yet had the time to confirm but will use this info when the prebuilds priority comes back up again. Thanks!!!

[metaskills]

Cool, that worked. I also found out about the CODESPACES_PREBUILD_TOKEN environment variable described here:

https://docs.github.com/en/codespaces/prebuilding-your-codespaces/allowing-a-prebuild-to-access-other-repositories

Did not try this method but I suspect it would have worked as well. Maybe we can close this issue?

[jkeech]

Glad to hear it worked! Yes, I think either approach would work, although managing the PAT manually with the CODESPACES_PREBUILD_TOKEN secret might be more of a maintenance challenge over time compared to the permissions in devcontainer.json.

Closing this issue as requested.